3/09/2009

Windows, Unix and ANSI C API Comparison

  The "Windows System Programming"(3E) has a great appendix about Windows, Unix, C Library API comparison. It's obvious that this appendix lacks of many APIs in Memory, DLL and Security related areas, but it is still very helpful for referencing when doing system programming on windows platform.
  Original content can be found here: http://my.safaribooksonline.com/0321256190/app02

1. I/O

1. 1 File & Directory


SubjectWindowsUNIXC LibraryComments
Console I/OAllocConsoleterminal I/ON/A
Console I/OFreeConsoleterminal I/ON/A
Console I/OReadConsolereadgetc, scanf, gets
Console I/OSetConsoleModeioctlN/A
Console I/OWriteConsolewriteputc, printf, puts
Directory MgtCreateDirectorymkdir*N/AMake a new directory
Directory MgtFindCloseclosedir*N/AClose a directory search handle
Directory MgtFindFirstFileopendir*, readdir*N/AFind first file matching a pattern
Directory MgtFindNextFilereaddir*N/AFind subsequent files
Directory MgtGetCurrentDirectorygetcwd*N/A
Directory MgtGetFullPathNameN/AN/A
Directory MgtGetSystemDirectoryWell-known pathnamesN/A
Directory MgtRemoveDirectoryrmdir, unlink*remove
Directory MgtSearchPathUse opendir, readdirN/ASearch for a file on a specified path
Directory MgtSetCurrentDirectorychdir*, fchdirN/AChange the working directory
Error HandlingFormatMessagestrerrorperror
Error HandlingGetLastErrorerrnoerrnoGlobal variable
Error HandlingSetLastErrorerrnoerrnoGlobal variable
File LockingLockFilefcntl (cmd=F_GETLK, ..)N/A
File LockingLockFileExfcntl (cmd=F_GETLK, ..)N/A
File LockingUnlockFilefcntl (cmd=F_GETLK, ..)N/A
File LockingUnlockFileExfcntl (cmd=F_GETLK, ..)N/A
File SystemCloseHandle (file handle)close*fcloseCloseHandle is not limited to files
File SystemCopyFileopen; read; write; closefopen; fread; fwrite; fcloseDuplicate a file
File SystemCreateFileopen*, creat*fopenOpen/create a file
File SystemDeleteFileunlink*removeDelete a file
File SystemFlushFileBuffersfsynchfflushWrite file buffers
File SystemGetFileAttributesstat*, fstat*, lstatN/A
File SystemGetFileInformationByHandlestat*, fstat*, lstatN/AFill structure with file info
File SystemGetFileSizestat*, fstat*, lstatftell, fseekGet length of file in bytes
File SystemGetFileTimestat*, fstat*, lstatN/A
File SystemGetFileTypestat*, fstat*, lstatN/ACheck for character stream device or file
File SystemGetStdHandleUse file desc 0, 1, or 2Use stdin, stdout, stderr
File SystemGetTempFileNameUse C librarytmpnamCreate a unique file name
File SystemGetTempFileName, CreateFileUse C librarytmpfileCreate a temporary file
File SystemGetTempPath/temp pathN/ADirectory for temp files
File SystemMoveFile, MoveFileExUse C libraryrenameRename a file or directory
File SystemCreateHardLinklink, unlink*N/AWindows does not support links
File SystemN/AsymlinkN/ACreate a symbolic link
File SystemN/AreadlinkN/ARead name in a symbolic link
File SystemN/A, ReadFile returns 0 bytesN/A, read returns 0 bytesfeofRest for end of file
File SystemN/A, use multiple ReadFilesreadvN/A, use multiple freadsScatter read
File SystemN/A, use multiple WriteFileswritevN/A, use multiple fwritesGather write
File SystemReadFilereadfreadRead data from a file
File SystemSetEndOfFile chsize*N/A
File SystemSetFileAttributesfcntlN/A
File SystemSetFilePointerlseekfseekSet file pointer
FileSystemSetFilePointer (to 0)lseek (0)rewind
File SystemSetFileTimeutime*N/A
File SystemSetStdHandleclose, dup*, dup2*, or fcntlfreopendup2 or fcntl
File SystemWriteFilewritefwriteWrite data to a file

1.2 Async I/O

SubjectWindowsUNIXC LibraryComments
Asynch I/OGetOverlappedResultN/AN/A
Asynch I/OReadFileExN/AN/AExtended I/O with completion routine
Asynch I/OSleepExN/AN/AAlertable wait
Asynch I/OWaitForMultipleObjects (file handles)poll, selectN/A
Asynch I/OWaitForMultipleObjectsExN/AN/AAlertable wait
Asynch I/OWriteFileExN/AN/AExtended I/O with completion routine
Asynch I/OWaitForSingleObjectExwaitpidN/AAlertable wait

2. Memory & DLL

SubjectWindowsUNIXC Library
Mapped FilesCreateFileMappingshmgetN/A
Mapped FilesMapViewOfFilemmap, shmatN/A
Mapped FilesMapViewOfFileExmmap, shmatN/A
Mapped FilesOpenFileMappingshmgetN/A
Mapped FilesUnmapViewOfFilemunmap, shmdt, shmctlN/A
Memory MgtGetProcessHeapN/AN/A
Memory MgtGetSystemInfoN/AN/A
Memory MgtHeapAllocsbrk, brk, or C librarymalloc, calloc
Memory MgtHeapCreateN/AN/A
Memory MgtHeapDestroyN/AN/A
Memory MgtHeapFreeUse C libraryfree
Memory MgtHeapReAllocUse C libraryrealloc
Memory MgtHeapSizeN/AN/A
Shared MemoryCloseHandle (map handle)shmctlN/A
Shared MemoryCreateFileMapping, OpenFileMappingshmgetN/A
Shared MemoryMapViewOfFileshmatN/A
Shared MemoryUnmapViewOfFileshmdtN/A
DLLsLoadLibrarydlopenN/A
DLLsFreeLibrarydlcloseN/A
DLLsGetProcAddressdlsynN/A
DLLsDllMainpthread_onceN/A

3. Process & Thread

3.1 Process


SubjectWindowsUNIXC LibraryComments
Process MgtCreateProcessfork (); execl ()*, system()N/AThere are 6 execxx functions
Process MgtExitProcess_exitexit
Process MgtGetCommandLineargv []argv []
Process MgtGetCurrentProcessgetpid*N/A
Process MgtGetCurrentProcessIdgetpid*N/A
Process MgtGetEnvironmentStringsN/Agetenv
Process MgtGetEnvironmentVariableN/Agetenv
Process MgtGetExitCodeProcesswait, waitpidN/A
Process MgtGetProcessTimestimes, wait3, wait4N/A
Process MgtGetProcessWorkingSetSizewait3, wait4N/A
Process MgtN/Aexecl*, execv*, execle*, execve*, execlp*, execvp*N/AWindows does not have a direct equivalent
Process MgtN/Afork, vforkN/AWindows does not have a direct equivalent
Process MgtN/AgetppidN/ANo parent/child relationships in Windows
Process MgtN/Agetgid, getegidN/ANo process groups in Windows
Process MgtN/AgetpgrpN/A
Process MgtN/AsetpgidN/A
Process MgtN/AsetsidN/A
Process MgtN/AtcgetpgrpN/A
Process MgtN/AtcsetpgrpN/A
Process MgtOpenProcessN/AN/A
Process MgtSetEnvironmentVariableputenvN/Aputenv is not part of the Standard C library
Process MgtTerminateProcesskillN/A
Synch: ProcessWaitForMultipleObjects (process handles)waitpidN/A
Synch: ProcessWaitForSingleObject (process handle)wait, waitpidN/A
TimersKillTimeralarm (0)N/A
TimersSetTimeralarmN/A
TimersSleepsleepN/A
TimersSleeppoll or select, no file descriptorN/A

3.2 Thread

SubjectWindowsUNIX/PthreadsComments
Thread MgtCreateRemoteThreadN/A
TLSTlsAllocpthread_key_alloc
TLSTlsFreepthread_key_delete
TLSTlsGetValuepthread_getspecific
TLSTlsSetValuepthread_setspecific
Thread MgtCreateThread, _beginthreadexpthread_create
Thread MgtExitThread, _endthreadexpthread_exit
Thread MgtGetCurrentThreadpthread_self
Thread MgtGetCurrentThreadIdN/A
Thread MgtGetExitCodeThreadpthread_yield
Thread MgtResumeThreadN/A
Thread MgtSuspendThreadN/A
Thread MgtTerminateThreadpthread_cancelpthread_cancel is safer
Thread MgtWaitForSingleObject(thread handle)pthread_join
Thread PriorityGetPriorityClasspthread_attr_getschedpolicy, getpriority
Thread PriorityGetThreadPrioritypthread_attr_getschedparam
Thread PrioritySetPriorityClasspthread_attr_setschedpolicy, setpriority, nice
Thread PrioritySetThreadPrioritypthread_attr_setschedparam
Note: Pthreads, while a part of all modern UNIX offerings, are available on non-UNIX systems as well.

3.3 Synchronization

SubjectWindowsUNIX/PthreadsComments
Synch: CritSecDeleteCriticalSectionUse mutexes to emulate critical sections. Some systems provide proprietary equivalents.C library is not applicable
Synch: CritSecEnterCriticalSectionC library is not applicable
Synch: CritSecInitializeCriticalSection
Synch: CritSecLeaveCriticalSection
Synch: EventCloseHandle (event handle)pthread_cond_destroy
Synch: EventCreateEventpthread_cond_init
Synch: EventPulseEventpthread_cond_signalManual-reset event
Synch: EventResetEventN/A
Synch: EventSetEventpthread_cond_broadcastAuto-reset event
Synch: EventWaitForSingleObject (event handle)pthread_cond_wait
Synch: EventWaitForSingleObject (event handle)pthread_timed_wait
Synch: MutexCloseHandle (mutex handle)pthread_mutex_destroy
Synch: MutexCreateMutexpthread_mutex_init
Synch: MutexReleaseMutexpthread_mutex_unlock
Synch: MutexWaitForSingleObject (mutex handle)pthread_mutex_lock
Synch: SemCreateSemaphoresemget
Synch: SemN/AsemctlWindows does not directly support all these options
Synch: SemOpenSemaphoresemget
Synch: SemReleaseSemaphoresemop (+)
Synch: SemWaitForSingleObject (semaphore handle)semop (-)Windows can wait for only one count

3.4 IPC

SubjectWindowsUNIXC LibraryComments
IPCCallNamedPipeN/AN/ACreateFile, WriteFile, ReadFile, CloseHandle
IPCCloseHandle (pipe handle)close, msgctlpcloseNot part of the Standard C library—see Stevens
IPCConnectNamedPipeN/AN/A
IPCCreateMailslotN/AN/A
IPCCreateNamedPipemkfifo, msggetN/A
IPCCreatePipepipepopenNot part of the Standard C library—see Stevens
IPCDuplicateHandledup, dup2, or fcntlN/AOr use file names CONIN$, CONOUT$
IPCGetNamedPipeHandleStatestat, fstat, lstat64N/A
IPCGetNamedPipeInfostat, fstat, lstatN/A
IPCImpersonateNamedPipeClientN/AN/A
IPCPeekNamedPipeN/AN/A
IPCReadFile (named pipe handle)read (fifo), msgsndN/A
IPCRevertToSelfN/AN/A
IPCSetNamedPipeHandleStateN/AN/A
IPCTransactNamedPipeN/AN/AWriteFile; ReadFile
IPCWriteFile (named pipe handle)write (fifo), msgrcvN/A
Misc.GetComputerNameunameN/A
Misc.SetComputerNameN/AN/A
SecuritySetNamedPipeIdentityUse directory sticky bitN/A

4. Misc

4.1 Security

SubjectWindowsUNIXComments
SecurityAddAccessAllowedAcechmod, fchmodC library does not support security

SecurityAddAccessDeniedAcechmod, fchmod
SecurityAddAuditAceN/A
SecurityCreatePrivateObjectSecurityN/A
SecurityDeleteAcechmod, fchmod
SecurityDestroyPrivateObjectSecurityN/A
SecurityGetAcestat*, fstat*, lstat
SecurityGetAclInformationstat*, fstat*, lstat
SecurityGetFileSecuritystat*, fstat*, lstat
SecurityGetPrivateObjectSecurityN/A
SecurityGetSecurityDescriptorDaclstat*, fstat*, lstat
SecurityGetUserNamegetlogin
SecurityInitializeAclN/A
SecurityInitializeSecurityDescriptorUmask
SecurityLookupAccountNamegetpwnam, getgrnam
SecurityLookupAccountSidgetpwuid, getuid, geteuid
SecurityN/Agetpwend, setpwent, endpwent
SecurityN/Agetgrent, setgrent, endgrent
SecurityN/ASetuid, seteuid, setreuid
SecurityN/ASetgid, setegid, setregid
SecurityOpenProcessTokengetgroups, setgroups, initgroups
SecuritySetFileSecuritychmod*, fchmod
SecuritySetPrivateObjectSecurityN/A
SecuritySetSecurityDescriptorDaclUmask
SecuritySetSecurityDescriptorGroupchown, fchown, lchown
SecuritySetSecurityDescriptorOwnerchown, fchown, lchown
SecuritySetSecurityDescriptorSaclN/A

4.2 Exception Handling

SubjectWindowsUNIXC Library
SEH_try – _exceptUse C library signalsUse C library signals
SEH_try – _finallyUse C library signalsUse C library signals
SEHAbnormalTerminationUse C library signalsUse C library signals
SEHGetExceptionCodeUse C library signalsUse C library signals
SEHRaiseExceptionUse C library signalssignal, raise
SignalsUse _finally blockUse C libraryatexit
SignalsUse C library or terminate processkillraise
SignalsUse C libraryUse C librarysignal
SignalsUse SEH, VEHsigemptysetN/A
SignalsUse SEH, VEHsigfillsetN/A
SignalsUse SEH, VEHsigaddsetN/A
SignalsUse SEH, VEHsigdelsetN/A
SignalsUse SEH, VEHsigismemberN/A
SignalsUse SEH, VEHsigprocmaskN/A
SignalsUse SEH, VEHsigpendingN/A
SignalsUse SEH, VEHsigactionN/A
SignalsUse SEH, VEHsigsetjmpN/A
SignalsUse SEH, VEHsiglongjmpN/A
SignalsUse SEH, VEHsigsuspendfN/A
SignalsUse SEH, VEHpsignalN/A
SignalsUse SEH, VEH, or C libraryUse C libraryabort
Note: Many UNIX vendors provide proprietary exception handling capabilities.

4.3 System Information & Time

SubjectWindowsUNIXC LibraryComments
System InfoGetDiskFreeSpaceN/AN/A
System InfoGetSystemInfogetrusageN/A
System InfoGetVersionunameN/A
System InfoGetVolumeInformationN/AN/A
System InfoGlobalMemoryStatusgetrlimitN/A
System InfoVarious defined constantssysconf, pathconf, fpathconfN/A
TimeGetSystemTimeUse C librarytime, gmtime
TimeSee ls program,
Use C libraryasctime
TimeCompareFileTimeUse C librarydifftimeCompare "calendar" times
TimeFileTimeToLocalFileTime, FileTimeToSystemTimeUse C librarylocaltime
TimeFileTimeToSystemTimeUse C librarygmtime
TimeGetLocalTimeUse C librarytime, localtime
TimeSee touch program,
Use C librarystrftime
TimeSetLocalTimeN/AN/A
TimeSetSystemTimeN/AN/A
TimeSubtract file timesUse C librarydifftime
TimeSystemTimeToFileTimeUse C librarymktime

3/07/2009

common Problems with Flex on Windows

The followings are common errors when using Flex to do language processing on Windows platform.

1. isatty()

Problem:
error LNK2019: unresolved external symbol "int __cdecl isatty(int)" referenced in function "void __cdecl yy_init_buffer(struct yy_buffer_state *,struct _iobuf *)"

Solution:
  1.1 define YY_NEVER_INTERACTIVE as 1 or o in your lex syntax file

2. unistd.h

Problem:
fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory


Solution:
  2.1 use flex parameter: --nounistd
  2.2 use flex option: %option nounistd
  2.3 try other version of flex, which will produce "#ifndef _MSC_VER #include #endif" like code
  2.4 create an empty unistd.h file under the include search directory of your compiler

3/05/2009

The World, When Real Meets Virtual

  《三联生活周刊》(Issue 2, Vol. 2009) 做了一期关于互联网时代虚拟世界的专题。虽然专题的切入点是“虚拟博物馆”这个文化话题,但作为一个非技术类大众刊物,能把虚拟世界的现状、问题和前景,以及前生后世分析得如此清晰透彻,实属难得。

  从一个IT技术从业人士的角度看,这个专题的意义在于它用另外一个更加全局的视野和高度(用文中的术语:Topsight)来认识看待今天表面上纷繁复杂商业、技术之争,揭示了互联网虚拟世界应用,在社会、心理、文化层面上的潜在动因。

1. The Origination of Virtual World
在科学院念书的时候,我们从事的工作被人批评为是"错误地"以"Vision Driven"做为指导原则,我们被告之:"正确的"原则应该是"Problem Driven"。当多年过去,视野变得更加广阔之后才发现,历史上很多重要的创新其实都是Vision Driven的。
1982年, William Gibson在短篇科幻小说"Burning Chrome"中发明了Cyberspace一词。Gibson两年后出版了荣获雨果、星云、菲利普三大科幻界最高奖项的小说"Neuromancer(神经漫游者)",Cyberspace的概念也随着该书的成功逐渐被大众所接受流传。此后的四分之一个世纪,无数仁人志士前赴后继不断努力,试图从各种角度将此Idea变成Reality。从Internet到Web, 从Facebook到Secondlife,都可以看到这一Vision对于技术创新的指导意义。

  1992年, Neal Stephenson出版了深受Geek追捧的"Snow Crash"一书,该书发明了Metaverse一词,Metaverse = Meta + Universe,是一个巨大的3D虚拟城市,人们通过Avatar登录,寻找娱乐、朋友和交易。Google Earth和Second Life被认为是此Vision激励下的产物。

2. The State of the Art of Virtual World


  目前可以称得上Virtual World的应用可以简单分为2D与3D两大类别:

  2D应用:主要是指基于纯Web的Social Network System
  - Facebook, 记录你在现实世界中的人际关系和相关活动
  - Twitter, 记录你个人的日常活动和状态
  - Flickr, 以图片方式记录你日常所见
  - Youtube, 以视频方式记录日常所见、表达自我

  3D应用:主要是指基于GIS和3D建模技术的网络应用系统
  - Second Life, 近乎游戏的在线网络应用,和现实世界关联不大
  - Google Earth, 将现实世界中的地理、位置、建筑信息数字化
  - Digital Library/Museum, 以网络化、数字化方式呈现博物馆、图书馆服务

  可以看到:
  1. 这两大类应用的共同点是系统运营商提供基础架构和工具,用户在上面创造各类反映现实世界的数据信息,并同自己的现实生活发生双向交互。对于2D类的应用,数据都是关于用户的,自然只有用户才能创造数据。对于3D类应用,服务提供商往往提供一些公共数据(如google earth的卫星地图、SecondLife的小岛地形、博物馆的展品资料等),但各种细节信息(比如google earth上的建筑信息)由于数量实在庞大,仍然需要以全民自愿参与的方式由用户提供。

  2. 2D应用的意义在于将人类的活动数字化、网络化,3D应用的作用在于将现实世界中的物理部分数字化。这主要源自各自的表现能力不同,3D内的应用由于表现能力丰富具体,更适合展示诸如建筑、地形、地图等自然世界元素,而基于2D的web类应用更适合对带关联的数据进行组织、整理和检索。

  3. 虚拟博物馆和数字图书馆为什么在SecondLife这样的环境中遍地开花而鲜有和Facebook这样的应用结合的主要原因在于:一,在表现媒介上,它们都需要丰富的(3D, Multimedia)表现能力;二,在交互方式上,除了Link/Search这样的基本操作,它们还需要提供类同现实世界的Browse/Navigation功能。

3. The Tough Challenges & Great Future

  1992年,耶鲁大学计算机系教授David Gelernter出版了"Mirror World" 一书,他在书中预言:计算机和互联网络发展的结果,将会把现实世界数字化、虚拟化,在数字世界中以空间和时间两个角度创造出一个Mirror World。它是现实世界的数字化表示,给人类提供一个Topsight,让我们以前所未有的能力和角度来观察、认识、理解进而控制、改变现实世界。

  在Gelernter看来:人们最终关心的,依然是自己在一个真实的、而不是一个虚构或者虚假的世界里的存在;互联网的最终形态会是Mirror World - 物理世界的虚拟映射。

  SecondLife在吸引了过多眼球之后,用户开始慢慢流失,因为大家找不到停留在上面的理由。对一小部分Geek而言,也许能找到类似"山在那里"的自我价值实现带来的满足感。但对大部分普通用户而言,在系统中游历一番之后才发觉,除了无谓地消费Geek们的创造,自己似乎无所事事浪费生命。

  反观Facebook/Twitter,以及同样俱备3D能力的Google Earth,都受到了用户的持续追捧。究其原因,在于它们营造的虚拟世界,有太多真实世界的影子: they are just part of our real lives, not another/second disconnected life in Cyberspace, like those in SecondLife。

  Virtual World is Great Only When it Connects With Reality.

  虚拟世界如果想要成为现实世界的Mirror还将面临另外一个挑战:static VS dynamic/realtime:
  现实世界是随着时间流动的,每个人、每件事乃至每个地点都是data stream,不是静态的一幅图、一段文字。如何记录、呈现、整理这些信息?
  对于现实世界,我们更多的是想了解当下Realtime的信息和状态,不是几年前的Snapshot。我们能不能在Google Earth/SecondLife上看到西直门现在的交通路况?能不能把它们的focus定位在意甲球场,看一场酣畅的球赛?

  不论虚拟和现实世界如何进化、如何相互影响,它们始终是两个不同的世界,连接这两个世界的只能是人,而连接的方式是所谓的"人机交互(HCI)"。在HCI领域,一直以模拟自然世界作为Vision来推动技术发展。从语音识别、Motion Capture到Multitouch无不在向现实世界逼近。当各种虚拟世界最终融合为"One Machine"[1,2,3]的时候,什么才是最Natural的交互方式? 是3D Touch还是大脑直接驳接电缆?

4. Apocalypse Now

  无线传感器及其网络如何将现实世界实时数字化?
  如将个人的生活数字化并同世界上其它数据关联、融合?
  数量巨大的数据如何收集、存储、处理、快速呈现?
  关联广告依然是最有效的商业模式?
  如何处理信息的精确、丰富与隐私、安全的矛盾?

[References]

About One Machine
[1] http://highscalability.com/scalability-perspectives-4-kevin-kelly-one-machine
[2] http://www.kk.org/thetechnium/archives/2007/11/dimensions_of_t.php
[3] http://www.ted.com/index.php/talks/kevin_kelly_on_the_next_5_000_days_of_the_web.html

About Mirror World & David Gelernter
[4]http://en.wikipedia.org/wiki/David_Gelernter
[5]http://java.sun.com/developer/technicalArticles/Interviews/gelernter_qa.html
[6]http://www.uh.edu/~jbutler/anon/mirror.html
[7]http://www.metaverseroadmap.org/MetaverseRoadmapOverview.pdf

About Neal Stephenson
[8]http://en.wikipedia.org/wiki/Neal_Stephenson
[9]http://en.wikipedia.org/wiki/Snow_Crash
[10]http://mihuan.yo2.cn/archives/645304 (Snow Crash, Chinese Version)

About William Gibson
[11] http://en.wikipedia.org/wiki/William_Gibson
[12] http://blog.sina.com.cn/s/blog_5cd0c1e80100b4ef.html(Burning Chrome, Chinese Version)
[13] http://services.exeter.ac.uk/cmit/media/texts/gibson/burning_chrome.pdf
About the Reports by 三联生活周刊
[14] 世界的镜子 http://www.lifeweek.com.cn/2009-01-12/0000123806.shtml
[15] David Gelernter专访 http://www.lifeweek.com.cn/2009-01-12/0000123805.shtml