September 2009 - libdispatch-dev - lists.macosforge.org

libkqueue
by Mark Heily 19 Oct '09

19 Oct '09

After looking at the differences between kqueue(2) and libevent, I decided it would be best to write a very thin wrapper library to emulate kqueue on non-BSD platforms. I've gotten pretty far with the effort to port kqueue to Linux, and wanted to let everyone know that this is a work-in-progress. The working title of this effort is "libkqueue". If you are curious, you can download the libkqueue source code from here: http://mark.heily.com/src/ Currently, I have implemented the EVFILT_READ, EVFILT_WRITE, and EVFILT_SIGNAL filter types. I plan to implement EVFILT_VNODE using inotify, and EVFILT_TIMER using timerfds. I might have to implement EVFILT_PROC using a helper thread. After the Linux port is complete, I'll consider other Unixes such as OpenSolaris. I'm not a Windows developer, so if anyone wants to help with a Windows port, that would be great. It is my hope that the quality and completeness of libkqueue will soon be sufficient enough to be used with libdispatch. Regards, - Mark

3 3

GCD libdispatch w/Blocks support working on FreeBSD
by Robert Watson 26 Sep '09

26 Sep '09

Dear all: This is a quick update for those interested in the general portability of libdispatch, and perhaps specifically FreeBSD. With a bit of help from Jordan Hubbard, I now have libdispatch working with Blocks on FreeBSD. Jordan has put up a FreeBSD clang-devel package that includes Blocks parts and the C runtime bits here: http://static.macosforge.org/libdispatch/downloads/clang-devel.tgz I've updated the libdispatch build parts to detect and use Blocks when compiled with clang, and fixed a few nits in libdispatch that I ran into along the way (and one apparent clang bug). With Jordan's package as a starting point, Blocks pretty much "just worked", so I'm optimistic that people will be able to reproduce this on other platforms able to run the non-Blocks libdispatch without much difficulty. If you update to at least r45 of libdispatch, you should now be able to do: CC=clang ./configure --with-blocks-runtime=/usr/local/lib and get a libdispatch with Blocks support. The reason for the configure argument is that the current clang-devel package doesn't automatically add libBlocksRuntime dependency for binaries compiled with -fblocks. Once Blocks support shakes out a bit more in clang/FreeBSD, this should go away. I am able to run basic Blocks-based test tools with GCD on FreeBSD without difficulty. The FreeBSD port should be updated to reflect this shortly. Robert N M Watson Computer Laboratory University of Cambridge

1 0

Windows compatibility
by Vincent Bourdier 26 Sep '09

26 Sep '09

Hi everybody ! I just discover libdispatch, and I didn't found any information about the portability on other OS than MAC ... is it available on Windows ? Thanks. Regards, Vincent.

8 12

linux port
by Chris Pardy 25 Sep '09

25 Sep '09

Hello Everybody I'm Chris Pardy. I'm looking to compile libdispatch (userspace) on linux (fc 11). Is anybody else on the list working on this? 'Pardy

5 4

[PATCH] Add --enable-optimized and --disable-debug & set --march properly
by Bobby Powers 25 Sep '09

25 Sep '09

Hi folks, This patch cleans up the -march=i686 from src/Makefile.am, setting it in configure.ac if we're not on a 64-bit system. It also adds --enable-optimized and --disable-debug to the configure script, giving slightly more control over CFLAGS (and if you set CFLAGS as part of the configure arguments it overrides all these settings. I had a bit more work to split the headers off into their own directory, but I need to clean up the patchset. It applies with: patch -p1 <path/to/patch Let me know how it looks, in particular the detection of 64-bit is a bit hackish. yours, Bobby

5 5

dispatch source for multi-client sockets
by Mikkel Fahnøe Jørgensen 22 Sep '09

22 Sep '09

How does one create a dispatch source that responds to multiple connections on the same socket, that is, listening on a socket with a backlog larger than 1? In the online doc it seems that you should have a connected socket to read from, the same way as reading from a file. But documentation also says you need a socket returned from listen(). So where does the accept() call fit into this? Specifically it would be nice to avoid blocking on accept() before creating a read source. I found this echo server example based on kqueue events which seems to receive a read event. First the handler will accept the socket, then do normal read processing. Is this also how it should work with libdispatch, and if so, what about other platforms. http://www.monkeys.com/freeware/kqueue-echo.c And how about similar event sources for DNS lookup? Libevent did a bit of work on these issues, and I see it has already been mentioned on this mailing list. Related: I don't see any examples of custom event sources - but I thought that might also be a way to schedule client handlers? Finally, what is the policy on busy loops and locking operations? One could for example push a blocking socket accept() call onto a queue, but this consumes a thread from a thread pool that is designed to reflect number of cpus not number of long running tasks. Alternatively one could create a dedicated thread for this purpose, but that doesn't seem very elegant. Regards, Mikkel

3 2

Compiling OpenMP directives as calls to libdispatch API?
by Stéphane Letz 17 Sep '09

17 Sep '09

Hi, Would it be technically feasible to compile OpenMP directives as calls to libdispatch API? This would possibly allow OpenMP based code to be easily ported on GCD. Comments? Stephane Letz

1 0

libdispatch port to FreeBSD (was: [libdispatch-changes] [27] trunk/src)
by Robert Watson 15 Sep '09

15 Sep '09

Dear all-- I have committed an initial autoconf/automake/libtool build framework to libdispatch svn, and a first cut at conditionally compiling Apple-specific pieces of libdispatch. This allows libdispatch to build on both FreeBSD and Mac OS X Snow Leopard. Some notes for FreeBSD hackers: - Several kqueue extensions (EVFILT_USER, NOTE_TRIGGER, EV_RECEIPT, and EV_DISPATCH) are required in the FreeBSD kernel, and hopefully will go into the FreeBSD tree in the next couple of days. - Blocks are definitely not yet supported -- there seems to be a lot of interest, so I'm optimistic. As compilers aren't my forte, any assistance in this department would be most appreciated. :-) - On FreeBSD, there is no support for Apple's pthread work queue primitive, so I'm using the fallback worker code in libdispatch. The FreeBSD kernel community needs to have a conversation about the right way to address this requirement, but in the mean time the worker model should be fine. - There are also some kqueue events present in FreeBSD but not in Mac OS X; these may require tweaks to libdispatch to fully support by GCD-aware applications using these FreeBSD events. - I have developed and tested on a 32-bit Intel system; the comments below on -march on Mac OS X likely apply for FreeBSD/amd64 users as well. Some notes for Mac OS X hackers: - For now, you will need to hand-trim the -march line from src/Makefile. This should be fixed soon by some further autoconf work. - When I use the externally-built dylib on Snow Leopard, test programs spin. This may be due to interference between the libSystem dispatch code and the dylib -- possibly forcing the use of pthread_create_key rather than reserved keys would help. - I have not attempted to configure/build for Leopard as yet, but this change should make that a lot easier. EVFILT_USER is likely the biggest obstacle. Some notes for Linux hackers: - The lack of kqueue support is the critical obstacle in a port to Linux. It may be that a kqueue emulation library based on epoll (or even libevent) could fill this gap. If someone wants to look at this, pay very careful attention to kqueue semantics, as libdispatch relies on them heavily. Please consider this a work-in-progress -- don't be surprised when it doesn't build, let alone work. :-) Finally: this work would not have been possible without the support of Jordan Hubbard, Kevin Van Vechten, Dave Zarzycki, Bill Siegrist, and Apple. I am surprised by how easily the porting has gone so far -- it's a credit to their careful planning and quality of implementation that it has gone so smoothly. Thanks also to Peter O'Gorman and Colin Percival for their assistance with auto*, my nemesis. With any luck, there will be more good porting news to report in the near future! Thanks, Robert N M Watson Computer Laboratory University of Cambridge ---------- Forwarded message ---------- Date: Mon, 14 Sep 2009 11:46:19 -0700 (PDT) From: source_changes(a)macosforge.org Reply-To: libdispatch-dev(a)lists.macosforge.org To: libdispatch-changes(a)lists.macosforge.org Subject: [libdispatch-changes] [27] trunk/src Revision 27 Author robert(a)fledge.watson.org Date 2009-09-14 11:46:18 -0700 (Mon, 14 Sep 2009) Log Message Port of libdispatch to the FreeBSD operating system, which consisted of disabling various Mac OS X-specific portions of the library (such as Mach), providing portable versions of certain services internally (POSIX semaphores instead of Mach semaphores, clock_gettime() instead of mac_absolute_time()), and providing an autoconf/automake/libtool build infrastructure. This is just a first cut, refinement to follow. Conditionalize portions of the libdispatch build as follows: HAVE_MACH If Mach is present, support Mach ports throughout libdispatch. USE_LIBDISPATCH_INIT_CONSTRUCTOR If defined, add the 'constructor' attribute to libdispatch_init so that it runs as part of library initialization. This is not needed in Mac OS X libSystem due to explicit invocation of libpthread_init by libsyscall, but is for a stand-alone library. HAVE_DECL_EVFILT_SESSION, HAVE_DECL_NOTE_SIGNAL, HAVE_DECL_NOTE_REAP, HAVE_DECL_NOTE_NONE, HAVE_DECL_VQ_UPDATE, HAVE_DECL_VQ_VERYLOWDISK Conditionally support kqueue events that are not present on FreeBSD. DISPATCH_NO_LEGACY Expand the scope of this existing macro to allow libdispatch to build without legacy.c. HAVE_MACH_ABSOLUTE_TIME Wrap mach_absolute_time() in _dispatch_absolute_time(), and provide a portable clock_gettime() implementation. Some further refinment will be required here. USE_APPLE_SEMAPHORE_OPTIMIZATIONS Only use non-portable assembly optimizations of dispatch semaphores if supported. HAVE_SEM_INIT Use POSIX semaphores if Mach semaphores aren't present; further ifdef'ing possibly required here. If neither HAVE_MACH nor HAVE_SEM_INIT are present, the compile will fail. USE_APPLE_CRASHREPORTER_INFO Only define Mac OS X crashreporter state if requested. HAVE_PTHREAD_WORKQUEUES Only compile in support for pthread work queues if present; otherwise force use of normal worker threads. HAVE_PTHREAD_KEY_INIT_NP If defined, use non-portable allocation of pthread keys--otherwise, use pthread_create_key(). USE_APPLE_TSD_OPTIMIZATIONS Only use non-portable assembly optimizations if dispatch thread- specific data if this is defined. HAVE_MALLOC_CREATE_ZONE If not present, use compat/malloc_zone.h which implements the malloc zone APIs using normal malloc(3). Also fix some portability nits for building, such as casting before using long as a format string for several types. Note that this port will not build or run on a stock FreeBSD release or svn distribution yet, as it relies on kqueue extensions not committed back to FreeBSD yet. There are several known issues, such as how best to map CLOCK_UPTIME or CLOCK_MONOTONIC struct timespec timestamps into a Mach absolute time int64_t number space. Also, the POSIX semaphore ifdefing is adequate but not complete. Modified Paths * trunk/src/benchmark.c * trunk/src/dispatch.h * trunk/src/internal.h * trunk/src/legacy.h * trunk/src/os_shims.h * trunk/src/private.h * trunk/src/queue.c * trunk/src/queue_internal.h * trunk/src/semaphore.c * trunk/src/semaphore_internal.h * trunk/src/shims.c * trunk/src/source.c * trunk/src/source.h * trunk/src/source_private.h * trunk/src/time.c Diff Modified: trunk/src/benchmark.c (26 => 27)

8 9

Does GCD has a "global view" of all running processes?
by Stéphane Letz 15 Sep '09

15 Sep '09

Hi, After reading several things on GCD on the net, it is not yet clear if GCD is mainly a "per process" library, or is supposed to have a "global view" of all processes, so that to allocate "not too much" threads in each process, and avoid having a number of threads that finally exceed the number of available cores on the machine. What is the situation? Thanks Stephane Letz

4 4

Using libdispatch in a real-time context
by Stéphane Letz 15 Sep '09

15 Sep '09

Hi, Audio developers on OSX would be interested to be able to use libdispatch in their applications. We understand that right now libdispatch is not designed to be used in a RT context (I guess because of memory issues when using blocks...). Would it be technically possible to extend libdispatch to be used in a RT context? so basically having worker threads to be setup with "time- constraint" kind of priority on OSX... and possibly working on a RT capable memory manager for blocks management ? Thanks Stéphane Letz

2 1