On Mar 19, 2010, at 4:12 AM, Arild Nilsen wrote:
GCD is centered around blocks that are scheduled on thread pools. According to Apple's white paper about GCD, "the thread pools are dynamically sized by the system to maximize the performance of the applications using GCD while minimizing the number of idle or competing threads."
Exactly how is the system monitoring and dynamically adjusting the size of these thread pools? From what I have learned so far is that the GCD user-level runtime, libdispatch, makes use of and forwards blocks to an extended pthread interface, referred to as pthread_workqueue. The implementation of this interface executes at the bottom of the user-level, and realizes pools of kernel-level threads. However, I do not quite understand how and where these thread pools are monitored and dynamically adjusted. Is the kernel responsible for doing this explicitly at kernel-level? Or, on the other hand, is it done at the the user-level, with support from the kernel, in a way that the kernel is only concerned with time slicing the kernel-level threads?
In the apple implementation workqueue items are submitted via the pthread.c code to the kernel using the __workq_kernreturn(WQOPS_QUEUE_ADD, ...) system call. The Xnu kernel will decide based on a number of heuristics with parameters that can be adjusted using sysctl(3) if it needs to start new threads to add to the pool. The kernel will also monitor when threads block by monitoring thread context switches and calls to thread yield. There is a timer that will check on the status of the pool to decide if threads need to be added or removed. Workqueue items, when they are executed, are returned to the user space using a thread upcall/continuation to the workqueue function. The current FreeBSD implementation of GCD (in 8.0) doesn't include the kernel support for the thread pool management but uses a user-level thread pool. There are experimental patches to add apple-like kernel managed thread workqueues to the FreeBSD kernel and pthread_workqueue_*_np() calls to the pthread library at http://people.freebsd.org/~sson/thrworkq/ . The pthread_workqueue_*_np() API is the same as Apple's but the system call interface is slightly different.
This forum post suggests that the kernel is responsible for creating the work pools, and hence is aware of which kernel-level threads belong to a particular thread pool. Moreover, this FreeBSD manual page for pthread_workqueue states that: "The pthread_workqueue_*_np() functions are used to create and submit work items to a thread pool. The size of the thread pool is managed by the kernel based on physical resources and the following tunable sysctl(3)...", and then it lists a couple of policy parameters, such as, "the number of microseconds until while a thread is idle until it is removed from the thread pool". Could anyone confirm the correctness of the content in the forum post and manual page?
The forum post seems correct. The quote from the manual page (which is from the experimental FreeBSD implementation) contains a typo and should read: "the number of microseconds while a thread is idle until it is removed from the thread pool". As mentioned above there are a number of tunable parameters for the heuristics in the kernel that manage the size of the thread pool. This man page quote describes one of those parameters.
After stumbling over the two links in the paragraph above I dug into the XNU kernel source code, and came across pthread_internal.h and pthread_synch.c. pthreads_internal.h resembles the policy parameters in the FreeBSD manual page, while pthread_synch.c makes use of these parameters and implements functions for thread pool maintenance, such as, workqueue_removethread() and workqueue_addnewthread(). This makes me believe that the kernel is actually aware of each and every thread pool defined via the pthread_workqueue interface, and that the kernel is continuously monitoring the thread pools and adjusting them transparently to the logic at the user-level. The user-level logic, on the other hand, is only concerned with executing blocks on the current kernel-level threads available in the particular thread pool. To what degree is my understanding representative to what is actually happening?
Yes, the current Apple implementation manages the thread pool size in the kernel and assigned work to threads based on the priority, affinity, etc. of the work items. The management of the thread pool size is mostly transparent to the user-level. As mentioned sysctl parameters can be tweaked to adjust this and there is an attribute to force the kernel to start a new thread for a work item, if needed. The FreeBSD 8.0 implementation uses a user-level managed thread pool. The FreeBSD implementation of GCD may use a kernel managed thread pool in the future like the experimental patches mentioned above. Best Regards, -stacey.