[libdispatch-dev] porting status

[Joakim Johansson]

Hi Jordan,

Some thoughts:

I think that is a good question - although it has been very convenient during the bring-up phase to have it in an external repository, it seems to me that it would be a better solution to integrate at least libpthread_workqueue, as you suggest. Libkqueue seems usable on its own merits, which suggests that it should remain stand-alone.

libpthread_workqueue is a fairly limited codebase and even though some refactoring/packaging may be required, it shouldn’t be too much work to integrate it nicely. 

Even though I do think it should be considered if it would be beneficial to use some other abstraction instead of the pthread_workqueue interface for the non-Mac OS X parts, it is not obviously required at present - the current approach seems to work quite well so far. 

If there would be a clear benefit, at least it is only in ~10 places that libdispatch references the pthread_workqueue API:s, so any such hypothetical #ifdefs would at least be limited in scope.

Another aspect of this though, would be if one wants to bring libpthread_workqueue further in terms of ‘cooperation’ between multiple clients of the library (as discussed between you and Mark a long time ago). 

Personally I think that would be desired an beneficial, even taking the DOS problems brought up then into consideration, at least as an optional input to the thread scale rampup/down logic.

For reference: currently libphtread_workqueue takes into account several inputs for deciding when to gracefully ramp up/down the number of threads used (e.g. number of available cores, current system load average, the number of currently runnable threads/LWP:s in the process, how long (and how many) threads that are idle, ...) - but it is only working really well within the scope of a single process, since the system load average is trailing reality quite significantly (and that is the only cross-process mechanism currently used).

Personally, I think it would make sense to:

1. Consider a (possibly optional) mechanism for better cooperation between multiple clients of libpthread_workqueue (cross-processes) to ‘finish’ the desired functionality
2. Review if there would be any benefit in using a different interface for the thread management for better ‘impedance’ with other platforms
3. Integrate libpthread_workqueue (or similar codebase depending on the outcome of 2. above) with libdispatch

Cheers,

Joakim

ps We (at Tbricks) are currently doing some validation of latency / scalability / performance aspects of libdispatch on non-Mac OS X platforms and hope to be able to verify ‘soon’ that the behavior is good for libdispatch/libpthread_workqueue/libkqueue in concert...


On 11 apr 2011, at 20.12, Jordan K. Hubbard wrote:
> 
> Hi guys,
> 
> Just out of curiosity, is this going to essentially be the "preferred interface" (libpthread_workqueue + libkqueue) going forward for non-MacOSX platforms?  Don't get me wrong:  Anything which requires the fewest contortions in libdispatch to work is great for us since it won't "cruft up" the code with lots of #ifdefs, but at the same time if there are better ways to plumb at least the pthread_workqueue stuff such that we can be more agile on other platforms, it would be interesting to at least *have* that architectural discussion.  Such discussion may or may not lead anywhere, but it seems like we're starting to gain traction on other platforms and it makes me curious as to how well we're "impedance matched" to them.  Any comments on that?
> 
> Thanks,
> 
> - Jordan
>