Feedback: Dispatch high-level wrappers
Hi all, As hopefully you all know, MacRuby 0.5 includes wrappers for almost all of the C "dispatch" APIs for Grand Central Dispatch. While relatively straightforward to use, the APIs are somewhat verbose and C-ish. http://svn.macosforge.org/repository/ruby/MacRuby/trunk/spec/macruby/core/gc... For MacRuby 0.6, I'd like to provide higher-level convenience APIs to reduce the "cognitive overhead" and "semantic noise" of using Grand Central Dispatch. In other words, let people do what they want without having to learn or type as much as they do now. My first attempt at this is a new 'dispatch' library: http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/ which (from the latest MacRuby nightly) you can access via: require 'dispatch' This defines: - A Dispatch::Actor class which wraps a given object with serialization and asynchronous callbacks http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/actor.r... - Convenience methods on the Dispatch module http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/dispatc... - Parallel extensions to Enumerable http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/enumera... - A Futures class for delayed execution (duck-typed to lambda) http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/future.... - a 'stride' function on top of Dispatch::Queue#apply http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/queue.r... - Queue-based source constructors http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/queue_s... I've also written up specs to both test and demonstrate these constructs: http://svn.macosforge.org/repository/ruby/MacRuby/trunk/spec/macruby/library... These are *extremely* preliminary, and lacking in both polish and documentation. They may not be entirely correct, or even the appropriate abstractions to release as public API. However, I wanted to get something out there to provoke feedback, so we can iterate together. Hopefully we can also use this thread to start learning how to explain both the concepts and the API, as it evolves. So, feedback away! - Ernie P.
Hi all, One quick change -- I redid Futures to duck-type to Thread instead of lambda:
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/future....
and changed Dispatch.fork to use that instead:
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/dispatc...
Do let me know if you have any other suggestions. Thanks! -- Ernie P. On Feb 8, 2010, at 5:46 PM, Ernest N. Prabhakar, Ph.D. wrote:
Hi all,
As hopefully you all know, MacRuby 0.5 includes wrappers for almost all of the C "dispatch" APIs for Grand Central Dispatch. While relatively straightforward to use, the APIs are somewhat verbose and C-ish.
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/spec/macruby/core/gc...
For MacRuby 0.6, I'd like to provide higher-level convenience APIs to reduce the "cognitive overhead" and "semantic noise" of using Grand Central Dispatch. In other words, let people do what they want without having to learn or type as much as they do now.
My first attempt at this is a new 'dispatch' library:
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/
which (from the latest MacRuby nightly) you can access via:
require 'dispatch'
This defines:
- A Dispatch::Actor class which wraps a given object with serialization and asynchronous callbacks
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/actor.r...
- Convenience methods on the Dispatch module
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/dispatc...
- Parallel extensions to Enumerable
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/enumera...
- A Futures class for delayed execution (duck-typed to lambda)
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/future....
- a 'stride' function on top of Dispatch::Queue#apply
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/queue.r...
- Queue-based source constructors
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/queue_s...
I've also written up specs to both test and demonstrate these constructs:
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/spec/macruby/library...
These are *extremely* preliminary, and lacking in both polish and documentation. They may not be entirely correct, or even the appropriate abstractions to release as public API. However, I wanted to get something out there to provoke feedback, so we can iterate together. Hopefully we can also use this thread to start learning how to explain both the concepts and the API, as it evolves.
So, feedback away!
- Ernie P.
_______________________________________________ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
I'm glad to hear about Ernie's GCD wrapper effort, both specifically and as an indication that some general patterns may be developing in the area of Rubifying wrappers for industry APIs. I have watched Rich Kilmer's HotCocoa effort with great interest; I'm currently involved in working on a similarly-motivated effort to make Jena (a Java Semantic Web framework) more pleasant to use under JRuby. It strikes me that these three projects (and others that may be around) will have commonalities in the fact that they are mapping similar C-like languages (eg, Java, ObjC) to Ruby. However, there are certain to be a number of differences, based on application areas, base languages, etc. Unfortunately, I don't have any specific patterns of my own to offer, as yet. So, I'm hoping that Rich and Ernie might be willing to tell the rest of us about their approaches and rationales. I'd also like to hear about other projects (and notions) in this area. -r -- http://www.cfcl.com/rdm Rich Morin http://www.cfcl.com/rdm/resume rdm@cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Technical editing and writing, programming, and web development
Hi Rich, On Feb 9, 2010, at 4:23 PM, Rich Morin wrote:
Unfortunately, I don't have any specific patterns of my own to offer, as yet. So, I'm hoping that Rich and Ernie might be willing to tell the rest of us about their approaches and rationales.
Interesting. For what it's worth, here's the process I've ended up following for GCD. Note that I didn't really start out with this plan, it largely emerged as I went along http://svn.macosforge.org/repository/ruby/MacRuby/trunk/gcd.c A. Create a thin veneer that "almost" matches the underlying C API - except: 1. Turn data structures into true objects, and their functions into methods 2. Map underlying memory management to Ruby garbage collection 3. Prevent Ruby code from ever creating hard C crashes 4. Take advantage of optional arguments to reduce API "noise" 5. Namespace constants 6. Where possible, combine related methods together to prevent sequencing errors The last is the most controversial. In this particular case, I implemented GCD sources in a way that automatically configures them during "new", which is much simpler but also reduces flexibility. I am fairly (but not 100%) certain that this handles all the common cases, so I believe it an acceptable tradeoff -- but I could be wrong. http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/ B. Create a high-level convenience API that: 1. Encapsulates common design patterns 2. Hides infrequently used options behind default arguments 3. Provides Ruby idioms and duck-types 4. Allows trivial invocation of the most common cases The jury is still out on whether I've succeeded, but my initial attempts to use the API to implement itself are promising.
I'd also like to hear about other projects (and notions) in this area.
I assume you're familiar with Ruby FFI? Worth asking over there, as presumably they've wrestled with this issue before... Hope this helps, -- Ernie P.
On Feb 8, 2010, at 5:46 PM, Ernest N. Prabhakar, Ph.D. wrote:
For MacRuby 0.6, I'd like to provide higher-level convenience APIs to reduce the "cognitive overhead" and "semantic noise" of using Grand Central Dispatch. In other words, let people do what they want without having to learn or type as much as they do now.
My first impression is "wow, cool!" followed by a second impression of "Hmmm! That's a lot of semi-parallel parallelization mechanisms to digest at one time!" Which is not to say that I think you've overdone it here - this may be exactly the right number of permutations - but I'm having a hard time getting my head around the totality of what you're proposing clients of this code can actually do with it. Having links to the implementations themselves is cool, and provides a fine API reference of a sort, but I think these sorts of concepts are far better conveyed through usage cases than implementation details. Maybe if each of your bullets came with a small code fragment which demonstrated how to use the wrapper to solve Real World Problem Foo, even managers like myself could make the individual cognitive leaps much more readily. :-) - Jordan
Hi Jordan, On Feb 10, 2010, at 12:30 AM, Jordan K. Hubbard wrote:
On Feb 8, 2010, at 5:46 PM, Ernest N. Prabhakar, Ph.D. wrote:
I've also written up specs to both test and demonstrate these constructs:
P.S. Is it just me, or do other people have a really hard time turning specs into usage-case examples when they read them?
I don't disagree. Specs are arguably a half-step towards documentation. That's the next step, to actually create some real documentation. Ironically, that is what I initially set out to do, but the API wasn't clean and complete enough to document, so I got sidetracked... -- Ernie P.
Hi Jordan, On Feb 10, 2010, at 12:30 AM, Jordan K. Hubbard wrote:
P.S. Is it just me, or do other people have a really hard time turning specs into usage-case examples when they read them?
In answer to your plea, I've not only commented up the Dispatch module with examples: http://svn.macosforge.org/repository/ruby/MacRuby/trunk/lib/dispatch/dispatc... I've expanded them into a sample-macruby script: http://svn.macosforge.org/repository/ruby/MacRuby/trunk/sample-macruby/Scrip... (also below) Note that this requires the latest nightly to work, as I redid the semantics of Dispatch.group. Fairly minimal, but something to start with. Does that help? -- Ernie P. #!/usr/local/bin/macruby require 'dispatch' puts "\n Use Dispatch.async to do stuff in the background" Dispatch.async { p "Did this later" } sleep 0.1 puts "\n Use Dispatch.group to track when stuff completes" g = Dispatch.group { p "Do this" } Dispatch.group(g) { p "and that" } g.wait p "Done" puts "\n Use Dispatch.fork to capture return values in a Future" f = Dispatch.fork { 2+2 } p f.value puts " - pass a block to return the value asynchronously" f.value { |v| p "Returns #{v}" } sleep 0.1 puts "\n Use Dispatch.queue_for to create a private serial queue" puts " - synchronizes access to shared data structures" a = Array.new q = Dispatch.queue_for(a) puts " - has a (mostly) unique name:" p q q.async { a << "change me" } puts " - uses sync to block and flush queue" q.sync { p a } puts "\n Use with a group for more complex dependencies, " q.async(g) { a << "more change" } Dispatch.group(g) do tmp = "complex calculation" q.async(g) { a << tmp } end puts " - uses notify to execute block when done" g.notify(q) { p a } q.sync {} puts "\n Use Dispatch.wrap to serialize object using an Actor" b = Dispatch.wrap(Array) b << "safely change me" p b.size # => 1 (synchronous return) b.size {|n| p "Size=#{n}"} # => "Size=1" (asynchronous return)
That's definitely a good start, yes, though I must confess to not quite understanding the dispatch.fork example? On Feb 12, 2010, at 10:20 AM, Ernest N. Prabhakar, Ph.D. wrote:
Note that this requires the latest nightly to work, as I redid the semantics of Dispatch.group.
Fairly minimal, but something to start with. Does that help?
Hi Jordan, On Feb 12, 2010, at 11:12 AM, Jordan K. Hubbard wrote:
That's definitely a good start, yes, though I must confess to not quite understanding the dispatch.fork example?
All Dispatch.fork does is allow you to access the return value of the block (whenever it is available), either synchronously or asynchronously. In other words, I'm trying to make Futures duck-type to Ruby threads, with allow a return 'value'. I've annotated the file with return values, which may make things a bit clearer.
http://svn.macosforge.org/repository/ruby/MacRuby/trunk/sample-macruby/Scrip...
Perhaps part of the problem is that I'm using the same method for return both synchronously (direct) and asynchronously (continuation passing style). It leads to a more concise API, and a fairly powerful pattern once you are used to it, but perhaps that is overly clever. What do you think? I'm also working on a full-fleged article that uses actual, y'know, sentences, which may help once it is done... -- Ernie P.
participants (3)
-
Ernest N. Prabhakar, Ph.D.
-
Jordan K. Hubbard
-
Rich Morin