In the comments of Charlie's latest blog post, someone showed their benchmarks of the 0.5 branch running tak(). http://blog.headius.com/2009/04/how-jruby-makes-ruby-fast.html#comments I'd like to do the same but rake isn't giving me a macruby executable. How do I go beyond miniruby and get a ruby capable of running Charlie's bench_tak.rb?
In the comments of Charlie's latest blog post, someone showed their benchmarks of the 0.5 branch running tak(). http://blog.headius.com/2009/04/how-jruby-makes-ruby-fast.html#comments I'd like to do the same but rake isn't giving me a macruby executable. How do I go beyond miniruby and get a ruby capable of running Charlie's bench_tak.rb?
You can't get a macruby executable with the current experimental branch. And anyway you can run the tak benchmark with only miniruby (./miniruby -I./lib bench_tak.rb 5) But anyway it seems no one seems to see the only interesting point of Charles' posts: mini benchmarks are not a good indicator of the speed of an implementation. I've never seen any real-life Ruby code that does heavy computations and recursion like tak or fibonacci... Such mini-benchmarks are mainly a tool for the implementers themselves to see if some code modification did something good in an area, but they generally do not mirror real speed... And for real speed the new MacRuby VM is still young - it does not even run IRB yet anyway (well I would not be surprised if it does next week) - and lots of optimization are still to be done. Fixnum computations however is already well optimized and is probably not an area where you're going to see many changes in speed.
On Apr 4, 2009, at 8:18 PM, Vincent Isambart wrote:
In the comments of Charlie's latest blog post, someone showed their benchmarks of the 0.5 branch running tak(). http://blog.headius.com/2009/04/how-jruby-makes-ruby-fast.html#comments I'd like to do the same but rake isn't giving me a macruby executable. How do I go beyond miniruby and get a ruby capable of running Charlie's bench_tak.rb?
You can't get a macruby executable with the current experimental branch. And anyway you can run the tak benchmark with only miniruby (./miniruby -I./lib bench_tak.rb 5)
To build what you can with experimental (miniruby) you should read the README file which shows how to build/install LLVM and then build macruby In there is says (after LLVM) to just do: rake To build. What you should instead do is: rake miniruby This will build miniruby. miniruby can be executed with: ,/miniruby -e 'p 1234' miniruby behaves like the ruby executable itself. You should be able to run tak.rb then.
But anyway it seems no one seems to see the only interesting point of Charles' posts: mini benchmarks are not a good indicator of the speed of an implementation. I've never seen any real-life Ruby code that
I don't necessarily agree that microbenchmarks are _NOT_ good indicators. I think a specific microbenchmark is not, but the point of these micro- benchmarks are to judge the speed of specific runtime units. If you run one really fast it is not a good indicator of general performance, but if you run all of them quickly, its a good indication that the general performance will exceed the performance of the runtimes you are comparing yourself to.
does heavy computations and recursion like tak or fibonacci... Such mini-benchmarks are mainly a tool for the implementers themselves to see if some code modification did something good in an area, but they generally do not mirror real speed...
And for real speed the new MacRuby VM is still young - it does not even run IRB yet anyway (well I would not be surprised if it does next week) - and lots of optimization are still to be done. Fixnum computations however is already well optimized and is probably not an area where you're going to see many changes in speed. _______________________________________________ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Richard Kilmer wrote:
I don't necessarily agree that microbenchmarks are _NOT_ good indicators. I think a specific microbenchmark is not, but the point of these micro- benchmarks are to judge the speed of specific runtime units. If you run one really fast it is not a good indicator of general performance, but if you run all of them quickly, its a good indication that the general performance will exceed the performance of the runtimes you are comparing yourself to.
Microbenchmarks are a good indicator of performance for the range of features that those microbenchmarks test. If all the microbenchmarks you examine are testing low-level VM operations and fixnum math, they're only good indicators of the performance of low-level VM operations and fixnum math. They may or may not be indicators of general performance, and simply running more of sthen does not improve the quality of the suite. Key features like core class performance, object/memory/GC performance, IO/network performance, and so on are often absent from such benchmarks, and have at least as much bearing on general application performance as low-level VM operations and fixnum math. - Charlie
On Apr 4, 2009, at 5:18 PM, Vincent Isambart wrote:
In the comments of Charlie's latest blog post, someone showed their benchmarks of the 0.5 branch running tak(). http://blog.headius.com/2009/04/how-jruby-makes-ruby-fast.html#comments I'd like to do the same but rake isn't giving me a macruby executable. How do I go beyond miniruby and get a ruby capable of running Charlie's bench_tak.rb?
You can't get a macruby executable with the current experimental branch. And anyway you can run the tak benchmark with only miniruby (./miniruby -I./lib bench_tak.rb 5)
But anyway it seems no one seems to see the only interesting point of Charles' posts: mini benchmarks are not a good indicator of the speed of an implementation. I've never seen any real-life Ruby code that does heavy computations and recursion like tak or fibonacci... Such mini-benchmarks are mainly a tool for the implementers themselves to see if some code modification did something good in an area, but they generally do not mirror real speed...
While I agree that micro-benchmarks do not represent the reality, generally they are written to cover a real-world scenario where the testing implementation suffers. Now, some micro-benchmarks written against implementation X might sometimes be meaningless in implementation Y. For instance, the optimizations that are (and will be) implemented in the experimental branch are not random but were selected after having profiled a big MacRuby/Cocoa application and found many areas where we performed badly. I assume that other implementations are using similar techniques.
And for real speed the new MacRuby VM is still young - it does not even run IRB yet anyway (well I would not be surprised if it does next week) - and lots of optimization are still to be done. Fixnum computations however is already well optimized and is probably not an area where you're going to see many changes in speed.
I think we can improve fixnum arithmetic a little bit (I would estimate 5-10%) once we enable secondary compilations, but this isn't clearly a top priority at this point. Laurent
Laurent Sansonetti wrote:
For instance, the optimizations that are (and will be) implemented in the experimental branch are not random but were selected after having profiled a big MacRuby/Cocoa application and found many areas where we performed badly. I assume that other implementations are using similar techniques.
The interesting thing about many of these microbenchmarks is that a substantial part of MacRuby's performance seems to come from one key optimization: recursive calls. MacRuby (experimental) fib(30) with fcalls (fib(...): 0.060000 0.000000 0.060000 ( 0.053868) 0.050000 0.000000 0.050000 ( 0.053773) 0.050000 0.000000 0.050000 ( 0.055541) 0.050000 0.000000 0.050000 ( 0.049992) 0.060000 0.000000 0.060000 ( 0.054645) MacRuby (experimental) fib(30) with calls (self.fib(...)): 0.180000 0.000000 0.180000 ( 0.182371) 0.180000 0.000000 0.180000 ( 0.178691) 0.170000 0.000000 0.170000 ( 0.177174) 0.170000 0.000000 0.170000 ( 0.179554) 0.180000 0.000000 0.180000 ( 0.181848) The latter numbers are roughly the same as JRuby performance without recursion optimization (with or without fcall). The recursion opt is actually pretty clever...I experimented with it a few weeks back (achieving similar results, http://gist.github.com/67329) and backed off temporarily to work on other things. I also wasn't sure how common it would be in real code, nor was I sure it could be done safely while maintaining all Ruby semantics. Maybe we'll be able to figure that out together. :) - Charlie
On Apr 4, 2009, at 7:57 PM, Charles Oliver Nutter wrote:
Laurent Sansonetti wrote:
For instance, the optimizations that are (and will be) implemented in the experimental branch are not random but were selected after having profiled a big MacRuby/Cocoa application and found many areas where we performed badly. I assume that other implementations are using similar techniques.
The interesting thing about many of these microbenchmarks is that a substantial part of MacRuby's performance seems to come from one key optimization: recursive calls.
If you're talking about fib or tak, then yes (with of course fast fixnum arithmetic). There are still a few thing that we can do to improve these recursive benchmarks (also I realized I forgot to re- enable the tail-call optimization :-)). Last time I benchmarked fib.rb we were able 2 times slower than the equivalent C version (compiled with -O0 though) and this included the startup time + parsing + compilation. Now there are other areas that are (and will be further) optimized, such as blocks, lvars/ivars, method dispatch, etc. Laurent
Laurent Sansonetti wrote:
On Apr 4, 2009, at 7:57 PM, Charles Oliver Nutter wrote:
The interesting thing about many of these microbenchmarks is that a substantial part of MacRuby's performance seems to come from one key optimization: recursive calls.
If you're talking about fib or tak, then yes (with of course fast fixnum arithmetic). There are still a few thing that we can do to improve these recursive benchmarks (also I realized I forgot to re-enable the tail-call optimization :-)).
Last time I benchmarked fib.rb we were able 2 times slower than the equivalent C version (compiled with -O0 though) and this included the startup time + parsing + compilation.
And I have been able to get JRuby's performance within a few times of Java performance, provided Java used boxed numerics. I have not put those optimizations in place because I could not be sure they were safe and I could not get them to pass tests/specs. In the future, that may change.
Now there are other areas that are (and will be further) optimized, such as blocks, lvars/ivars, method dispatch, etc.
I think the primary point I hoped to make was that we would both be well-served by exploring these optimizations together and determining which are safe and which are not. Most of the optimizations you will be able to do on LLVM will be applicable to JRuby on the JVM, and vice-versa. We could each go through the pain alone, or we could benefit from each other's efforts and discoveries. - Charlie
participants (5)
-
Charles Oliver Nutter
-
Laurent Sansonetti
-
Mike Moore
-
Richard Kilmer
-
Vincent Isambart