@Ernie, Thank you for the links. Actually I'm have read MRI 1.9's code base before, so I was able to find quite a few "entry points" that I needed to get up to speed with MacRuby. The problem is I'm not familiar with the Objective-C runtime, and that's big blocker for me, since MacRuby is tightly integrated with it. @Vincent
There's currently no documentation on the inner workings of MacRuby. Any help is welcome ;-) Most knowledge is in the code or Laurent's head (and a bit in other commiters' head ;-)). Feel free to ask questions on the ML, though answers may take some time.
Thank you very much for the reply. I'll see if I can understand enough of what's going on, before I can make myself useful :-)
The thing I find very confusing is that in vm.cpp, RoxorCore::RoxorCore(), there's a call to ee->DisableLazyCompilation(). This disables LLVM JIT's lazy compilation.
Yes I added it to be sure that LLVM does not use its lazy compilation mechanism (that is know not to work correctly in multiple threads). It's not needed in MacRuby because we do it our own way.
I saw the same changes have been applied to Unladen Swallow on LLVMdev. Looks like it's common for language implementations to build their own lazy mechanisms when using LLVM for its JIT.
In JIT mode, the conversion of Ruby code to LLVM code (or more strictly speaking LLVM C++ objects) is done as soon as a Ruby script is loaded. But LLVM's optimization phase and native code generation of a function is only done when this function is called.
The place where it's done is hard to find though: it's done using a functionality of the Objective-C runtime: we define the "resolveClassMethod:" and "resolveInstanceMethod:" on NSObject (in vm.cpp, in Init_PreVM) http://developer.apple.com/mac/library/documentation/cocoa/reference/Foundat...: There are also a few other cases where we compile a method even if it's not called, for instance when a method is aliased or a module included.
So that's where...yes, I got it!
For blocks it's done in dispatcher.cpp, in rb_vm_prepare_block. That's a function called just before any block is given to a function.
In AOT mode, of course, we always compile and optimize everything (blocks and methods) beforehand.
Just to make sure I got it right: MacRuby's VM is called Roxor. Currently in the code in trunk doesn't use any interpreter mode; Ruby code is always compiled before is can be run. Compilation can be done in either JIT or AOT mode. (I thought there was an experimental branch that actually used LLVM's interpreter to speed up eval, what happened to that?) JIT mode is the default mode. A translation unit is a Ruby source file. As a Ruby source file gets loaded, it's first parsed by a modified parser taken from MRI 1.9, and turned into NODE*, pretty much the same as the original Ruby AST. Then, RoxorCompiler will compile AST into LLVM IR, which is kept and JITted later on demand. Because in MacRuby all Ruby classes inherit from NSObject, MacRuby uses mechanism from the Objective-C runtime to drive the lazy compilation behavior: original implementations of NSObject's resolveClassMethod: and resolveInstanceMethod: are replaced with MacRuby's own ones, which act as a hook and call the JIT when method implementations are being resolved. For normal Ruby methods, resolveClassMethod_imp() and resolveInstanceMethod_imp() both call RoxorCore::resolve_method(), which calls RoxorCore::resolve_methods(), which calls RoxorCore::resolve_method(), and that'll call RoxorCore::compile() which compiles LLVM IR into native code. In AOT mode, Ruby code gets parsed in Ruby AST, compiled into LLVM IR by RoxorAOTCompiler, and then the LLVM bitcode is dumped to a temporary file. The bitcode is then translated into assembly by llc, then assembled by invoking gcc, and then optionally linked with other stuff. I'll continue reading the source for now... Thank you all, Raven
Great summary -- can someone validate and post on the MacRuby site, for future reference? Also, might be helpful to include links to the appropriates files so it is easier to cross-check. -- Ernie P. On Dec 8, 2009, at 5:49 AM, ravenex wrote: Just to make sure I got it right: MacRuby's VM is called Roxor. Currently in the code in trunk doesn't use any interpreter mode; Ruby code is always compiled before is can be run. Compilation can be done in either JIT or AOT mode. (I thought there was an experimental branch that actually used LLVM's interpreter to speed up eval, what happened to that?) JIT mode is the default mode. A translation unit is a Ruby source file. As a Ruby source file gets loaded, it's first parsed by a modified parser taken from MRI 1.9, and turned into NODE*, pretty much the same as the original Ruby AST. Then, RoxorCompiler will compile AST into LLVM IR, which is kept and JITted later on demand. Because in MacRuby all Ruby classes inherit from NSObject, MacRuby uses mechanism from the Objective-C runtime to drive the lazy compilation behavior: original implementations of NSObject's resolveClassMethod: and resolveInstanceMethod: are replaced with MacRuby's own ones, which act as a hook and call the JIT when method implementations are being resolved. For normal Ruby methods, resolveClassMethod_imp() and resolveInstanceMethod_imp() both call RoxorCore::resolve_method(), which calls RoxorCore::resolve_methods(), which calls RoxorCore::resolve_method(), and that'll call RoxorCore::compile() which compiles LLVM IR into native code. In AOT mode, Ruby code gets parsed in Ruby AST, compiled into LLVM IR by RoxorAOTCompiler, and then the LLVM bitcode is dumped to a temporary file. The bitcode is then translated into assembly by llc, then assembled by invoking gcc, and then optionally linked with other stuff. I'll continue reading the source for now...
Sorry for the late reply.
MacRuby's VM is called Roxor. Currently in the code in trunk doesn't use any interpreter mode; Ruby code is always compiled before is can be run.
Yes. There are plans to add a simple interpreter to be faster with simple evals. It would interpret directly the Ruby parse tree for a very restricted number of nodes. And we would fallback on the compiler for complicated code. But I don't think it's high on Laurent's priority list.
Compilation can be done in either JIT or AOT mode.
Yes
(I thought there was an experimental branch that actually used LLVM's interpreter to speed up eval, what happened to that?)
Laurent did indeed have a version at one time using LLVM's interpreter but it has was discarded. I don't remember the exact reason, but it was probably multiple limitations in LLVM's interpreter.
JIT mode is the default mode. A translation unit is a Ruby source file. As a Ruby source file gets loaded, it's first parsed by a modified parser taken from MRI 1.9, and turned into NODE*, pretty much the same as the original Ruby AST.
Yes, we try to modify the parser as little as possible. The main modifications are for the GC. In the grammar itself, the only modification is to allow declaration of methods with keyword arguments def foo(arg1, bar: arg2); end. There may also have been changes on MRI's parser done after we forked it that have not been put back in MacRuby yet.
Then, RoxorCompiler will compile AST into LLVM IR, which is kept and JITted later on demand. Because in MacRuby all Ruby classes inherit from NSObject, MacRuby uses mechanism from the Objective-C runtime to drive the lazy compilation behavior: original implementations of NSObject's resolveClassMethod: and resolveInstanceMethod: are replaced with MacRuby's own ones, which act as a hook and call the JIT when method implementations are being resolved. For normal Ruby methods, resolveClassMethod_imp() and resolveInstanceMethod_imp() both call RoxorCore::resolve_method(), which calls RoxorCore::resolve_methods(), which calls RoxorCore::resolve_method(), and that'll call RoxorCore::compile() which compiles LLVM IR into native code.
In AOT mode, Ruby code gets parsed in Ruby AST, compiled into LLVM IR by RoxorAOTCompiler, and then the LLVM bitcode is dumped to a temporary file. The bitcode is then translated into assembly by llc, then assembled by invoking gcc, and then optionally linked with other stuff.
That sounds right :-) We may one day do what llc does ourselves to compile a bit faster and reduce the size of the distribution (as LLVM is pretty big and linked statically to both macruby and llc), but well it's a very very low priority task.
participants (3)
-
Ernest N. Prabhakar, Ph.D.
-
ravenex
-
Vincent Isambart