I'm working on a feasibility to start xquartz from a java process with JNI. My goal would be to keep the X server in the same process as my java application so this way I can manipulate the X windows itself (mainly reparent, resize and show/hide). To able to do the window manipulation I have to keep X and java in the same process.
That's not really possible... I think what you want to do is write a window manager in java (in which case you don't actually need the server to be in the same process).
Actually, CrossOver seems to do this, though not in Java... I'm pretty sure they have a heavily customised X server though, they probably started with X11.app and built their interfaces around it (don't take my word for it, I'm just guessing).
Maybe if you really want this you can take the same approach. Instead of embedding X11 in your app, customize X11.app to load your java code. It's probably quite a bit of work though...
It's not that much of a mystery. They use quartz-wm, just like X11.app does ;) Like I said, what you're looking for is the functionality of a window manager, not the X server itself. --Jeremy