Running X11 with Fast User Switching

X11 wont' let me fire it up if I have left it running and switched to another user. What can I do to change this? This is what it spits into console: XFree86 Version 4.4.0 / X Window System (protocol Version 11, revision 0, vendor release 6600) [DRI] screen 0 installation complete Screen 0 added: 1280x800 @ (0,0) Could not init font path element /usr/X11R6/lib/X11/fonts/CID/, removing from list! Xlib: connection to ":0.0" refused by server Xlib: No protocol specified xterm Xt error: Can't open display: :0.0 Xlib: connection to ":0.0" refused by server Xlib: No protocol specified can't open default display Quitting XDarwin...