Hello! I'd like to make a port for MIT Scheme, an implementation of the Scheme programming language. The port is somewhat complicated, though, because of a bootstrapping process. However, I think that I have found a way to make the bootstrapping process work without introducing circular dependencies, which requires only a small change to MacPorts' dependency specifications. MIT Scheme is a large program written primarily in Scheme, and based on a native-code compiler. This would mean that building it is impossible without an older version of itself, but its compiler also has a C back end, and the next snapshot to be released very shortly will support the construction of distributions with pre-generated C code by which the whole system can be bootstrapped. This means that it is easy to write a port that just fetches a tarball with the generated C code and bootstraps the whole system from that. The trouble with this is that it is very slow, and only the C back end can be built from generated C code. I'd like to have another port for the native back end, so I'll call the port with the C back end `mit-scheme-c', and the port with the native back end `mit-scheme-native'. Building Scheme from scratch is slow (requiring about an hour or two on my 2 GHz MacBook), and building either back end with the other back end is also slow (C -> native or native -> C). If the user already has an older version of either back end, then upgrading to a newer version can run much faster (about ten or twenty minutes for me). In an ideal world, I could have two portfiles, something like <http://people.csail.mit.edu/riastradh/tmp/liarc-Portfile> and <http://people.csail.mit.edu/riastradh/tmp/native-Portfile>: mit-scheme-c has three variants, all of which are mutually exclusive, and which I'll write with a `+' prefix just to visually distinguish them from port names: +from-scratch requires nothing but a C compiler and an operating system, fetches a tarball containing pre-generated C files, and bootstraps the whole system from that. Slow. +from-c requires mit-scheme-c (usually an older version), fetches a pristine source tarball, and just builds the system, without any extra intermediate bootstrapping stages. Decently fast. +from-native requires mit-scheme-native, fetches a pristine source tarball, and bootstraps the system with an intermediate cross-compiler. (This one isn't actually very useful; it does more work than +from-scratch, because it has to prepare what was pre-generated for +from-scratch.) mit-scheme-native has two variants, both of which depend on an existing installation: +from-c requires mit-scheme-c, fetches a pristine source tarball, and bootstraps a native compiler from that. Slow, but this is the only way to get a native system to begin with. +from-native requires mit-scheme-native (usually an older version), fetches a pristine source tarball, and builds the system without any extra intermediate bootstrapping stages. Fast. Unfortunately, `mit-scheme-c +from-c' and `mit-scheme-native +from-native' don't work because of the obvious bootstrapping problem. I discussed the issue with Juan Manuel Palacios and Eric Hall on IRC, and was informed that the dependency engine would require an inordinate amount of work before this kind of nearly circular dependency would work at all. However, it occurred to me that we could somewhat dangerously work around the *port* dependency by rendering it into a *binary* dependency -- that is, the +from-c variants depend on the existence of a binary called `mit-scheme-c', and the +from-native variants depend on the existence of a binary called `mit-scheme-native'. MacPorts do have a way to specify a dependency on a binary file by some name -- `depends_* bin:foo:bar' --, but it requires a port to be associated with the file. What I need for MIT Scheme is *almost* this: if the file isn't there, I want MacPorts not to try to build the port associated with it, but just to give up; users shouldn't use the +from-c or +from-native variants if they aren't available. What I propose, then, is to allow a dependency of the form `bin:<filename>', without an associated port (or perhaps with `I-know-what-I-am-doing' or `just-complain-to-the-user-if-unavailable' instead of a port). Although it is slightly dangerous to do this, it's a very small change that lets us work around what would otherwise appear to be circular dependencies, and it would mean that upgrading the MIT Scheme port could take fifteen to thirty minutes, rather than anywhere from two to four hours. Thoughts? Does this sound reasonable? Is there another way to go about this that I hadn't thought of?
On Jun 22, 2007, at 12:12 PM, Taylor R Campbell wrote:
+from-scratch requires nothing but a C compiler and an operating system, fetches a tarball containing pre-generated C files, and bootstraps the whole system from that. Slow.
I realize the pain involved in going this route, but it seems like we're trading mere compilation time for engineering time, and the latter is almost always more expensive. I do understand your desire to not heat up your lap, but speaking from just my personal viewpoint, I'd far rather have a nice, generic port which took an hour to compile than have to figure out which of n ports to use and also deal with the fragility of these ports over what could be multiple OS releases and variable availability of suitable bootstrap compilers for whatever architecture / word size (ppc, ppc64, x86_32, x86_64, and so on) I might be trying to target. An hour is a fine amount of time to go warm up the latte maker and make myself a nice latte to drink while I wait. :-) - Jordan P.S. An hour today, sure, but probably only 5 minutes on the new 2048 core Mac Pro we haven't released yet. P.P.S. Yes, of course I'm joking. For now, anyway. :-)
Date: Fri, 22 Jun 2007 16:14:33 -0700 From: "Jordan K. Hubbard" <jkh@brierdr.com> I realize the pain involved in going this route, but it seems like we're trading mere compilation time for engineering time, and the latter is almost always more expensive. What engineering time is involved, aside from implementing the small change I proposed (which I imagine would be very easy), assuming it doesn't seem a bad idea? I do understand your desire to not heat up your lap, but speaking from just my personal viewpoint, I'd far rather have a nice, generic port which took an hour to compile than have to figure out which of n ports to use and also deal with the fragility of these ports over what could be multiple OS releases and variable availability of suitable bootstrap compilers for whatever architecture / word size (ppc, ppc64, x86_32, x86_64, and so on) I might be trying to target. Yes, I'd like to avoid this kind of complexity too -- which is why I meant exactly what I said in my previous email. If there is an existing native build on the system, then I want the native port to use that; otherwise, everything has to go through the C build. I'd like to avoid any cross-compilation complexity, except from C to native or vice versa, so I have no intention of having separate `mit-scheme-x86_32', `mit-scheme-x86_64', `mit-scheme-ppc', &c., ports. Just `mit-scheme-c' and `mit-scheme-native'.
Actually, it occurs to me that it's not really necessary to declare a dependency -- that it would suffice for the variant to have a command that checks for the existence of an executable by some name, and to refuse to continue if this command fails. There was some recent discussion about a `ui_fatal' command by which to fail, although I understand that it was simply a proposal not yet implemented. Does this sound plausible, however?
On Jun 25, 2007, at 09:46, Taylor R Campbell wrote:
Actually, it occurs to me that it's not really necessary to declare a dependency -- that it would suffice for the variant to have a command that checks for the existence of an executable by some name, and to refuse to continue if this command fails.
I haven't really followed why all this complexity with the multiple scheme ports is required -- I realize you tried to spell it out in your first email but it was a bit much for me at the moment. But assuming it really is required, then yes, this is the method that occurred to me as well: just test for the file's existence in, say, the pre-fetch phase, and fail if not found. I don't, however, know the portfile syntax for checking for a file's existence.
There was some recent discussion about a `ui_fatal' command by which to fail, although I understand that it was simply a proposal not yet implemented. Does this sound plausible, however?
Not really needed, though, either. Just do ui_error "the message" followed by exit 1. And, again, do this in the pre-fetch phase.
On Jun 26, 2007, at 11:39 PM, Ryan Schmidt wrote:
On Jun 25, 2007, at 09:46, Taylor R Campbell wrote:
Actually, it occurs to me that it's not really necessary to declare a dependency -- that it would suffice for the variant to have a command that checks for the existence of an executable by some name, and to refuse to continue if this command fails.
I haven't really followed why all this complexity with the multiple scheme ports is required -- I realize you tried to spell it out in your first email but it was a bit much for me at the moment.
But assuming it really is required, then yes, this is the method that occurred to me as well: just test for the file's existence in, say, the pre-fetch phase, and fail if not found. I don't, however, know the portfile syntax for checking for a file's existence.
There was some recent discussion about a `ui_fatal' command by which to fail, although I understand that it was simply a proposal not yet implemented. Does this sound plausible, however?
Not really needed, though, either. Just do ui_error "the message" followed by exit 1. And, again, do this in the pre-fetch phase.
Calling "exit" from a Portfile will cause any front-end using the dports API to exit, no? That seems bad form. -landonf
On Jun 29, 2007, at 13:33, Landon Fuller wrote:
On Jun 26, 2007, at 11:39 PM, Ryan Schmidt wrote:
On Jun 25, 2007, at 09:46, Taylor R Campbell wrote:
There was some recent discussion about a `ui_fatal' command by which to fail, although I understand that it was simply a proposal not yet implemented. Does this sound plausible, however?
Not really needed, though, either. Just do ui_error "the message" followed by exit 1. And, again, do this in the pre-fetch phase.
Calling "exit" from a Portfile will cause any front-end using the dports API to exit, no? That seems bad form.
I was under the impression that doing this within the pre-fetch phase was acceptable. I thought we discussed this on the list earlier. Many ports do this: xorg, wine, mozilla, sockstat, ipcs, gauche-gtk, mysql5-devel, TeXShop. If that's not ok, then we really do need someone to implement ui_fatal. But what would that do differently? There are some ports that still do it directly in a platform selector: xloops, nestedsums, cln, GiNaC, hugs98, ghc, klisp, kdelibs3. I can see how that's indeed wrong and needs to be changed.
On Jun 29, 2007, at 11:22 PM, Ryan Schmidt wrote:
On Jun 29, 2007, at 13:33, Landon Fuller wrote:
On Jun 26, 2007, at 11:39 PM, Ryan Schmidt wrote:
On Jun 25, 2007, at 09:46, Taylor R Campbell wrote:
There was some recent discussion about a `ui_fatal' command by which to fail, although I understand that it was simply a proposal not yet implemented. Does this sound plausible, however?
Not really needed, though, either. Just do ui_error "the message" followed by exit 1. And, again, do this in the pre-fetch phase.
Calling "exit" from a Portfile will cause any front-end using the dports API to exit, no? That seems bad form.
I was under the impression that doing this within the pre-fetch phase was acceptable. I thought we discussed this on the list earlier. Many ports do this: xorg, wine, mozilla, sockstat, ipcs, gauche-gtk, mysql5-devel, TeXShop.
It will still cause the entire front-end to exit, rather than throwing an error in the port sub-interpreter. It's the equivalent of a library calling abort() instead of returning an error code.
If that's not ok, then we really do need someone to implement ui_fatal. But what would that do differently?
return -code error "An error, here" It's still not perfect, in that it's simply not declarative -- how do I determine whether a port can run on my system ? Execute the fetch phase. However, It's still better than exit.
There are some ports that still do it directly in a platform selector: xloops, nestedsums, cln, GiNaC, hugs98, ghc, klisp, kdelibs3. I can see how that's indeed wrong and needs to be changed.
I'm surprised at the extent of the usage. -landonf
On Jun 30, 2007, at 01:31, Landon Fuller wrote:
On Jun 29, 2007, at 11:22 PM, Ryan Schmidt wrote:
On Jun 29, 2007, at 13:33, Landon Fuller wrote:
On Jun 26, 2007, at 11:39 PM, Ryan Schmidt wrote:
On Jun 25, 2007, at 09:46, Taylor R Campbell wrote:
There was some recent discussion about a `ui_fatal' command by which to fail, although I understand that it was simply a proposal not yet implemented. Does this sound plausible, however?
Not really needed, though, either. Just do ui_error "the message" followed by exit 1. And, again, do this in the pre- fetch phase.
Calling "exit" from a Portfile will cause any front-end using the dports API to exit, no? That seems bad form.
I was under the impression that doing this within the pre-fetch phase was acceptable. I thought we discussed this on the list earlier. Many ports do this: xorg, wine, mozilla, sockstat, ipcs, gauche-gtk, mysql5-devel, TeXShop.
It will still cause the entire front-end to exit, rather than throwing an error in the port sub-interpreter. It's the equivalent of a library calling abort() instead of returning an error code.
Sure... but I don't know the MacPorts base code well enough (or at all) to know what the implications of that are.
If that's not ok, then we really do need someone to implement ui_fatal. But what would that do differently?
return -code error "An error, here"
It's still not perfect, in that it's simply not declarative -- how do I determine whether a port can run on my system ? Execute the fetch phase. However, It's still better than exit.
Oh yes, now I remember seeing that "return" syntax used a couple places (libiconv, py-psyco, ghc-devel, sshfs, libfuse, fusefs) and not really understanding it. But so we should be replacing things like pre-fetch { ui_error "foo bar" exit 1 } with pre-fetch { return -code 1 "foo bar" } Yes? This will work the same? So long as you say that's what we should be doing, it shouldn't be too much trouble for me (or another volunteer?) to go through the ports listed in this email and fix things.
There are some ports that still do it directly in a platform selector: xloops, nestedsums, cln, GiNaC, hugs98, ghc, klisp, kdelibs3. I can see how that's indeed wrong and needs to be changed.
I'm surprised at the extent of the usage.
There are lots more ports that call exit but for other reasons so I didn't list them in the earlier mail. For example, kdelibs3 says: pre-configure { if {[file exists ${prefix}/bin/cups-config]} { ui_msg "port:cups-headers may prevent building this port." ui_msg "Please uninstall (or deactivate) cups-headers and restart the build." exit 1 } } To convert such multiline messages to return -code 1 "foo" syntax I guess we can use a "\n" to do the newline?
participants (4)
-
Jordan K. Hubbard
-
Landon Fuller
-
Ryan Schmidt
-
Taylor R Campbell