I'm quite distressed by the concept of too much work (and too much ugly code) going into building +universal variants of ports. I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Let's be reasonable: the case for universal builds is fringe. We don't have easy ways to transport binary ports between systems, and it's not in general a safe thing to do. I'd much rather see: - Smaller, cleaner, more maintainable portfiles and no universal. - Work on binary packages and a build/distribution system that could deliver the pre-built port for a given architecture. Let's not see more ugliness like the universal variant in the OpenSSL port, please! (Feel free to chime in on this issue, all who care...) James
On Mar 3, 2007, at 12:49 PM, James Berry wrote:
I'm quite distressed by the concept of too much work (and too much ugly code) going into building +universal variants of ports.
I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Let's be reasonable: the case for universal builds is fringe. We don't have easy ways to transport binary ports between systems, and it's not in general a safe thing to do.
I'd much rather see:
- Smaller, cleaner, more maintainable portfiles and no universal.
- Work on binary packages and a build/distribution system that could deliver the pre-built port for a given architecture.
Let's not see more ugliness like the universal variant in the OpenSSL port, please!
(Feel free to chime in on this issue, all who care...)
I agree. It's not that universal builds are a bad idea; it's that supporting universal software requires serious bastardization of Portfiles in most cases. Assuming we implement some sort of binary distribution, I'd personally much rather use the approach that fkr and shantonu used for supporting Darwin x86 and PowerPC -- build the software on each system, natively, and then join the resultant RPMs into a single fat RPM. The example jberry notes is OpenSSL's universal-ized portfile. I don't mean to pick on Pipping -- I've personally implemented a very similar approach for building OpenVPN universally. It's pretty much the only way to do it without drastically changing the software's build system and source, but it's blindingly painful. It's not unique to OpenSSL, or OpenVPN -- most software is going to have the same issues: +variant universal { + + post-configure { + cd ${worksrcpath} + # prepare building for ppc + if [variant_isset darwin_i386] { + reinplace "s|PLATFORM=darwin-i386-cc|PLATFORM=darwin-ppc- cc|g" Makefile + reinplace "s|DL_ENDIAN|DB_ENDIAN|g" Makefile + } + reinplace "s|-O3 -DB_ENDIAN$|-O3 -DB_ENDIAN -arch ppc - isysroot /Developer/SDKs/MacOSX10.4u.sdk|" Makefile + reinplace "s|^LDFLAGS=.*|LDFLAGS=-arch ppc|g" Makefile.shared + } + + build { + cd ${worksrcpath} + # build for ppc + system [command build] + + # determine which files will need to be lipo'ed together + set lList {} + foreach s {0.9.8.dylib a} { + foreach n {crypto ssl} { + lappend lList lib${n}.${s} + } + } + set eList {} + foreach f [glob engines/*.so] { + lappend eList ${f} + } + set bList apps/openssl + set tList {} + foreach f [glob test/*test] { + lappend tList ${f} + } + lappend tList test/sha256t + lappend tList test/sha512t + + # define a backup procedure to a temporary location + proc backup {bakPath} { + xinstall -d ${bakPath} + foreach a {l e b t} b {. engines apps test} { + upvar 1 [set a]List [set a]List + xinstall -d ${bakPath}/$b + foreach n [set [set a]List] { + xinstall ${n} ${bakPath}/${b} + } + } + } + # backup the output of the first run (ppc) + set ppcPath ${workpath}/ppc + backup ${ppcPath} + + # cleanup the worksrcdir + system "make clean" + foreach f [glob lib*.0.9.8.dylib] { + delete ${f} + } + + # prepare building for i386 + reinplace "s|darwin-ppc-cc|darwin-i386-cc|g" ${worksrcpath}/ Makefile + reinplace "s|DB_ENDIAN|DL_ENDIAN|g" ${worksrcpath}/Makefile + reinplace "s|-arch ppc|-arch i386|g" ${worksrcpath}/Makefile + reinplace "s|-arch ppc|-arch i386|g" Makefile.shared + + # build for i386 + system [command build] + + # backup the output of the first run (ppc) + set i386Path ${workpath}/i386 + backup ${i386Path} + + # run lipo on the output of both runs + foreach n {l e b t} { + foreach m [set [set n]List] { + delete ${m} + system "lipo \ + -arch i386 ${i386Path}/${m} \ + -arch ppc ${ppcPath}/${m} \ + -create -output ${m}" + } + } + + # make sure installing won't rebuild + reinplace "s|install: all |install: |g" Makefile + } + + # make sure we don't build a third time + post-build {} +} +
On Mar 3, 2007, at 12:49 PM, James Berry wrote:
I'm quite distressed by the concept of too much work (and too much ugly code) going into building +universal variants of ports.
I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Let's be reasonable: the case for universal builds is fringe. We don't have easy ways to transport binary ports between systems, and it's not in general a safe thing to do.
I understand what you're saying with respect to "lack of easy transport", but I can't agree that "the case for universal builds is fringe.". Perhaps you're talking about "fringe" purely with respect to MacPorts itself, but let's zoom out for a moment and look at the bigger picture again. First, universal binaries are here to stay and, once you start running Leopard, you will begin to look at anything which is NOT universal on your system with annoyance and disdain. The reason for this is something I've already covered - the Macintosh community is becoming increasingly heterogeneous in nature as more and more Intel machines join the PPC installed base, and the best thing we can do for this community ("we" being both Apple and its associated developers) is make it increasingly irrelevant just which architecture they're using, be it PPC, PPC64, x86_32 or x86_64. Stuff they download off the net or run off of a CD, network or firewire volume should Just Work™ and work in a highly performant way - Rosetta is really a technology of last resort so please don't imagine that "ppc everywhere" is a good idea. We're already seeing the user community put a lot of pressure on folks like Adobe to get their stuff universal since things like plug- ins and extensions make a non-Universal world really a nightmare (particularly when you need a PPC-only plug-in and an x86-only plug- in running in a web browser - you end up having to run two browsers). So, bigger picture, Universal is the way to go and it's the message we're sending to everyone, both because it's the right thing to do from an advanced-technology perspective and because we want to present a united front on this issue to the commercial ISVs who might otherwise be tempted to let this one slide for awhile longer and continue to impact the overall Macintosh user experience in negative ways. So, enter MacPorts. Does MacPorts absolutely need to do this or nobody will ever use it again? Of course not. There is, however, still a lot of confusion and ignorance in the OSS community about how to build things universal (particularly in the presence of configure scripts, which definitely need to evolve). A build-framework like MacPorts has the ability to substantially lead the way here and, by example, demonstrate to literally thousands of open source developers just how to do it. Just as importantly, by grasping the nettle at the build stage, MacPorts is not setting itself up for more work at the packaging stage it (I hope) eventually wants to get to. Nobody is seriously suggesting (I HOPE) having parallel collections of many thousands of packages, one for x86 and one for ppc (and, at some point, one full of experimental 64-bit capable stuff), when it's perfectly possible and encouraged to have just ONE package collection that works for everything. By not figuring out how to build universal, MacPorts would merely be stealing resources from the future in order to be lazy in the present. There would be no net gain and definitely a net loss as the ability to share a single /opt among multiple machines on a network went out the window too. I also think MacPorts will increasingly find itself to be the odd man out if it doesn't go down the universal road. People ARE figuring out how to do it and it's become something of a badge of honor to get your application and frameworks building n-way universal so that your users don't have to think about anything more than which version of your product or project they want. People who are using xcode have an easier time of it - they just click the universal checkbox and, unless they've written something egregious into their code, it just works. I hope you're not saying that the Unix command-line driven community can't possibly build its stuff as well as xcode can since them's fightin' words. :-) Now, if you're saying that the initial efforts to get there have been clunky and inelegant then I'm not going to dispute that since you're probably right. That doesn't mean you throw the baby out with the bathwater, however, it means you figure out how to do a better job of it by learning from your early mistakes! - Jordan
On Mar 3, 2007, at 1:09 PM, Landon Fuller wrote:
I agree. It's not that universal builds are a bad idea; it's that supporting universal software requires serious bastardization of Portfiles in most cases. Assuming we implement some sort of binary distribution, I'd personally much rather use the approach that fkr and shantonu used for supporting Darwin x86 and PowerPC -- build the software on each system, natively, and then join the resultant RPMs into a single fat RPM.
I think this may have gotten lost in all my earlier verbiage, but I also pointed out early in the conversation that there are many many open source projects building universal at Apple without such "bastardization". The Makefile for OpenSSL is just 147 lines, for example, and of that at LEAST 50% has to do strictly with Apple's own requirements (the whole SRCROOT and DSTROOT thing, order files, all the things you probably still have nightmares about from time to time). Perhaps 5 lines in the Makefile are devoted to the fact that it's building universal. The rest is in the "generic machinery" and helper scripts (which, you'll instantly notice, have a lineage dating back to 10.1) like this one: This gets used by the generic CoreOS Makefile machinery in place of ar (1), given a bit of metadata expressed in other parts of the build system. Are lipo(1) tricks like this the best way to do it? Not even saying that. I'm simply saying that it's a fallacy to assume that OpenSSL, OpenVPN and hundreds of other ports are all going to need to grow universal variant rules as long and complex as the one you posted. Just because someone hasn't templated that stuff and stuck it somewhere for every port to use doesn't mean it can't be done, and Apple's own open source projects are living proof of that. I'm furthermore assuming that once a few more universal ports get done and the commonality in the rules starts to shine through the complexity, someone WILL sweep it into a pile and make it port- generic. I can recall a time when darwin^H^H^H^H^Hmacports had a lot of pretty crufty Portfiles which, over time, got substantially less crufty as more primitives were added and consolidation mechanisms like PortGroup got added. Why is everyone being so quick to slam pipping's changes before they even have a chance to go through an evolution or two? I know there are still a few people here who still don't get the whole point of universal builds, or doubt their value to MacPorts, but that's orthogonal to the whole issue of "quality of implementation", something which has to at least be given a chance to improve before coming to any firm conclusions about it. - Jordan
On Mar 3, 2007, at 2:33 PM, Jordan K. Hubbard wrote:
On Mar 3, 2007, at 1:09 PM, Landon Fuller wrote:
I agree. It's not that universal builds are a bad idea; it's that supporting universal software requires serious bastardization of Portfiles in most cases. Assuming we implement some sort of binary distribution, I'd personally much rather use the approach that fkr and shantonu used for supporting Darwin x86 and PowerPC -- build the software on each system, natively, and then join the resultant RPMs into a single fat RPM.
I think this may have gotten lost in all my earlier verbiage, but I also pointed out early in the conversation that there are many many open source projects building universal at Apple without such "bastardization". The Makefile for OpenSSL is just 147 lines, for example, and of that at LEAST 50% has to do strictly with Apple's own requirements (the whole SRCROOT and DSTROOT thing, order files, all the things you probably still have nightmares about from time to time). Perhaps 5 lines in the Makefile are devoted to the fact that it's building universal.
I don't think it's reasonable to compare Apple's static manually managed B&I build system to MacPorts. We could also write shell scripts and Makefiles and random glue to build software, but we don't -- MacPorts, instead, requires ports to be declarative, and then takes care of the rest. These are two different problem spaces. Apple, due to B&I's requirements, *needs* software to build universal on single systems. We don't, and there are arguably many benefits to not doing so, the greatest of which is reduced maintenance and implementation complexity.
Why is everyone being so quick to slam pipping's changes before they even have a chance to go through an evolution or two? I know there are still a few people here who still don't get the whole point of universal builds, or doubt their value to MacPorts, but that's orthogonal to the whole issue of "quality of implementation", something which has to at least be given a chance to improve before coming to any firm conclusions about it.
Because the right place to fix this is in the upstream project's build, not as an add-on set of extensive hacks. It's admittedly not black and white -- if a project builds universal with a few CFLAG tweaks, and works fine under testing, then super! But blithely forcing software into the +universal universe just doesn't add up when there are only fringe benefits and *great cost* in terms of added complexity. -landonf
On Mar 3, 2007, at 1:39 PM, Jordan K. Hubbard wrote:
Now, if you're saying that the initial efforts to get there have been clunky and inelegant then I'm not going to dispute that since you're probably right. That doesn't mean you throw the baby out with the bathwater, however, it means you figure out how to do a better job of it by learning from your early mistakes!
Nobody is arguing against universal builds. The goal is to avoid a mountain of impossibly complex, difficult to maintain code in the Ports tree. The fact that universal is a happy land of milk and honey is great, but I don't think any of your points refute jberry's original statement: I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Me too. -landonf
On Mar 3, 2007, at 1:39 PM, Jordan K. Hubbard wrote:
On Mar 3, 2007, at 12:49 PM, James Berry wrote:
I'm quite distressed by the concept of too much work (and too much ugly code) going into building +universal variants of ports.
I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Let's be reasonable: the case for universal builds is fringe. We don't have easy ways to transport binary ports between systems, and it's not in general a safe thing to do.
I understand what you're saying with respect to "lack of easy transport", but I can't agree that "the case for universal builds is fringe.". Perhaps you're talking about "fringe" purely with respect to MacPorts itself, but let's zoom out for a moment and look at the bigger picture again.
I am speaking about macports, not the general case. Adding huge hacks to individual portfiles in order to support universal builds gets us nowhere -- particularly when we don't have the distribution mechanisms in place to really do anything with such universal port builds.
So, enter MacPorts. Does MacPorts absolutely need to do this or nobody will ever use it again? Of course not. There is, however, still a lot of confusion and ignorance in the OSS community about how to build things universal (particularly in the presence of configure scripts, which definitely need to evolve). A build- framework like MacPorts has the ability to substantially lead the way here and, by example, demonstrate to literally thousands of open source developers just how to do it. Just as importantly, by grasping the nettle at the build stage, MacPorts is not setting itself up for more work at the packaging stage it (I hope) eventually wants to get to. Nobody is seriously suggesting (I HOPE) having parallel collections of many thousands of packages, one for x86 and one for ppc (and, at some point, one full of experimental 64-bit capable stuff), when it's perfectly possible and encouraged to have just ONE package collection that works for everything. By not figuring out how to build universal, MacPorts would merely be stealing resources from the future in order to be lazy in the present. There would be no net gain and definitely a net loss as the ability to share a single /opt among multiple machines on a network went out the window too.
Rather than big hacks on individual ports, it would seem better to have a couple of declarative statements for the universal strategy of a port: - port may be built universal: yes/no - port builds universal out of box: yes/no - port builds in single pass with flags: xxx - port can be built in multiple passes by lipoing together the following binaries... (all others are assumed the same builds) Something like that would allow us to declaratively describe what's needed to build many ports, and would allow macports to use several strategies to build universal ports: (1) This port can't be build universal (for whatever reason) (2) This port already builds universal (we don't need to do anything special) (3) This port can be built universal by setting these flags... (4) This port can be built as multiple versions that may then be lipoed together, using all binaries or those specified We already do (2), and can mostly do (3) using a default set of flags. I'll argue that I'd rather see support for (4) built-in once, than built into each portfile. James
On Mar 3, 2007, at 3:16 PM, James Berry wrote:
I am speaking about macports, not the general case. Adding huge hacks to individual portfiles in order to support universal builds gets us nowhere -- particularly when we don't have the distribution mechanisms in place to really do anything with such universal port builds.
And I hope I've made it clear that I'm not defending the "huge hacks" either. I don't like huge hacks. I like generic support that can be leveraged. So far, I think the generic support for universal building is still pretty green, however, and I guess the ultimate question is whether or not MacPorts wants to invest anything in this area at all if people are still questioning the worth of universal executables and frameworks (in the context of MP) at all. I've said my piece and if people still don't agree, that's cool, I just thought it was something worth taking a fairly strong stand over and I'm not going to go in the corner and cry if MP decides it's always going to be a "build your own stuff" solution (which still leaves a fairly big hole in what end-users are looking for, sadly).
Rather than big hacks on individual ports, it would seem better to have a couple of declarative statements for the universal strategy of a port:
- port may be built universal: yes/no - port builds universal out of box: yes/no - port builds in single pass with flags: xxx - port can be built in multiple passes by lipoing together the following binaries... (all others are assumed the same builds)
I'm not sure what value is added by having so many states. I think, as far as the builder is concerned, the only state that counts for anything is the first one. Does it build universal? Yes? OK, then the builder can choose to build it universal if that's valuable to them. If not, then it's a moot point. As far as an internal macports developer is concerned, there's also not a lot of value in splitting hairs here. If it builds universal out of the box vs tweaking it, that's great, but it's no different than having a port which compiles on MacOSX with no patches and respects $prefix properly in all ways vs one which has to be coerced into doing those things. Whether to lipo or not depends as much on what the macports developer wants to do (e.g. how much trouble to go through in the "coercion process") as anything else, so I don't see what value a declarative statement adds there either. - Jordan
On Mar 3, 2007, at 5:06 PM, Jordan K. Hubbard wrote:
Rather than big hacks on individual ports, it would seem better to have a couple of declarative statements for the universal strategy of a port:
- port may be built universal: yes/no - port builds universal out of box: yes/no - port builds in single pass with flags: xxx - port can be built in multiple passes by lipoing together the following binaries... (all others are assumed the same builds)
I'm not sure what value is added by having so many states. I think, as far as the builder is concerned, the only state that counts for anything is the first one. Does it build universal? Yes? OK, then the builder can choose to build it universal if that's valuable to them. If not, then it's a moot point. As far as an internal macports developer is concerned, there's also not a lot of value in splitting hairs here. If it builds universal out of the box vs tweaking it, that's great, but it's no different than having a port which compiles on MacOSX with no patches and respects $prefix properly in all ways vs one which has to be coerced into doing those things. Whether to lipo or not depends as much on what the macports developer wants to do (e.g. how much trouble to go through in the "coercion process") as anything else, so I don't see what value a declarative statement adds there either.
As always, my mind was racing ahead to implementation ;) What I was thinking out, though I didn't clearly say so, was what syntax we might need to add to portfiles to support building software universally in a more general fashion than having a separate implementation for each port. The user doesn't care about any of that, surely, except for #1. James
Hi all: I've been lurking for the past little while and have been reading the arguments for both sides. I think something has been missed though. To have any meaningful discussion one must specify what is the target audience. Now, if one only considers the typical user as the one that will be using MacPorts, then having universal binaries is, of course, optional. BUT, if one includes developers in the MacPorts user audience, then this is a different story. Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous. So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)? It's just that I've seen so many discussions got south because of some people's tendency to hyper-focus on one little thing. It'd be a shame to see universal binaries not be implemented if only because of this. best regards, Reid ____________________________________________________________________________________ No need to miss a message. Get email on-the-go with Yahoo! Mail for Mobile. Get started. http://mobile.yahoo.com/mail
On 2007-03-03 19:44:32 -0800, Reid Nichol wrote: [...]
Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous.
So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)?
Yes, and as this has been said somewhere, this should be done upstream (perhaps at the autoconf level, with a new option --enable-universal). Indeed, I don't see why only MacPorts users could be interested in universal binaries. -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
On Mar 4, 2007, at 1:23 PM, Vincent Lefevre wrote:
On 2007-03-03 19:44:32 -0800, Reid Nichol wrote: [...]
Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous.
So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)?
Yes, and as this has been said somewhere, this should be done upstream (perhaps at the autoconf level, with a new option --enable-universal). Indeed, I don't see why only MacPorts users could be interested in universal binaries.
I guess nobody feels responsible for taking care of universal binaries: http://lists.gnu.org/archive/html/bug-coreutils/2007-02/msg00320.html
-- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/ blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS- Lyon) _______________________________________________ macports-dev mailing list macports-dev@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo/macports-dev
On Mar 4, 2007, at 11:10 PM, Elias Pipping wrote:
On Mar 4, 2007, at 1:23 PM, Vincent Lefevre wrote:
On 2007-03-03 19:44:32 -0800, Reid Nichol wrote: [...]
Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous.
So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)?
Yes, and as this has been said somewhere, this should be done upstream (perhaps at the autoconf level, with a new option --enable- universal). Indeed, I don't see why only MacPorts users could be interested in universal binaries.
I guess nobody feels responsible for taking care of universal binaries:
http://lists.gnu.org/archive/html/bug-coreutils/2007-02/ msg00320.html
What do you mean? I spent dozens of hours getting GNU libtool to work with universal binaries. It could create and unpack fat static archives etc before Apple announced their switch to intel. Since the switch I have also added support for their odd -isysroot etc flags. My point about autoconf is vaild. The fix for AC_C_BIGENDIAN is incomplete, it does not work if the package does not use a config header, and the other issues with sizeof checks and offsetof etc still remain. Peter
On Mar 5, 2007, at 5:32 AM, Peter O'Gorman wrote:
On Mar 4, 2007, at 11:10 PM, Elias Pipping wrote:
On Mar 4, 2007, at 1:23 PM, Vincent Lefevre wrote:
On 2007-03-03 19:44:32 -0800, Reid Nichol wrote: [...]
Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous.
So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)?
Yes, and as this has been said somewhere, this should be done upstream (perhaps at the autoconf level, with a new option --enable- universal). Indeed, I don't see why only MacPorts users could be interested in universal binaries.
I guess nobody feels responsible for taking care of universal binaries:
http://lists.gnu.org/archive/html/bug-coreutils/2007-02/ msg00320.html
What do you mean?
I spent dozens of hours getting GNU libtool to work with universal binaries.
Well, how would I know? Anyway - sorry, I must've misunderstood you.
participants (7)
-
Elias Pipping
-
James Berry
-
Jordan K. Hubbard
-
Landon Fuller
-
Peter O'Gorman
-
Reid Nichol
-
Vincent Lefevre