On Mar 3, 2007, at 12:49 PM, James Berry wrote:

I'm quite distressed by the concept of too much work (and too much ugly code) going into building +universal variants of ports.

I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Let's be reasonable: the case for universal builds is fringe. We don't have easy ways to transport binary ports between systems, and it's not in general a safe thing to do.

I understand what you're saying with respect to "lack of easy transport", but I can't agree that "the case for universal builds is fringe.". Perhaps you're talking about "fringe" purely with respect to MacPorts itself, but let's zoom out for a moment and look at the bigger picture again. First, universal binaries are here to stay and, once you start running Leopard, you will begin to look at anything which is NOT universal on your system with annoyance and disdain. The reason for this is something I've already covered - the Macintosh community is becoming increasingly heterogeneous in nature as more and more Intel machines join the PPC installed base, and the best thing we can do for this community ("we" being both Apple and its associated developers) is make it increasingly irrelevant just which architecture they're using, be it PPC, PPC64, x86_32 or x86_64. Stuff they download off the net or run off of a CD, network or firewire volume should Just Work™ and work in a highly performant way - Rosetta is really a technology of last resort so please don't imagine that "ppc everywhere" is a good idea. We're already seeing the user community put a lot of pressure on folks like Adobe to get their stuff universal since things like plug- ins and extensions make a non-Universal world really a nightmare (particularly when you need a PPC-only plug-in and an x86-only plug- in running in a web browser - you end up having to run two browsers). So, bigger picture, Universal is the way to go and it's the message we're sending to everyone, both because it's the right thing to do from an advanced-technology perspective and because we want to present a united front on this issue to the commercial ISVs who might otherwise be tempted to let this one slide for awhile longer and continue to impact the overall Macintosh user experience in negative ways. So, enter MacPorts. Does MacPorts absolutely need to do this or nobody will ever use it again? Of course not. There is, however, still a lot of confusion and ignorance in the OSS community about how to build things universal (particularly in the presence of configure scripts, which definitely need to evolve). A build-framework like MacPorts has the ability to substantially lead the way here and, by example, demonstrate to literally thousands of open source developers just how to do it. Just as importantly, by grasping the nettle at the build stage, MacPorts is not setting itself up for more work at the packaging stage it (I hope) eventually wants to get to. Nobody is seriously suggesting (I HOPE) having parallel collections of many thousands of packages, one for x86 and one for ppc (and, at some point, one full of experimental 64-bit capable stuff), when it's perfectly possible and encouraged to have just ONE package collection that works for everything. By not figuring out how to build universal, MacPorts would merely be stealing resources from the future in order to be lazy in the present. There would be no net gain and definitely a net loss as the ability to share a single /opt among multiple machines on a network went out the window too. I also think MacPorts will increasingly find itself to be the odd man out if it doesn't go down the universal road. People ARE figuring out how to do it and it's become something of a badge of honor to get your application and frameworks building n-way universal so that your users don't have to think about anything more than which version of your product or project they want. People who are using xcode have an easier time of it - they just click the universal checkbox and, unless they've written something egregious into their code, it just works. I hope you're not saying that the Unix command-line driven community can't possibly build its stuff as well as xcode can since them's fightin' words. :-) Now, if you're saying that the initial efforts to get there have been clunky and inelegant then I'm not going to dispute that since you're probably right. That doesn't mean you throw the baby out with the bathwater, however, it means you figure out how to do a better job of it by learning from your early mistakes! - Jordan

On Mar 3, 2007, at 2:33 PM, Jordan K. Hubbard wrote:

On Mar 3, 2007, at 1:09 PM, Landon Fuller wrote:

...

I agree. It's not that universal builds are a bad idea; it's that supporting universal software requires serious bastardization of Portfiles in most cases. Assuming we implement some sort of binary distribution, I'd personally much rather use the approach that fkr and shantonu used for supporting Darwin x86 and PowerPC -- build the software on each system, natively, and then join the resultant RPMs into a single fat RPM.

I think this may have gotten lost in all my earlier verbiage, but I also pointed out early in the conversation that there are many many open source projects building universal at Apple without such "bastardization". The Makefile for OpenSSL is just 147 lines, for example, and of that at LEAST 50% has to do strictly with Apple's own requirements (the whole SRCROOT and DSTROOT thing, order files, all the things you probably still have nightmares about from time to time). Perhaps 5 lines in the Makefile are devoted to the fact that it's building universal.

I don't think it's reasonable to compare Apple's static manually managed B&I build system to MacPorts. We could also write shell scripts and Makefiles and random glue to build software, but we don't -- MacPorts, instead, requires ports to be declarative, and then takes care of the rest. These are two different problem spaces. Apple, due to B&I's requirements, *needs* software to build universal on single systems. We don't, and there are arguably many benefits to not doing so, the greatest of which is reduced maintenance and implementation complexity.

Why is everyone being so quick to slam pipping's changes before they even have a chance to go through an evolution or two? I know there are still a few people here who still don't get the whole point of universal builds, or doubt their value to MacPorts, but that's orthogonal to the whole issue of "quality of implementation", something which has to at least be given a chance to improve before coming to any firm conclusions about it.

Because the right place to fix this is in the upstream project's build, not as an add-on set of extensive hacks. It's admittedly not black and white -- if a project builds universal with a few CFLAG tweaks, and works fine under testing, then super! But blithely forcing software into the +universal universe just doesn't add up when there are only fringe benefits and *great cost* in terms of added complexity. -landonf

On Mar 3, 2007, at 1:39 PM, Jordan K. Hubbard wrote:

On Mar 3, 2007, at 12:49 PM, James Berry wrote:

...

I'm quite distressed by the concept of too much work (and too much ugly code) going into building +universal variants of ports.

I'd like to see people refrain from completely bastardizing portfiles in order to support universal builds. Let's be reasonable: the case for universal builds is fringe. We don't have easy ways to transport binary ports between systems, and it's not in general a safe thing to do.

I understand what you're saying with respect to "lack of easy transport", but I can't agree that "the case for universal builds is fringe.". Perhaps you're talking about "fringe" purely with respect to MacPorts itself, but let's zoom out for a moment and look at the bigger picture again.

I am speaking about macports, not the general case. Adding huge hacks to individual portfiles in order to support universal builds gets us nowhere -- particularly when we don't have the distribution mechanisms in place to really do anything with such universal port builds.

So, enter MacPorts. Does MacPorts absolutely need to do this or nobody will ever use it again? Of course not. There is, however, still a lot of confusion and ignorance in the OSS community about how to build things universal (particularly in the presence of configure scripts, which definitely need to evolve). A build- framework like MacPorts has the ability to substantially lead the way here and, by example, demonstrate to literally thousands of open source developers just how to do it. Just as importantly, by grasping the nettle at the build stage, MacPorts is not setting itself up for more work at the packaging stage it (I hope) eventually wants to get to. Nobody is seriously suggesting (I HOPE) having parallel collections of many thousands of packages, one for x86 and one for ppc (and, at some point, one full of experimental 64-bit capable stuff), when it's perfectly possible and encouraged to have just ONE package collection that works for everything. By not figuring out how to build universal, MacPorts would merely be stealing resources from the future in order to be lazy in the present. There would be no net gain and definitely a net loss as the ability to share a single /opt among multiple machines on a network went out the window too.

Rather than big hacks on individual ports, it would seem better to have a couple of declarative statements for the universal strategy of a port: - port may be built universal: yes/no - port builds universal out of box: yes/no - port builds in single pass with flags: xxx - port can be built in multiple passes by lipoing together the following binaries... (all others are assumed the same builds) Something like that would allow us to declaratively describe what's needed to build many ports, and would allow macports to use several strategies to build universal ports: (1) This port can't be build universal (for whatever reason) (2) This port already builds universal (we don't need to do anything special) (3) This port can be built universal by setting these flags... (4) This port can be built as multiple versions that may then be lipoed together, using all binaries or those specified We already do (2), and can mostly do (3) using a default set of flags. I'll argue that I'd rather see support for (4) built-in once, than built into each portfile. James

On Mar 3, 2007, at 5:06 PM, Jordan K. Hubbard wrote:

...

Rather than big hacks on individual ports, it would seem better to have a couple of declarative statements for the universal strategy of a port:

- port may be built universal: yes/no - port builds universal out of box: yes/no - port builds in single pass with flags: xxx - port can be built in multiple passes by lipoing together the following binaries... (all others are assumed the same builds)

I'm not sure what value is added by having so many states. I think, as far as the builder is concerned, the only state that counts for anything is the first one. Does it build universal? Yes? OK, then the builder can choose to build it universal if that's valuable to them. If not, then it's a moot point. As far as an internal macports developer is concerned, there's also not a lot of value in splitting hairs here. If it builds universal out of the box vs tweaking it, that's great, but it's no different than having a port which compiles on MacOSX with no patches and respects $prefix properly in all ways vs one which has to be coerced into doing those things. Whether to lipo or not depends as much on what the macports developer wants to do (e.g. how much trouble to go through in the "coercion process") as anything else, so I don't see what value a declarative statement adds there either.

As always, my mind was racing ahead to implementation ;) What I was thinking out, though I didn't clearly say so, was what syntax we might need to add to portfiles to support building software universally in a more general fashion than having a separate implementation for each port. The user doesn't care about any of that, surely, except for #1. James

On 2007-03-03 19:44:32 -0800, Reid Nichol wrote: [...]

Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous.

So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)?

Yes, and as this has been said somewhere, this should be done upstream (perhaps at the autoconf level, with a new option --enable-universal). Indeed, I don't see why only MacPorts users could be interested in universal binaries. -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

On Mar 4, 2007, at 11:10 PM, Elias Pipping wrote:

On Mar 4, 2007, at 1:23 PM, Vincent Lefevre wrote:

...

On 2007-03-03 19:44:32 -0800, Reid Nichol wrote: [...]

...

Now, as for the messing up the Portfile thing. Yes, everyone agrees that having the universal binary hack in every Portfile is messy and undesirable. Please, stop bringing this up as it really is beating a dead horse at this point. Also, to assume that this is the only option for including this functionality is ridiculous.

So, instead of trashing one bad idea (over and over), how about discussing ways that might get this wanted functionality in with minimal pain. A couple ideas have already been thrown out there. How about sticking to discussing those (or other ones)?

Yes, and as this has been said somewhere, this should be done upstream (perhaps at the autoconf level, with a new option --enable- universal). Indeed, I don't see why only MacPorts users could be interested in universal binaries.

I guess nobody feels responsible for taking care of universal binaries:

http://lists.gnu.org/archive/html/bug-coreutils/2007-02/ msg00320.html

What do you mean? I spent dozens of hours getting GNU libtool to work with universal binaries. It could create and unpack fat static archives etc before Apple announced their switch to intel. Since the switch I have also added support for their odd -isysroot etc flags. My point about autoconf is vaild. The fix for AC_C_BIGENDIAN is incomplete, it does not work if the package does not use a config header, and the other issues with sizeof checks and offsetof etc still remain. Peter