Please Reply To All so that your reply goes to the mailing list too, not just to me. On Dec 11, 2007, at 13:00, Daniel Oberhoff wrote:
Am 11.12.2007 um 09:50 schrieb Ryan Schmidt:
On Dec 11, 2007, at 02:35, Daniel Oberhoff wrote:
Am 10.12.2007 um 01:33 schrieb Ryan Schmidt:
On Dec 9, 2007, at 14:43, Daniel Oberhoff wrote:
Now that Leopard is out and already at 10.5.1 will macports be supporting 64bit libraries? It's just I need 64bit in my octave installation. I pull my octave from octave.org's cvs, but it needs quite a lot of support libraries. From what I gather it should be possible on Leopard to build fat libraries, i.e. ones that contain 64 and 32 bit code (i think it works using -arch x84_64 -arch i686 as gcc flags). Or will this be left to the separate ports?
MacPorts is supposed to build libraries for whatever system it's running on. So I would have thought that if you're on a 64-bit Intel system, it should build 64-bit Intel libraries. Is it building 32-bit Intel libraries for you?
We have the +universal variant for building 2-way (32-bit, Intel and PowerPC) universal binaries. We are still in the process of getting this to work with many of the ports. It could be changed to build 4-way (32-bit and 64-bit, Intel and PowerPC) universal binaries. This should be possible on Tiger too, as far as I know. It won't fix any ports that are having trouble building 2- way universal binaries. Not sure if it would mess up any ports that are already working. Are all the ports that you need already working as 2-way universal binaries?
I haven't heard anyone suggest building libraries that contain 32-bit and 64-bit code for just one processor family before (in relation to MacPorts). It would of course be possible, but I think it would make most sense to continue along our current path: software should be default install for the architecture you're on, and if you need multiple architectures, then you need the +universal variant.
It has been said before that maybe 64-bit binaries aren't all that helpful, but the Ars Technica review of Leopard explains that while 64-bit binaries aren't that helpful on the PowerPC architecture, they really are quite good for secondary reasons on the Intel architecture. The 32-bit Intel architecture has often been called inferior to the 32-bit PowerPC architecture, but the 64-bit Intel architecture seems to fix many of the issues. Also, maybe Leopard being a full 64-bit system makes 64- bit binaries more relevant.
Perhaps we could do 4-way universal binaries only when MacPorts is running on Leopard.... but that might be a bad idea, since only people running Leopard could then develop and test this.
We could introduce a new automatic variant... +universal4? +universal64? People could test with this new variant and if any problems are encountered it would not prevent anyone from using the existing 2-way 32-bit +universal variant. I'm wary of this though... I wouldn't want, say, "universal64" directives to start appearing in portfiles, if we want to eventually fold +universal64 into +universal.
Just some thoughts off the top of my head.
Hmm, ok. That makes sense, since I upgraded with an installed macports base. I suppose I should really reinstall all then, since otherwise I will end up with a crazy mix?
Yes, if you're moving from one architecture to another (G4->G5, PPC->Intel, 32-bit->64-bit, etc.) you should really throw away your old MacPorts (saving any conf files or databases or other data) and rebuild all your ports.
Though, per default on Leopard compilation seems to be 32 bit. When you just compile something without further options pointer size is 32bit. Only when you specify -arch x86_64 or -m64 do you get 64bit.
Ah, I see. I didn't know. I haven't had a 64-bit machine.
So it would really make sense to have both (but not ppc). So I would vote for an autamatic variant to get intel only 32/64bit.
So by default we build 32-bit binaries for the host architecture. And we have a +universal variant for building 32-bit binaries for both architectures. You want an additional variant for building fat 32-bit and 64-bit Intel binaries. Do we then also need a separate variant for building fat 32-bit and 64-bit PowerPC binaries? And what about a fat universal variant with all 4 options? This proliferation of variants is what I want to avoid. Most users will not need it and it will confuse some of them. It's much extra work for some portfile authors, because some software (rather a lot, actually) does not build successful universal binaries using the simple tricks, and much more work is required to build universal. See for example the openssl port (which, to build universal, you'll have to back-date to version 0.9.8e because of bug #12911) which goes through crazy hoops to build first for one architecture, then the other, then lipo them together. Similar (though perhaps not identical) hoops would have to be gone through for fat binaries or 4-way binaries. Well, that is until MacPorts base grows some universal-build helpers. (I still really think MacPorts base should have an easy way to build a port multiple times, once for each architecture, and lipo it together, automatically, selected with just a single line in the portfile).
Yah, that doesn't sound fun. Would it be feasible to make these incremental? After all the mechanism to combine architectures is always the same (use -arch flags, lipo...). I.e. make it possible to say +64 +32 +ppc +intel to get the 4way build or just +ppc +intel to get a two way build. Or at least have something like that behind the scenes, and just offer specializations if some of the variants need some special care.
I don't want users to have to think about what architecture(s) they're on. Macs are supposed to be easy. It's easier if the +universal variant includes all architectures, and if one day +universal becomes the default. "Software just works," as it says on http://www.apple.com/universal/ The mechanism to combine architectures is not always the same. For example, it's different if it's a configure-based project vs. an Xcode-based project. Also, while many configure-based projects accept multiple -arch flags to build for multiple architectures at once, some need you to build for just one arch, then later lipo these together. MacPorts does not have any special support for the latter at this time, though I think it should.
Alternatively maybe have two macports trees, one with 64bit, one with 32bit. Kinda like those lib64 dirs on some linux flavours. Is that possible right now?
You're talking about two collections of port binaries, presumably? Since we don't have any binaries at this point, or any 64-bit builds, anything's possible, of course. Is it wise to have 64-bit-only binaries? I wouldn't have thought so but I really have no idea. If you mean, is it possible to have 64-bit builds go into lib64 instead of lib, then I would say if that's not the default for the software already, it would be a rather large hassle to fix every portfile to do this, wouldn't you say? And why is this better than a single all-encompassing superuniversal library installed in the normal expected location?
Why, by the way, do you need 64-bit octave? Just curious.
Various reasons:
1) I do get data clumps in the gigabyte range sometimes. Yes, it does lead to some paging, but that's better than a crash with out of memory :). 2) this is more of a hack: I use Judy for some sparse double precision structs, and since Judy constrains data size to sizeof (void*) I need 64 bit. 3) afaik 64bit has more registers than 32bit on intel architecture, avoiding register spillage, which sometimes snails some of my code.
all in all, it is not critical right now. But every now and then 64bit would help a lot. Maybe not always though. Also due to the policy on Leopard to default to 32bit would probably make it hard to make do with an exclusive 64bit set of libraries.
Ok.
Hmm, thinking of that: is it possible to convince port _not_ to switch over to 64bit now? I'd rather have a clean set of 32 bit libs :).
I don't understand?
On 11.12.2007, at 20:20, Ryan Schmidt wrote:
Please Reply To All so that your reply goes to the mailing list too, not just to me.
On Dec 11, 2007, at 13:00, Daniel Oberhoff wrote:
Am 11.12.2007 um 09:50 schrieb Ryan Schmidt:
On Dec 11, 2007, at 02:35, Daniel Oberhoff wrote:
Am 10.12.2007 um 01:33 schrieb Ryan Schmidt:
On Dec 9, 2007, at 14:43, Daniel Oberhoff wrote:
Now that Leopard is out and already at 10.5.1 will macports be supporting 64bit libraries? It's just I need 64bit in my octave installation. I pull my octave from octave.org's cvs, but it needs quite a lot of support libraries. From what I gather it should be possible on Leopard to build fat libraries, i.e. ones that contain 64 and 32 bit code (i think it works using -arch x84_64 -arch i686 as gcc flags). Or will this be left to the separate ports?
MacPorts is supposed to build libraries for whatever system it's running on. So I would have thought that if you're on a 64- bit Intel system, it should build 64-bit Intel libraries. Is it building 32-bit Intel libraries for you?
We have the +universal variant for building 2-way (32-bit, Intel and PowerPC) universal binaries. We are still in the process of getting this to work with many of the ports. It could be changed to build 4-way (32-bit and 64-bit, Intel and PowerPC) universal binaries. This should be possible on Tiger too, as far as I know. It won't fix any ports that are having trouble building 2-way universal binaries. Not sure if it would mess up any ports that are already working. Are all the ports that you need already working as 2-way universal binaries?
I haven't heard anyone suggest building libraries that contain 32-bit and 64-bit code for just one processor family before (in relation to MacPorts). It would of course be possible, but I think it would make most sense to continue along our current path: software should be default install for the architecture you're on, and if you need multiple architectures, then you need the +universal variant.
It has been said before that maybe 64-bit binaries aren't all that helpful, but the Ars Technica review of Leopard explains that while 64-bit binaries aren't that helpful on the PowerPC architecture, they really are quite good for secondary reasons on the Intel architecture. The 32-bit Intel architecture has often been called inferior to the 32-bit PowerPC architecture, but the 64-bit Intel architecture seems to fix many of the issues. Also, maybe Leopard being a full 64-bit system makes 64- bit binaries more relevant.
Perhaps we could do 4-way universal binaries only when MacPorts is running on Leopard.... but that might be a bad idea, since only people running Leopard could then develop and test this.
We could introduce a new automatic variant... +universal4? +universal64? People could test with this new variant and if any problems are encountered it would not prevent anyone from using the existing 2-way 32-bit +universal variant. I'm wary of this though... I wouldn't want, say, "universal64" directives to start appearing in portfiles, if we want to eventually fold +universal64 into +universal.
Just some thoughts off the top of my head.
Hmm, ok. That makes sense, since I upgraded with an installed macports base. I suppose I should really reinstall all then, since otherwise I will end up with a crazy mix?
Yes, if you're moving from one architecture to another (G4->G5, PPC->Intel, 32-bit->64-bit, etc.) you should really throw away your old MacPorts (saving any conf files or databases or other data) and rebuild all your ports.
Though, per default on Leopard compilation seems to be 32 bit. When you just compile something without further options pointer size is 32bit. Only when you specify -arch x86_64 or -m64 do you get 64bit.
Ah, I see. I didn't know. I haven't had a 64-bit machine.
So it would really make sense to have both (but not ppc). So I would vote for an autamatic variant to get intel only 32/64bit.
So by default we build 32-bit binaries for the host architecture. And we have a +universal variant for building 32-bit binaries for both architectures. You want an additional variant for building fat 32-bit and 64-bit Intel binaries. Do we then also need a separate variant for building fat 32-bit and 64-bit PowerPC binaries? And what about a fat universal variant with all 4 options? This proliferation of variants is what I want to avoid. Most users will not need it and it will confuse some of them. It's much extra work for some portfile authors, because some software (rather a lot, actually) does not build successful universal binaries using the simple tricks, and much more work is required to build universal. See for example the openssl port (which, to build universal, you'll have to back-date to version 0.9.8e because of bug #12911) which goes through crazy hoops to build first for one architecture, then the other, then lipo them together. Similar (though perhaps not identical) hoops would have to be gone through for fat binaries or 4-way binaries. Well, that is until MacPorts base grows some universal-build helpers. (I still really think MacPorts base should have an easy way to build a port multiple times, once for each architecture, and lipo it together, automatically, selected with just a single line in the portfile).
Yah, that doesn't sound fun. Would it be feasible to make these incremental? After all the mechanism to combine architectures is always the same (use -arch flags, lipo...). I.e. make it possible to say +64 +32 +ppc +intel to get the 4way build or just +ppc +intel to get a two way build. Or at least have something like that behind the scenes, and just offer specializations if some of the variants need some special care.
I don't want users to have to think about what architecture(s) they're on. Macs are supposed to be easy. It's easier if the +universal variant includes all architectures, and if one day +universal becomes the default. "Software just works," as it says on
http://www.apple.com/universal/
The mechanism to combine architectures is not always the same. For example, it's different if it's a configure-based project vs. an Xcode-based project. Also, while many configure-based projects accept multiple -arch flags to build for multiple architectures at once, some need you to build for just one arch, then later lipo these together. MacPorts does not have any special support for the latter at this time, though I think it should.
we put some research into this during GSOC this year: It's quite hard to come up with a magic lipo for everything as it is necessary to combine not only binaries (thats the easy part) but also stuff like header files (also easy), heap images of interpreters, scripting language programs, etc. -- which can get practically impossible. We could enable four-way builds for (probably) all ports that we can build universal right now, but there is currently no way of saying "no ppc64 please" (etc.) in a port (and no tests/experience on what exactly will be necessary).
Alternatively maybe have two macports trees, one with 64bit, one with 32bit. Kinda like those lib64 dirs on some linux flavours. Is that possible right now?
You're talking about two collections of port binaries, presumably? Since we don't have any binaries at this point, or any 64-bit builds, anything's possible, of course. Is it wise to have 64-bit- only binaries? I wouldn't have thought so but I really have no idea.
If you mean, is it possible to have 64-bit builds go into lib64 instead of lib, then I would say if that's not the default for the software already, it would be a rather large hassle to fix every portfile to do this, wouldn't you say? And why is this better than a single all-encompassing superuniversal library installed in the normal expected location?
you could always pass a "-arch x86_64" (or "-arch ppc64") to the cflags/cxxflags/.. manually if you desperately need 64 bit support, e.g.: $> port -d install readline configure.cflags="-O2 -arch x86_64" (no idea if that one compiles though) If you want to make some tests with four-way universal builds, replace each occurrence (there are 3) of "-arch i386 -arch ppc" in / opt/local/share/macports/Tcl/port1.0/portconfigure.tcl with "-arch i386 -arch x86_64 -arch ppc -arch ppc64". We are of course interested in your results... ;) For octave you're out of luck though as it is build with the vanilla gcc 4.2 which cannot produce universal binaries. Perhaps you can test the Apple gcc 4.2 beta preview for octave to achieve this (if you're on 10.5) -- though the Fortran part may be a problem here as octave seems to use it (the Apple compilers don't do Fortan while the vanilla ones cannot create universal binaries).
Why, by the way, do you need 64-bit octave? Just curious.
Various reasons:
1) I do get data clumps in the gigabyte range sometimes. Yes, it does lead to some paging, but that's better than a crash with out of memory :). 2) this is more of a hack: I use Judy for some sparse double precision structs, and since Judy constrains data size to sizeof (void*) I need 64 bit. 3) afaik 64bit has more registers than 32bit on intel architecture, avoiding register spillage, which sometimes snails some of my code.
all in all, it is not critical right now. But every now and then 64bit would help a lot. Maybe not always though. Also due to the policy on Leopard to default to 32bit would probably make it hard to make do with an exclusive 64bit set of libraries.
Ok.
Hmm, thinking of that: is it possible to convince port _not_ to switch over to 64bit now? I'd rather have a clean set of 32 bit libs :).
As you said: Even if a port can be build four way universal (with 64 bit support), the 64 bit version doesn't necessarily get executed... Of course you could always strip the unwanted code out of the binary manually afterwards (with lipo). Regards, -Markus --- Dipl. Inf. (FH) Markus W. Weissmann http://www.mweissmann.de/ http://www.macports.org/
participants (2)
-
Ryan Schmidt
-
Weissmann Markus