Can the port command take advantage of multiple cores?
As the subject says, can I take advantage my my shiny new dual-core MacBook Pro? Something like "port install -j2"? Thx, -- Skip Montanaro - skip@pobox.com - http://www.webfast.com/~skip/ The major difference between Democrats and Republicans is that Republicans don't know that Randy Newman's lyrics are full of sarcasm.
On Feb 2, 2008, at 12:53, skip@pobox.com wrote:
As the subject says, can I take advantage my my shiny new dual-core MacBook Pro? Something like "port install -j2"?
Not exactly. Not all software supports parallel building. Each port must be tested to ensure that it works. As ports are tested, "use_parallel_build yes" is added to the portfile. Then MacPorts will build those ports using -j$jobs (where $jobs is the number of CPU cores in your computer). This was added for MacPorts 1.6.
Le 08-02-02 à 13:53, skip@pobox.com a écrit :
As the subject says, can I take advantage my my shiny new dual-core MacBook Pro? Something like "port install -j2"?
The answer is yes and no ... $prefix/etc/macports/macports.conf lets you specify a -j arg. However, macports will not use it unless the Portfile says it'S OK to do so with each specific port. Now, very few (almost none) Portfiles have this flag defined. It's just too bad because most ports will build fine with -j > 1 (I have tried over 150 ports with -j 8 and 5 have failed because of that, mostly in the destroot phase). yves
Yves de Champlain wrote:
However, macports will not use it unless the Portfile says it'S OK to do so with each specific port. Now, very few (almost none) Portfiles have this flag defined. It's just too bad because most ports will build fine with -j > 1 (I have tried over 150 ports with -j 8 and 5 have failed because of that, mostly in the destroot phase).
Why is this opt-in and not opt-out? We could add a flag to ports that they should be built with -j1 instead of what was specified in macports.conf. Is there a reason or some problem I don't know of? Rainer
On Feb 2, 2008, at 18:37, Rainer Müller wrote:
Yves de Champlain wrote:
However, macports will not use it unless the Portfile says it'S OK to do so with each specific port. Now, very few (almost none) Portfiles have this flag defined. It's just too bad because most ports will build fine with -j > 1 (I have tried over 150 ports with -j 8 and 5 have failed because of that, mostly in the destroot phase).
Why is this opt-in and not opt-out? We could add a flag to ports that they should be built with -j1 instead of what was specified in macports.conf. Is there a reason or some problem I don't know of?
Changes in base should not break existing ports, if at all possible. Ports may fail with this option enabled. Therefore it's opt-in on a port-by-port basis.
Ryan Schmidt wrote:
Changes in base should not break existing ports, if at all possible. Ports may fail with this option enabled. Therefore it's opt-in on a port-by-port basis.
So every maintainer has to check for himself if his/her ports built fine with parallel building enabled? With this approach it will last a long time until we get real benefits of parallel building. Although many users already own multi-core Macs and want to have faster builds... I don't think that many ports will break with this. As Yves said, 5 out of 150 failed for him. Tagging 5 ports is easier than tagging 150. And it would be not a real problem if a port fails, there is always the workaround to disable this option in macports.conf again until the Portfile for a failing port is modified accordingly. Rainer
On Feb 2, 2008, at 19:25, Rainer Müller wrote:
Ryan Schmidt wrote:
Changes in base should not break existing ports, if at all possible. Ports may fail with this option enabled. Therefore it's opt-in on a port-by-port basis.
So every maintainer has to check for himself if his/her ports built fine with parallel building enabled? With this approach it will last a long time until we get real benefits of parallel building. Although many users already own multi-core Macs and want to have faster builds...
I don't think that many ports will break with this. As Yves said, 5 out of 150 failed for him. Tagging 5 ports is easier than tagging 150. And it would be not a real problem if a port fails, there is always the workaround to disable this option in macports.conf again until the Portfile for a failing port is modified accordingly.
There was a long thread on this topic leading up to the current implementation; please review: http://lists.macosforge.org/pipermail/macports-users/2007-October/ 006479.html In particular Markus's comment here seems to exemplify why it's the way it is: http://lists.macosforge.org/pipermail/macports-users/2007-October/ 006601.html "The point is that ports that are not given 110% love (e.g. unmaintained ones, busy maintainers) will simply break [if parallel builds are enabled by default] probably in spectacular non- deterministic ways." We have enough other problems with ports right now. Let's not make more, especially spectacular non-deterministic ones, by enabling parallel builds by default.
Ryan> As ports are tested, "use_parallel_build yes" is added to the Ryan> portfile. Then MacPorts will build those ports using -j$jobs Ryan> (where $jobs is the number of CPU cores in your computer). This Ryan> was added for MacPorts 1.6. Excellent, thanks. That makes it transparent even. ;-) Skip
Rainer Müller wrote:
And it would be not a real problem if a port fails, there is always the workaround to disable this option in macports.conf again until the Portfile for a failing port is modified accordingly.
You can also disable it at runtime, by giving the argument build.jobs=1 --anders
On Feb 2, 2008, at 18:08, skip@pobox.com wrote:
Ryan> As ports are tested, "use_parallel_build yes" is added to the Ryan> portfile. Then MacPorts will build those ports using -j$jobs Ryan> (where $jobs is the number of CPU cores in your computer). This Ryan> was added for MacPorts 1.6.
Excellent, thanks. That makes it transparent even. ;-)
Well, that's what I thought happened. But now others are saying there's an option in macports.conf that needs to be set. Can someone please clarify? Is there an option, and if so, what does it need to be set to?
It looks to me like all you have to do is set "buildmakejobs" to the number of jobs you want make to use. I assume port will then use that many jobs when a portfile allows it. On Feb 3, 2008 11:03 AM, Ryan Schmidt <ryandesign@macports.org> wrote:
On Feb 2, 2008, at 18:08, skip@pobox.com wrote:
Ryan> As ports are tested, "use_parallel_build yes" is added to the Ryan> portfile. Then MacPorts will build those ports using -j$jobs Ryan> (where $jobs is the number of CPU cores in your computer). This Ryan> was added for MacPorts 1.6.
Excellent, thanks. That makes it transparent even. ;-)
Well, that's what I thought happened. But now others are saying there's an option in macports.conf that needs to be set. Can someone please clarify? Is there an option, and if so, what does it need to be set to?
-- James Sumners http://james.roomfullofmirrors.com/ "All governments suffer a recurring problem: Power attracts pathological personalities. It is not that power corrupts but that it is magnetic to the corruptible. Such people have a tendency to become drunk on violence, a condition to which they are quickly addicted." Missionaria Protectiva, Text QIV (decto) CH:D 59
Ryan Schmidt wrote:
Ryan> As ports are tested, "use_parallel_build yes" is added to the Ryan> portfile. Then MacPorts will build those ports using -j$jobs Ryan> (where $jobs is the number of CPU cores in your computer). This Ryan> was added for MacPorts 1.6.
Excellent, thanks. That makes it transparent even. ;-)
Well, that's what I thought happened. But now others are saying there's an option in macports.conf that needs to be set. Can someone please clarify? Is there an option, and if so, what does it need to be set to?
There is a config parameter in macports.conf called "buildmakejobs", that controls the default value for build.jobs (default config is 1) However, this was considered dangerous so the Portfile opt-in setting of "use_parallel_build" was added - with a default setting of no. So in order to get MacPorts to use -j2 or more, you need to both set your local preference and then also each Portfile to opt-in... Opt-in was added in http://trac.macports.org/projects/macports/changeset/30714 before that it assumed opt-out -(by using "build.jobs 1") instead. --anders
James> It looks to me like all you have to do is set "buildmakejobs" to James> the number of jobs you want make to use. Where do I set that, on the command line, in a config file? Thanks, -- Skip Montanaro - skip@pobox.com - http://www.webfast.com/~skip/ The major difference between Democrats and Republicans is that Republicans don't know that Randy Newman's lyrics are full of sarcasm.
James Sumners wrote:
It looks to me like all you have to do is set "buildmakejobs" to the number of jobs you want make to use. I assume port will then use that many jobs when a portfile allows it.
The number of build jobs is ignored, until the port says that it works. (otherwise the above is right, you can also say "0" to mean "# of cores") Setting of "makefile_isnt_buggy" was renamed to "use_parallel_build"... --anders
skip> Where do I set that, on the command line, in a config file? Sorry. Saw it in a later message in the thread. I'll be more patient next time. Skip
On Feb 3, 2008, at 10:26, Anders F Björklund wrote:
James Sumners wrote:
It looks to me like all you have to do is set "buildmakejobs" to the number of jobs you want make to use. I assume port will then use that many jobs when a portfile allows it.
The number of build jobs is ignored, until the port says that it works. (otherwise the above is right, you can also say "0" to mean "# of cores")
Setting of "makefile_isnt_buggy" was renamed to "use_parallel_build"...
Thanks, everyone. I also checked out the code to confirm that it does work this way. http://trac.macosforge.org/projects/macports/browser/trunk/base/src/ port1.0/portbuild.tcl I filed a ticket to request it be documented in the guide. http://trac.macosforge.org/projects/macports/ticket/14164
On 2008-02-02 17:45:35 -0600, Ryan Schmidt wrote:
Not exactly. Not all software supports parallel building. Each port must be tested to ensure that it works. As ports are tested, "use_parallel_build yes" is added to the portfile. Then MacPorts will build those ports using -j$jobs (where $jobs is the number of CPU cores in your computer). This was added for MacPorts 1.6.
Is there any reason why this option isn't documented in the portfile(7) man page? -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
On Feb 3, 2008, at 18:48, Vincent Lefevre wrote:
On 2008-02-02 17:45:35 -0600, Ryan Schmidt wrote:
Not exactly. Not all software supports parallel building. Each port must be tested to ensure that it works. As ports are tested, "use_parallel_build yes" is added to the portfile. Then MacPorts will build those ports using -j$jobs (where $jobs is the number of CPU cores in your computer). This was added for MacPorts 1.6.
Is there any reason why this option isn't documented in the portfile(7) man page?
You could file a ticket to get it added to the manpage. Though I thought our manpages were out of date anyway, since we're not yet shipping the new manpages built with our new documentation system, due to some problem or other.
Le 08-02-03 à 19:48, Vincent Lefevre a écrit :
On 2008-02-02 17:45:35 -0600, Ryan Schmidt wrote:
Not exactly. Not all software supports parallel building. Each port must be tested to ensure that it works. As ports are tested, "use_parallel_build yes" is added to the portfile. Then MacPorts will build those ports using -j$jobs (where $jobs is the number of CPU cores in your computer). This was added for MacPorts 1.6.
Is there any reason why this option isn't documented in the portfile(7) man page?
BTW, gcc42 which is flagged as parrallel safe breaks for me at destroot. So it might happen that stuff that works on 2 cores fail with more cores. yves
On Feb 5, 2008, at 16:41, Yves de Champlain wrote:
Le 08-02-03 à 19:48, Vincent Lefevre a écrit :
On 2008-02-02 17:45:35 -0600, Ryan Schmidt wrote:
Not exactly. Not all software supports parallel building. Each port must be tested to ensure that it works. As ports are tested, "use_parallel_build yes" is added to the portfile. Then MacPorts will build those ports using -j$jobs (where $jobs is the number of CPU cores in your computer). This was added for MacPorts 1.6.
Is there any reason why this option isn't documented in the portfile(7) man page?
BTW, gcc42 which is flagged as parrallel safe breaks for me at destroot. So it might happen that stuff that works on 2 cores fail with more cores.
Well, how many cores do you have, and what error are you getting?
Le 08-02-05 à 21:14, Ryan Schmidt a écrit :
On Feb 5, 2008, at 16:41, Yves de Champlain wrote:
Le 08-02-03 à 19:48, Vincent Lefevre a écrit :
On 2008-02-02 17:45:35 -0600, Ryan Schmidt wrote:
Not exactly. Not all software supports parallel building. Each port must be tested to ensure that it works. As ports are tested, "use_parallel_build yes" is added to the portfile. Then MacPorts will build those ports using -j$jobs (where $jobs is the number of CPU cores in your computer). This was added for MacPorts 1.6.
Is there any reason why this option isn't documented in the portfile(7) man page?
BTW, gcc42 which is flagged as parrallel safe breaks for me at destroot. So it might happen that stuff that works on 2 cores fail with more cores.
Well, how many cores do you have, and what error are you getting?
It was a general warning rather than a bug report ... 8 cores and happens during destroot. I did not keep the output but it had something to do with "waiting for unfinished jobs" yves
On 2008-02-06 00:44:51 -0500, Yves de Champlain wrote:
It was a general warning rather than a bug report ... 8 cores and happens during destroot. I did not keep the output but it had something to do with "waiting for unfinished jobs"
I don't think this is a bug. Doesn't this mean that "make" doesn't have any new job to start (e.g. because there has been an error, possibly unrelated to parallel make) and waits for the current ones to finish before quitting? -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
participants (7)
-
Anders F Björklund
-
James Sumners
-
Rainer Müller
-
Ryan Schmidt
-
skip＠pobox.com
-
Vincent Lefevre
-
Yves de Champlain