Re: Speed up build phase with "make -j"
On 28.10.2007, at 23:52, js wrote:
On 10/29/07, Anders F Björklund <afb@macports.org> wrote:
Note that some ports will fail to build, when using "make -j 2"
What causes the build erorr? is it for bad Makefile or problem of "-j" option itself?
Bad makefiles, usually missing dependencies.
So, in a way, using -j encourage us to find wrong makefiles:)
Well, if you're lucky -- parallel builds get non-deterministic, so software _may_ build correctly. You may (!) even end up with a successful build that created "bad" installations with misbehaving software; An intermediate step that makes changes to a file may get scheduled after the last step compiles an object file from this file; this way you might get a successful build but w/o the changes from step 2. Those are extremely hard to trace bugs and maintainers might reject bug reports if builds got made parallel this way -- after all they cannot reproduce failing builds due to the non-determinism. I'd rather encourage a mechanism for port that provides a mechanism for port authors to know how many processors there are and to enable parallel builds if they know this to work. Perhaps a switch "use_parallel_build [yes|no]" that will add "-j 2*CPUS" to build.args. -Markus --- Markus W. Weissmann http://www.mweissmann.de/
I'd rather encourage a mechanism for port that provides a mechanism for port authors to know how many processors there are and to enable parallel builds if they know this to work. Perhaps a switch "use_parallel_build [yes|no]" that will add "-j 2*CPUS" to build.args.
There's a bunch of other influences, such as number of cores and amount of RAM and whether the build is using some bloated language like C++ and many other considerations, for the -j parameter. So in the end I just made it an integer (build.jobs), and left the heuristics up to the user :-) --anders PS. There's also a build.nice, if you want to go the other route (build slower, like in background)
On 30.10.2007, at 12:13, Anders F Björklund wrote:
I'd rather encourage a mechanism for port that provides a mechanism for port authors to know how many processors there are and to enable parallel builds if they know this to work. Perhaps a switch "use_parallel_build [yes|no]" that will add "-j 2*CPUS" to build.args.
There's a bunch of other influences, such as number of cores and amount of RAM and whether the build is using some bloated language like C++ and many other considerations, for the -j parameter.
So in the end I just made it an integer (build.jobs), and left the heuristics up to the user :-)
--anders
PS. There's also a build.nice, if you want to go the other route (build slower, like in background)
Yes, but I meant this to be a per-Portfile choice so a maintainer can mark his/her port as being able to build in parallel. Doing this with the sledgehammer for all ports that use 'make' might be a bit too much. ;) For build.nice it might be cool++ to do change the nice level for port (1) directly so all phases run with low priority, not just the build phase. -Markus --- Markus W. Weissmann http://www.mweissmann.de/
Weissmann Markus:
Yes, but I meant this to be a per-Portfile choice so a maintainer can mark his/her port as being able to build in parallel. Doing this with the sledgehammer for all ports that use 'make' might be a bit too much. ;)
It is supposed to be the default... But broken ports might need some way of signalling so, maybe "build.jobs 1".
For build.nice it might be cool++ to do change the nice level for port(1) directly so all phases run with low priority, not just the build phase.
"nice port" ? I'm not sure if Mac OS X has ionice, but it might be useful for resource hogs like e.g. bzip2... --anders
On 2007-10-30 12:43:52 +0100, Anders F Björklund wrote:
Weissmann Markus:
Yes, but I meant this to be a per-Portfile choice so a maintainer can mark his/her port as being able to build in parallel. Doing this with the sledgehammer for all ports that use 'make' might be a bit too much. ;)
It is supposed to be the default... But broken ports might need some way of signalling so, maybe "build.jobs 1".
The bug could also be reported upstream, so that a ".NOTPARALLEL:" is added to the Makefile. -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
o The bug could also be reported upstream, so that a ".NOTPARALLEL:" o is added to the Makefile. Interesting pseudo-target. Maybe MacPorts uses, and ports that can't should patch their makefiles? -------------- Salvatore Domenick Desiano Doctoral Candidate Robotics Institute Carnegie Mellon University On Tue, 30 Oct 2007, Vincent Lefevre wrote: o On 2007-10-30 12:43:52 +0100, Anders F Björklund wrote: o > Weissmann Markus: o >> Yes, but I meant this to be a per-Portfile choice so a maintainer can mark o >> his/her port as being able to build in parallel. Doing this with the o >> sledgehammer for all ports that use 'make' might be a bit too much. ;) o > o > It is supposed to be the default... But broken ports might need some way of o > signalling so, maybe "build.jobs 1". o o o -- o Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> o 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> o Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon) o _______________________________________________ o macports-dev mailing list o macports-dev@lists.macosforge.org o http://lists.macosforge.org/mailman/listinfo/macports-dev o o
On 30.10.2007, at 13:52, Vincent Lefevre wrote:
On 2007-10-30 12:43:52 +0100, Anders F Björklund wrote:
Weissmann Markus:
Yes, but I meant this to be a per-Portfile choice so a maintainer can mark his/her port as being able to build in parallel. Doing this with the sledgehammer for all ports that use 'make' might be a bit too much. ;)
It is supposed to be the default... But broken ports might need some way of signalling so, maybe "build.jobs 1".
The bug could also be reported upstream, so that a ".NOTPARALLEL:" is added to the Makefile.
Well, this simply is not going to work -- this has to be an opt-in, not an opt-out option! If you do not know if a parallel build will work for a port, you must assume it won't. Once upon a day when we see that 80% of our ports all have that "build_in_parallel yes" option set, we can make it the default, but not as long as only _very_ few do. Otherwise we'd have to tag all our Portfiles with the "dont_even_think_about_a_parallel_build yes" option the moment we release version 1.6 and gradually remove them for the one we know to work... -Markus -- Markus W. Weissmann http://www.mweissmann.de/
On Oct 30, 2007, at 11:20 AM, Markus Weissmann wrote:
Well, this simply is not going to work -- this has to be an opt-in, not an opt-out option!
I agree with Markus. It doesn't make sense to create a new default option that could (and probably will) break a bunch of ports.
If you do not know if a parallel build will work for a port, you must assume it won't. Once upon a day when we see that 80% of our ports all have that "build_in_parallel yes" option set, we can make it the default, but not as long as only _very_ few do. Otherwise we'd have to tag all our Portfiles with the "dont_even_think_about_a_parallel_build yes" option the moment we release version 1.6 and gradually remove them for the one we know to work...
If someone wanted to run some build-tests of the entire tree and determine which ones work or not (and if they discover that most work), then perhaps it would make sense to tag the broken ones and set a new default. ... I don't really see that as something that's likely to happen, though. -- Daniel J. Luke +========================================================+ | *---------------- dluke@geeklair.net ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+
On 2007-10-30 16:20:18 +0100, Markus Weissmann wrote:
On 30.10.2007, at 13:52, Vincent Lefevre wrote:
The bug could also be reported upstream, so that a ".NOTPARALLEL:" is added to the Makefile.
Well, this simply is not going to work -- this has to be an opt-in, not an opt-out option!
That's not the point. Whether MacPorts uses a parallel build by default or not, a Makefile should be correct (either by supporting parallel build or by having a ".NOTPARALLEL:"). If it is not, this is a bug. If such bugs are not reported upstream, you're not going anywhere (experience shows that in general, upstream doesn't test parallel builds).
If you do not know if a parallel build will work for a port, you must assume it won't.
I disagree. Port maintainers should test their port to see if they work with "make -j".
Once upon a day when we see that 80% of our ports all have that "build_in_parallel yes" option set, we can make it the default, but not as long as only _very_ few do.
Yes, but how can you hope that 80% of ports will have "build_in_parallel yes" if one assumes that parallel build doesn't work? -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
On 31.10.2007, at 05:13, Vincent Lefevre wrote:
On 2007-10-30 16:20:18 +0100, Markus Weissmann wrote:
On 30.10.2007, at 13:52, Vincent Lefevre wrote:
The bug could also be reported upstream, so that a ".NOTPARALLEL:" is added to the Makefile.
Well, this simply is not going to work -- this has to be an opt-in, not an opt-out option!
That's not the point. Whether MacPorts uses a parallel build by default or not, a Makefile should be correct (either by supporting parallel build or by having a ".NOTPARALLEL:"). If it is not, this is a bug. If such bugs are not reported upstream, you're not going anywhere (experience shows that in general, upstream doesn't test parallel builds).
If you do not know if a parallel build will work for a port, you must assume it won't.
I disagree. Port maintainers should test their port to see if they work with "make -j".
Well, you cannot reliably test this. If you're lucky it might work one time and fail the other.
Once upon a day when we see that 80% of our ports all have that "build_in_parallel yes" option set, we can make it the default, but not as long as only _very_ few do.
Yes, but how can you hope that 80% of ports will have "build_in_parallel yes" if one assumes that parallel build doesn't work?
I didn't say I assume this to be the case one day, it was more of an utopian perspective; one that currently is not true for sure. The point is that ports that are not given 110% love (e.g. unmaintained ones, busy maintainers) will simply break -- probably in spectacular non-deterministic ways. -Markus -- Markus W. Weissmann http://www.mweissmann.de/
On 2007-10-31 12:26:44 +0100, Markus Weissmann wrote:
On 31.10.2007, at 05:13, Vincent Lefevre wrote:
I disagree. Port maintainers should test their port to see if they work with "make -j".
Well, you cannot reliably test this. If you're lucky it might work one time and fail the other.
So, what do you suggest? Remain with sequential builds forever? -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
On Oct 31, 2007, at 7:36 AM, Vincent Lefevre wrote:
On 2007-10-31 12:26:44 +0100, Markus Weissmann wrote:
On 31.10.2007, at 05:13, Vincent Lefevre wrote:
I disagree. Port maintainers should test their port to see if they work with "make -j".
Well, you cannot reliably test this. If you're lucky it might work one time and fail the other.
So, what do you suggest? Remain with sequential builds forever?
Vincent, are you just arguing for fun? Markus suggested that maintainers be able to flag ports that they know build fine with "make -j" so that macports can do parallel builds in those cases. I think this is probably the best solution for our current situation (it will cause the fewest problems while allowing parallel builds where possible). If you are very concerned with maximizing the possibility of parallel builds, you're of course free to validate every port and contribute updates. ;-) -- Daniel J. Luke +========================================================+ | *---------------- dluke@geeklair.net ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+
On 31.10.2007, at 12:36, Vincent Lefevre wrote:
On 2007-10-31 12:26:44 +0100, Markus Weissmann wrote:
On 31.10.2007, at 05:13, Vincent Lefevre wrote:
I disagree. Port maintainers should test their port to see if they work with "make -j".
Well, you cannot reliably test this. If you're lucky it might work one time and fail the other.
So, what do you suggest? Remain with sequential builds forever?
No, but we should only enable parallel builds for software we know to behave correctly. I wont kick you if you enable a parallel build for a port that you maintain and you are not 100% sure that it'll work _always_ -- the bug reports are yours. But I do not like 1. unmaintained and 2. my ports to automatically try to build in parallel -- and I suppose there are other maintainers that don't like their ports to get broken, too. Therefore I prefer this to be an opt-in and I will enable this feature e.g. for the gcc ports I maintain. -Markus -- Markus W. Weissmann http://www.mweissmann.de/
Markus Weissmann wrote:
No, but we should only enable parallel builds for software we know to behave correctly. I wont kick you if you enable a parallel build for a port that you maintain and you are not 100% sure that it'll work _always_ -- the bug reports are yours. But I do not like 1. unmaintained and 2. my ports to automatically try to build in parallel -- and I suppose there are other maintainers that don't like their ports to get broken, too. Therefore I prefer this to be an opt-in and I will enable this feature e.g. for the gcc ports I maintain.
Parallell builds are an optional feature, and will never be disabled by default. If the build breaks, the user can always go back to build as the regular -j 1 ? The only thing for Portfiles is where you *know* that -j doesn't work at all, sorta like violate destroot ? All the others "might" work, just like you say... --anders
On 31.10.2007, at 15:45, Anders F Björklund wrote:
Parallell builds are an optional feature, and will never be disabled by default.
*enabled* by default, darnit.
So the discussion has narrowed to: 1.) disabled by default on an "per-installation" option: Can be toggled system-wide (with default "off"); ports have to actively deny a parallel build attempt; 2.) disabled by default on a per-port option Can be toggled system-wide (with default ?); ports have to actively declare to be build-able in parallel; The point is: The primary goal when building software is working installations, everything else is of less priority. For getting the build time reduced, you get defunct software, which is a bad trade-off. Therefore even the power-user with an 8-core Xeon will rather wait a bit longer than get defunct software. So enabling parallel builds will suck, as stuff starts to break. I'd rather give them a real benefit that IF they enable parallel builds, they get working software _always_. Therefore I think 2.) is the option of choice so everyone gets happy. The default in 2.) can be set manually or with some auto-number-of- cores detection, I dont care specifically. -Markus -- Markus W. Weissmann http://www.mweissmann.de/
On 2007-10-31 15:04:45 +0100, Markus Weissmann wrote:
On 31.10.2007, at 12:36, Vincent Lefevre wrote:
On 2007-10-31 12:26:44 +0100, Markus Weissmann wrote:
On 31.10.2007, at 05:13, Vincent Lefevre wrote:
I disagree. Port maintainers should test their port to see if they work with "make -j".
Well, you cannot reliably test this. If you're lucky it might work one time and fail the other.
So, what do you suggest? Remain with sequential builds forever?
No, but we should only enable parallel builds for software we know to behave correctly.
But you've said: "Well, you cannot reliably test this. If you're lucky it might work one time and fail the other.". So, under these conditions, you cannot know that they behave correctly.
I wont kick you if you enable a parallel build for a port that you maintain and you are not 100% sure that it'll work _always_ -- the bug reports are yours. But I do not like 1. unmaintained and 2. my ports to automatically try to build in parallel
I've never suggested that. -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
On 2007-10-31 15:57:56 +0100, Markus Weissmann wrote:
So the discussion has narrowed to: 1.) disabled by default on an "per-installation" option: Can be toggled system-wide (with default "off"); ports have to actively deny a parallel build attempt;
2.) disabled by default on a per-port option Can be toggled system-wide (with default ?); ports have to actively declare to be build-able in parallel;
Portfiles should be able to say that they support a parallel build and that they don't support a parallel build, i.e. you have 3 possibilities: "yes" (which can later be changed to "no" if someone finds that it doesn't work on some particular machine), "no" (that could be changed to "yes" once the bug has been fixed upstream), and "don't know". -- Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/> Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)
Markus Weissmann wrote:
So the discussion has narrowed to: 1.) disabled by default on an "per-installation" option: Can be toggled system-wide (with default "off"); ports have to actively deny a parallel build attempt;
2.) disabled by default on a per-port option Can be toggled system-wide (with default ?); ports have to actively declare to be build-able in parallel;
The point is: The primary goal when building software is working installations, everything else is of less priority. For getting the build time reduced, you get defunct software, which is a bad trade-off. Therefore even the power-user with an 8-core Xeon will rather wait a bit longer than get defunct software. So enabling parallel builds will suck, as stuff starts to break. I'd rather give them a real benefit that IF they enable parallel builds, they get working software _always_.
Therefore I think 2.) is the option of choice so everyone gets happy. The default in 2.) can be set manually or with some auto-number-of-cores detection, I dont care specifically.
Feel free to implement whatever is needed for the portfile parsing of this new toggle, I only wanted to add something a little more accessible than tweaking the build.cmd directly. If having a global default for -j is considered too dangerous, then it could be pulled from macports.conf altogether and only be available as an option. Just got tired of getting the "how do I pass -j flags over to make" question over and over... http://trac.macports.org/projects/macports/ticket/12280 --anders
On 31.10.2007, at 16:35, Vincent Lefevre wrote:
On 2007-10-31 15:57:56 +0100, Markus Weissmann wrote:
So the discussion has narrowed to: 1.) disabled by default on an "per-installation" option: Can be toggled system-wide (with default "off"); ports have to actively deny a parallel build attempt;
2.) disabled by default on a per-port option Can be toggled system-wide (with default ?); ports have to actively declare to be build-able in parallel;
Portfiles should be able to say that they support a parallel build and that they don't support a parallel build, i.e. you have 3 possibilities: "yes" (which can later be changed to "no" if someone finds that it doesn't work on some particular machine), "no" (that could be changed to "yes" once the bug has been fixed upstream), and "don't know".
I've just added an option use_parallel_build [yes|no] to trunk; if it is set to "yes", "-j N" is added if the number of desired parallel processes is configured in port.conf (nothing changed here). The options defaults to "no", so no ports will break when we release v1.6. -Markus --- Markus W. Weissmann http://www.mweissmann.de/
participants (6)
-
Anders F Björklund
-
Daniel J. Luke
-
Markus Weissmann
-
Salvatore Domenick Desiano
-
Vincent Lefevre
-
Weissmann Markus