Hi all, I tried creating a small "buildfarm" using my MacBook Pro. I already have some results, but it's just a small test, so please be not too strict with it ;-) You can find the results here: http://ruderich.com/simon/macports/ But the results are some days old, so it may be possible a port is already fixed. I'm sorry but at the moment I don't have any dates. It would be nice if some of you could use these results to fix broken ports. I separated fetch failures from build/other failures so they should be easy to fix. If you have any suggestions/improvements/bugs or questions please tell me. Thanks, Simon
On Oct 14, 2007, at 17:00, Simon Ruderich wrote:
I tried creating a small "buildfarm" using my MacBook Pro. I already have some results, but it's just a small test, so please be not too strict with it ;-)
You can find the results here: http://ruderich.com/simon/macports/ But the results are some days old, so it may be possible a port is already fixed. I'm sorry but at the moment I don't have any dates.
It would be nice if some of you could use these results to fix broken ports. I separated fetch failures from build/other failures so they should be easy to fix.
If you have any suggestions/improvements/bugs or questions please tell me.
Very interesting and useful! Some random comments: I hope you will regenerate this occasionally, at least those that are failing right now. How long did it take to generate that list? It would help to know the date/time when you encountered each failure, and/or the revision of the portfile that was used. You may want to add a table of contents at the top so we can jump to specific ports. You may want to sort the ports in some way. It currently seems to be somewhat alphabetical, but with other ports in between (maybe dependencies?). In build.html you have gnome-session, gedit, gnu-crypto, and gnustep- base listed several times. Probably others as well. I fixed iksemel's fetch failure. Note that you had iksemel listed build.html. You have other fetch failures listed in build.html (DarwinPortsStartup, cook, p5-email-address, etc.). Lots of ports complaining about /opt-devel/bin/python2.4 being missing. Aren't those ports in the python portgroup? Wonder what's going on there. Some of your failures are suspicious. For example: php5 does not fail under normal circumstances, so I'd like to know how your build farm is operating. You report a gettext/libintl build failure for grep that I am unable to reproduce. The portfile includes the --disable-nls flag so I cannot imagine why it would be trying to link against gettext/libintl. The reported build failure for ufraw ("Image error: /opt-devel/bin/ dcraw is being used by the active dcraw port") is curious since ufraw does not depend on dcraw (not even indirectly). Why was dcraw installed? I would think a build farm would need to build each port in a clean system.
Hello again Simon! Top posting to make one general and quick comment on this topic. I am very very interested in this attempt of yours and will look into it in detail later on, so do expect many questions on what you did coming my way ;-) But in the mean time, would you care to read my entry on logging support for MacPorts "action targets" (read, those that drive a port toward its destroot)? http://trac.macports.org/projects/macports/wiki/PortLoggingProposal Following the website redesign, this is my number one priority and would indeed love to spark some interest from our developers base. It is one of the goals of my proposal to use the results of logs to create build result pages & summaries like yours on an automated fashion, as detailed in the Wiki doc, so I hope you'll find it interesting. Regards,... -jmpp On Oct 14, 2007, at 6:00 PM, Simon Ruderich wrote:
Hi all,
I tried creating a small "buildfarm" using my MacBook Pro. I already have some results, but it's just a small test, so please be not too strict with it ;-)
You can find the results here: http://ruderich.com/simon/macports/ But the results are some days old, so it may be possible a port is already fixed. I'm sorry but at the moment I don't have any dates.
It would be nice if some of you could use these results to fix broken ports. I separated fetch failures from build/other failures so they should be easy to fix.
If you have any suggestions/improvements/bugs or questions please tell me.
Thanks, Simon _______________________________________________ macports-dev mailing list macports-dev@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo/macports-dev
On Oct 14, 2007, at 11:32 PM, Ryan Schmidt wrote:
On Oct 14, 2007, at 17:00, Simon Ruderich wrote:
I tried creating a small "buildfarm" using my MacBook Pro. I already have some results, but it's just a small test, so please be not too strict with it ;-)
You can find the results here: http://ruderich.com/simon/macports/ But the results are some days old, so it may be possible a port is already fixed. I'm sorry but at the moment I don't have any dates.
It would be nice if some of you could use these results to fix broken ports. I separated fetch failures from build/other failures so they should be easy to fix.
If you have any suggestions/improvements/bugs or questions please tell me.
Very interesting and useful!
Very useful indeed! Tabulation of build-run results is another one of the "services" MacPorts is sorely lacking. And I say "services" because I see this functionality more as a server side thing that can be fed day in and day out by logs created by, you guessed it, http:// trac.macports.org/projects/macports/wiki/PortLoggingProposal ;-)
Some random comments:
I hope you will regenerate this occasionally, at least those that are failing right now. How long did it take to generate that list?
It would help to know the date/time when you encountered each failure, and/or the revision of the portfile that was used.
You may want to add a table of contents at the top so we can jump to specific ports.
You may want to sort the ports in some way. It currently seems to be somewhat alphabetical, but with other ports in between (maybe dependencies?).
I would totally love it if any scripting that's put together to preprocess build result logs is done openly (that is, in our svn ;-) and in coordination with the login proposal above and the new website. Admittedly there's still a lot to discuss to even figure out some basic guidelines for the development of such script(s), so basically lets consider this message of mine a call to start tying some of those loose ends. For starters, I propose that, at least for starters: *) the script(s) is (are) written in php and checked into some subdir of trunk/www (trunk/www/buildresults/ ?), so that the resulting web pages tie nicely into our new project homepage. What I envision moving forward is a full blown section of our project website dedicated to the tabulation of build-run result logs once we get our infrastructure (both hardware and software wise) in shape to host the build-runs themselves. Links to this new section could be created anywhere, like on the sidebar (to an introductory web page for our build-run results) and off the "Available Ports" page (which could sport direct links to individual ports build result tables), among others.
Some of your failures are suspicious. For example:
php5 does not fail under normal circumstances, so I'd like to know how your build farm is operating.
You report a gettext/libintl build failure for grep that I am unable to reproduce. The portfile includes the --disable-nls flag so I cannot imagine why it would be trying to link against gettext/ libintl.
The reported build failure for ufraw ("Image error: /opt-devel/bin/ dcraw is being used by the active dcraw port") is curious since ufraw does not depend on dcraw (not even indirectly). Why was dcraw installed? I would think a build farm would need to build each port in a clean system.
Much of the protocol for build-run results to be reliable still has to be defined (things like build frequency and creation of a "clean" environment, among many others), but getting this show on the road by taking some stabs at how results page might look like (the proposed scripts above) is by no means out of the question, in my opinion. So... Simon and Ryan, did I manage to nourish your interest into putting some lines of php code together? ;-) Regards,... -jmpp
On Sun, Oct 14, 2007 at 10:32:20PM -0500, Ryan Schmidt wrote:
On Oct 14, 2007, at 17:00, Simon Ruderich wrote:
Very interesting and useful! Some random comments:
I hope you will regenerate this occasionally, at least those that are failing right now. How long did it take to generate that list?
It took my MacBook about 2-3 days which is quite long. I guess the whole port tree would need at least a week, maybe (much) more.
It would help to know the date/time when you encountered each failure, and/or the revision of the portfile that was used.
Yes, this would be indeed very nice.
You may want to add a table of contents at the top so we can jump to specific ports.
Good idea, I will add this in the future.
You may want to sort the ports in some way. It currently seems to be somewhat alphabetical, but with other ports in between (maybe dependencies?).
I thought it's already sorted alphabetical. Maybe the problem is that ports with a uppercase name are sorted in an own group at the top, then all lowercased ports.
In build.html you have gnome-session, gedit, gnu-crypto, and gnustep-base listed several times. Probably others as well.
Yes, this is a problem with the parser I wrote to get the ports. I hope I can improve it.
I fixed iksemel's fetch failure. Note that you had iksemel listed build.html. You have other fetch failures listed in build.html (DarwinPortsStartup, cook, p5-email-address, etc.).
Thanks, I removed it from build.html. This is also a parser problem, sorry.
Lots of ports complaining about /opt-devel/bin/python2.4 being missing. Aren't those ports in the python portgroup? Wonder what's going on there.
Not sure either. I will look into it.
Some of your failures are suspicious. For example:
php5 does not fail under normal circumstances, so I'd like to know how your build farm is operating.
You report a gettext/libintl build failure for grep that I am unable to reproduce. The portfile includes the --disable-nls flag so I cannot imagine why it would be trying to link against gettext/libintl.
The reported build failure for ufraw ("Image error: /opt-devel/bin/dcraw is being used by the active dcraw port") is curious since ufraw does not depend on dcraw (not even indirectly). Why was dcraw installed? I would think a build farm would need to build each port in a clean system.
As this was my first test I just executed "sudo port install all" and wrote a small script which parsed the output of this command. I know see how limited this approach was, but I'm already working on a better one. A big problem for example is that some fetches are not failing after a timeout even if the server is not responding. I'm not sure how to fix this as an own timeout in my buildfarm would stop long downloads. My current idea is to get a list of all ports ("port list") and then build each port after another. Before each new build all other ports are removed ("port uninstall installed"). The output of port is parsed and then based on this saved in a database which is then used to create the html files. It would also be good if maintainers could mark a port fixed so it's removed from the failure list. But the best thing would be if the buildfarm is linked to the svn tree and just tries to rebuild new or failed ports so we always know which ones are failing. I already have some ideas for this and will try to integrate it in my new approach I'm currently working on. The only problem is that some ports (like gtk2) need a long time to build so maybe they shouldn't be removed after each port build. What do you think of my ideas? Thanks for your help, Simon
On Mon, Oct 15, 2007 at 05:09:52AM -0400, Juan Manuel Palacios wrote:
Hello again Simon!
Top posting to make one general and quick comment on this topic. I am very very interested in this attempt of yours and will look into it in detail later on, so do expect many questions on what you did coming my way ;-)
But in the mean time, would you care to read my entry on logging support for MacPorts "action targets" (read, those that drive a port toward its destroot)?
http://trac.macports.org/projects/macports/wiki/PortLoggingProposal
Following the website redesign, this is my number one priority and would indeed love to spark some interest from our developers base. It is one of the goals of my proposal to use the results of logs to create build result pages & summaries like yours on an automated fashion, as detailed in the Wiki doc, so I hope you'll find it interesting.
Regards,...
-jmpp
Hi Juan, I'm glad you like my small attempt ;-) I like your entry very much, it's very interesting and a good idea. Your idea with the possibility to send the log messages to macports in case of an error is perfect. This makes maintaining much easier as we (the maintainers) have all necessary information and don't have to ask the user to use -d or something similar. It would also be easier to handle build failures as -d is no longer necessary, which clutters the terminal. And of course for a buildfarm it would make parsing the output of port much easier and less error prone. I hope after the website we can work on a design for this and add it to macports as it's a real good and I think also important feature. Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229
On Oct 17, 2007, at 15:53, Simon Ruderich wrote:
On Sun, Oct 14, 2007 at 10:32:20PM -0500, Ryan Schmidt wrote:
Very interesting and useful! Some random comments:
I hope you will regenerate this occasionally, at least those that are failing right now. How long did it take to generate that list?
It took my MacBook about 2-3 days which is quite long. I guess the whole port tree would need at least a week, maybe (much) more.
I figured it would take quite awhile.
It would help to know the date/time when you encountered each failure, and/or the revision of the portfile that was used.
Yes, this would be indeed very nice.
You may want to add a table of contents at the top so we can jump to specific ports.
Good idea, I will add this in the future.
You may want to sort the ports in some way. It currently seems to be somewhat alphabetical, but with other ports in between (maybe dependencies?).
I thought it's already sorted alphabetical. Maybe the problem is that ports with a uppercase name are sorted in an own group at the top, then all lowercased ports.
Well, build.html currently lists failures in this order: ArpSpyX gnustep-base DesktopManager gnustep-base gnustep-base GNUMail-Aqua HandBrake gnustep-base ID3 gnustep-base qt3-mac NotificationWatcher gnustep-base You see my confusion.
In build.html you have gnome-session, gedit, gnu-crypto, and gnustep-base listed several times. Probably others as well.
Yes, this is a problem with the parser I wrote to get the ports. I hope I can improve it.
I fixed iksemel's fetch failure. Note that you had iksemel listed build.html. You have other fetch failures listed in build.html (DarwinPortsStartup, cook, p5-email-address, etc.).
Thanks, I removed it from build.html. This is also a parser problem, sorry.
Lots of ports complaining about /opt-devel/bin/python2.4 being missing. Aren't those ports in the python portgroup? Wonder what's going on there.
Not sure either. I will look into it.
Some of your failures are suspicious. For example:
php5 does not fail under normal circumstances, so I'd like to know how your build farm is operating.
You report a gettext/libintl build failure for grep that I am unable to reproduce. The portfile includes the --disable-nls flag so I cannot imagine why it would be trying to link against gettext/libintl.
The reported build failure for ufraw ("Image error: /opt-devel/bin/ dcraw is being used by the active dcraw port") is curious since ufraw does not depend on dcraw (not even indirectly). Why was dcraw installed? I would think a build farm would need to build each port in a clean system.
As this was my first test I just executed "sudo port install all" and wrote a small script which parsed the output of this command. I know see how limited this approach was, but I'm already working on a better one.
A big problem for example is that some fetches are not failing after a timeout even if the server is not responding. I'm not sure how to fix this as an own timeout in my buildfarm would stop long downloads.
MacPorts uses curl to download. Curl has options that can be used to consider the download failed if the download speed drops below some threshold for some period of time. If MacPorts is not currently using this option, perhaps it should. It would alleviate this problem. Here's how it's done on the command line (from "man curl"): --connect-timeout <seconds> Maximum time in seconds that you allow the connection to the server to take. This only limits the connection phase, once curl has connected this option is of no more use. See also the -m/--max-time option. -y/--speed-time <time> If a download is slower than speed-limit bytes per second during a speed-time period, the download gets aborted. If speed-time is used, the default speed-limit will be 1 unless set with -y. This option controls transfers and thus will not affect slow connects etc. If this is a concern for you, try the --connect- timeout option. -Y/--speed-limit <speed> If a download is slower than this given speed, in bytes per sec- ond, for speed-time seconds it gets aborted. speed- time is set with -Y and is 30 if not set.
My current idea is to get a list of all ports ("port list") and then build each port after another. Before each new build all other ports are removed ("port uninstall installed"). The output of port is parsed and then based on this saved in a database which is then used to create the html files.
It would also be good if maintainers could mark a port fixed so it's removed from the failure list.
But the best thing would be if the buildfarm is linked to the svn tree and just tries to rebuild new or failed ports so we always know which ones are failing. I already have some ideas for this and will try to integrate it in my new approach I'm currently working on.
Yes, that would be better.
The only problem is that some ports (like gtk2) need a long time to build so maybe they shouldn't be removed after each port build.
What do you think of my ideas?
Some ports do take a very long time, and some ports have very many dependencies. To start with, maybe it would be good to limit your build farm to ports that don't have so many dependencies (including indirect dependencies). Just so it doesn't take forever, and so that we start learning about the easier-to-fix failures.
I'd like to point out that mpwa, as deployed at http:// db.macports.org has schema support for most of what is required for this, including problem reports and build output/logs from ports. James On Oct 17, 2007, at 2:17 PM, Simon Ruderich wrote:
On Mon, Oct 15, 2007 at 05:09:52AM -0400, Juan Manuel Palacios wrote:
Hello again Simon!
Top posting to make one general and quick comment on this topic. I am very very interested in this attempt of yours and will look into it in detail later on, so do expect many questions on what you did coming my way ;-)
But in the mean time, would you care to read my entry on logging support for MacPorts "action targets" (read, those that drive a port toward its destroot)?
http://trac.macports.org/projects/macports/wiki/PortLoggingProposal
Following the website redesign, this is my number one priority and would indeed love to spark some interest from our developers base. It is one of the goals of my proposal to use the results of logs to create build result pages & summaries like yours on an automated fashion, as detailed in the Wiki doc, so I hope you'll find it interesting.
Regards,...
-jmpp
Hi Juan,
I'm glad you like my small attempt ;-)
I like your entry very much, it's very interesting and a good idea. Your idea with the possibility to send the log messages to macports in case of an error is perfect. This makes maintaining much easier as we (the maintainers) have all necessary information and don't have to ask the user to use -d or something similar.
It would also be easier to handle build failures as -d is no longer necessary, which clutters the terminal. And of course for a buildfarm it would make parsing the output of port much easier and less error prone.
I hope after the website we can work on a design for this and add it to macports as it's a real good and I think also important feature.
Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229 _______________________________________________ macports-dev mailing list macports-dev@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo/macports-dev
On Mon, Oct 15, 2007 at 01:26:30PM -0400, Juan Manuel Palacios wrote:
On Oct 14, 2007, at 11:32 PM, Ryan Schmidt wrote:
Very useful indeed! Tabulation of build-run results is another one of the "services" MacPorts is sorely lacking. And I say "services" because I see this functionality more as a server side thing that can be fed day in and day out by logs created by, you guessed it, http://trac.macports.org/projects/macports/wiki/PortLoggingProposal ;-)
This would be indeed a very useful service. If we could create a (private) distributed build farm which each builds ports and reports the results back to the server it would be a very good addition.
[snip]
I would totally love it if any scripting that's put together to preprocess build result logs is done openly (that is, in our svn ;-) and in coordination with the login proposal above and the new website. Admittedly there's still a lot to discuss to even figure out some basic guidelines for the development of such script(s), so basically lets consider this message of mine a call to start tying some of those loose ends. For starters, I propose that, at least for starters:
*) the script(s) is (are) written in php and checked into some subdir of trunk/www (trunk/www/buildresults/ ?), so that the resulting web pages tie nicely into our new project homepage.
I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later. But the question is if we should "wait" for an integration of this logging feature to be integrated into macports or if we first try to create a (distributed) buildfarm which parses the port output and later uses the macports logging feature.
What I envision moving forward is a full blown section of our project website dedicated to the tabulation of build-run result logs once we get our infrastructure (both hardware and software wise) in shape to host the build-runs themselves. Links to this new section could be created anywhere, like on the sidebar (to an introductory web page for our build-run results) and off the "Available Ports" page (which could sport direct links to individual ports build result tables), among others.
This would be a real nice feature and so everybody who wants to help macports can just grab a port name and try to build and fix it on his own and then then the patch to us. It would also make it clear how much of our ports are not working.
[snip]
Much of the protocol for build-run results to be reliable still has to be defined (things like build frequency and creation of a "clean" environment, among many others), but getting this show on the road by taking some stabs at how results page might look like (the proposed scripts above) is by no means out of the question, in my opinion.
The question is how we want to design the logging/buildfarm features. I would suggest we use a line based logging as the first try, as it's very easy to parse/handle and also easy to generate. Maybe we can use xml later, which is by the way good integrated into ruby. The buildfarm is also another section. Should we create a real server based build system where each build computer polls the macports server for a port to build and then sends the results to the server, or should be make simpler by just making it based on a computer which tries to build everything and then sends it to the server. I like the first approach (distributed) better but it's a bit more complicated as we also have to create an own protocol to poll the server.
So... Simon and Ryan, did I manage to nourish your interest into putting some lines of php code together? ;-)
Of course ;-) It would like to help macports with this interesting problem. But before we can get into coding we should collect as much ideas as possible so we don't waste any energy. Ryan, Juan and everybody who likes to help with this or has any ideas. What do you think about our/my ideas/comments? Any ideas are welcome.
Regards,...
-jmpp
Thanks, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229
On 17 Oct 2007, at 17:40, Simon Ruderich wrote:
On Mon, Oct 15, 2007 at 01:26:30PM -0400, Juan Manuel Palacios wrote:
On Oct 14, 2007, at 11:32 PM, Ryan Schmidt wrote:
*) the script(s) is (are) written in php and checked into some subdir of trunk/www (trunk/www/buildresults/ ?), so that the resulting web pages tie nicely into our new project homepage.
I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later.
BTW, I use the built-in php that comes with OS X for scripting myself... I would go for python since (cool sexy feature in 10.5) XCode 3 knows what it is, Apples supports Cocoa in Python (if I read it correctly) and it has shipped with OS X in one form or another for a while. Randall Wood rhwood@mac.com http://shyramblings.blogspot.com "The rules are simple: The ball is round. The game lasts 90 minutes. All the rest is just philosophy."
Randall Wood wrote:
I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later.
BTW, I use the built-in php that comes with OS X for scripting myself...
I would go for python since (cool sexy feature in 10.5) XCode 3 knows what it is, Apples supports Cocoa in Python (if I read it correctly) and it has shipped with OS X in one form or another for a while.
Why not Tcl ? (apart from the total lack of sex appeal, that is) --anders
Simon Ruderich wrote:
My current idea is to get a list of all ports ("port list") and then build each port after another. Before each new build all other ports are removed ("port uninstall installed"). The output of port is parsed and then based on this saved in a database which is then used to create the html files.
It would also be good if maintainers could mark a port fixed so it's removed from the failure list.
But the best thing would be if the buildfarm is linked to the svn tree and just tries to rebuild new or failed ports so we always know which ones are failing. I already have some ideas for this and will try to integrate it in my new approach I'm currently working on.
The only problem is that some ports (like gtk2) need a long time to build so maybe they shouldn't be removed after each port build.
What do you think of my ideas?
You will find some scripts that help with building ports "in order" and reusing results from previous builds in the portmgr section... http://trac.macports.org/projects/macports/browser/trunk/base/portmgr/ packaging They recreate the entire chroot rather than `port uninstall installed` and of course they haven't been updated since DarwinPorts and 10.2/6.0. --anders
On Oct 18, 2007, at 01:16, Anders F Björklund wrote:
Randall Wood wrote:
I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later.
BTW, I use the built-in php that comes with OS X for scripting myself...
I would go for python since (cool sexy feature in 10.5) XCode 3 knows what it is, Apples supports Cocoa in Python (if I read it correctly) and it has shipped with OS X in one form or another for a while.
Why not Tcl ? (apart from the total lack of sex appeal, that is)
Right. I don't know Ruby or Python and am not planning on learning them at this point. Would be unfortunate to introduce yet another language one has to learn to contribute to MacPorts. IMHO PHP would be ok, since we already use that for the web site. But my opinion may be influenced by the fact that I already know PHP very well and already use it for web sites and command-line scripts. But Tcl would also be a good choice given the rest of the project's source.
On Thu, 2007-10-18 at 01:27 -0500, Ryan Schmidt wrote:
On Oct 18, 2007, at 01:16, Anders F Björklund wrote:
Randall Wood wrote:
I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later.
BTW, I use the built-in php that comes with OS X for scripting myself...
I would go for python since (cool sexy feature in 10.5) XCode 3 knows what it is, Apples supports Cocoa in Python (if I read it correctly) and it has shipped with OS X in one form or another for a while.
Why not Tcl ? (apart from the total lack of sex appeal, that is)
Right. I don't know Ruby or Python and am not planning on learning them at this point. Would be unfortunate to introduce yet another language one has to learn to contribute to MacPorts. IMHO PHP would be ok, since we already use that for the web site. But my opinion may be influenced by the fact that I already know PHP very well and already use it for web sites and command-line scripts. But Tcl would also be a good choice given the rest of the project's source.
It also makes evaluation much easier if we don't need to parse the output of port(1) but use the API instead. The parsing approach will break every time someone fiddles with the logging stuff in port(1). Using the API would also make it much easier to build ports "in order", that is if the dependency chain is like A -> B, the best approach would be to build B then A. Otherwise the output of B will be missing. Someone knows the state of the "chroot-build"? This would make building all ports much easier as it wouldn't be necessary to remove all ports before building another one. -Markus -- Markus Weissmann http://www.mweissmann.de/ http://www.macports.org/
On Wed, Oct 17, 2007 at 02:48:43PM -0700, James Berry wrote:
I'd like to point out that mpwa, as deployed at http://db.macports.org has schema support for most of what is required for this, including problem reports and build output/logs from ports.
James
Hi, first I want to apologize for my late response. I was on vacation and busy again. Sorry. I just looked at it and it looks good. But I couldn't find any support for a buildfarm. Is this only hidden in the code? If so it would be nice if there could be svn access so we can look at it and use it. The rest looks really good. Thanks, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229
Hi, first I want to say sorry for my late reply. I was on vacation and busy. Sorry. On Wed, Oct 17, 2007 at 04:45:55PM -0500, Ryan Schmidt wrote:
Well, build.html currently lists failures in this order:
ArpSpyX gnustep-base DesktopManager gnustep-base gnustep-base GNUMail-Aqua HandBrake gnustep-base ID3 gnustep-base qt3-mac NotificationWatcher gnustep-base
You see my confusion.
Yes, this is weird. Don't know how this happened. Maybe a problem with my (stupid) parser.
MacPorts uses curl to download. Curl has options that can be used to consider the download failed if the download speed drops below some threshold for some period of time. If MacPorts is not currently using this option, perhaps it should. It would alleviate this problem. Here's how it's done on the command line (from "man curl"):
[snip]
It would be super if someone with knowledge of the port base could check this. I'm sorry but I can't read the macports source.
Some ports do take a very long time, and some ports have very many dependencies. To start with, maybe it would be good to limit your build farm to ports that don't have so many dependencies (including indirect dependencies). Just so it doesn't take forever, and so that we start learning about the easier-to-fix failures.
Yes, that should be done in our next attempt. But I think we should wait with a new (and hopeful) better try until the new logging system is integrated. That should make many things easier. Thanks, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229
On Thu, Oct 18, 2007 at 08:21:55AM +0200, Anders F Björklund wrote:
Simon Ruderich wrote:
You will find some scripts that help with building ports "in order" and reusing results from previous builds in the portmgr section...
http://trac.macports.org/projects/macports/browser/trunk/base/portmgr/packag...
They recreate the entire chroot rather than `port uninstall installed` and of course they haven't been updated since DarwinPorts and 10.2/6.0.
--anders
This sounds good. But if they weren't updated some with some knowledge of the port base should check if they still work properly. I'm sorry, but my knowledge of Tcl and macports source is still about zero. Thanks for your help, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229
On Thu, Oct 18, 2007 at 01:27:07AM -0500, Ryan Schmidt wrote:
Right. I don't know Ruby or Python and am not planning on learning them at this point. Would be unfortunate to introduce yet another language one has to learn to contribute to MacPorts. IMHO PHP would be ok, since we already use that for the web site. But my opinion may be influenced by the fact that I already know PHP very well and already use it for web sites and command-line scripts. But Tcl would also be a good choice given the rest of the project's source.
Even if I don't know any Tcl (yet) I also think it's the best way as it's already used by macports. If it would be possible to directly integrate it into the macports base, so that every client can just run a buildfarm then it would be perfect. But even a second application is fine. The server could be implemented in PHP and distribute the builds to the clients. So nobody does work multiple times. With this it would also possible for everyone to help the macports project with just donating some of his/her cpu cycles to build/test some applications. What do you think? Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229
Le 4 nov. 07 à 18:51, Simon Ruderich a écrit :
On Wed, Oct 17, 2007 at 02:48:43PM -0700, James Berry wrote:
I'd like to point out that mpwa, as deployed at http:// db.macports.org has schema support for most of what is required for this, including problem reports and build output/logs from ports.
James
Hi,
first I want to apologize for my late response. I was on vacation and busy again. Sorry.
I just looked at it and it looks good. But I couldn't find any support for a buildfarm. Is this only hidden in the code? If so it would be nice if there could be svn access so we can look at it and use it. The rest looks really good.
Thanks, Simon
I vaguely recall seeing that thread, but I was too busy to reply then. We have a very nice support for building ports in a controlled environment called "virtual chroot" as it was implemented by Eugene as part of his GSoC internship. The code is there in svn. You need to configure your macports installation with --with-trace-sdk=... to specify which cross SDK to use for building ports, and then you need to use -t to enable the trace mode (which is called trace mode for historical reasons). I believe all we need is a script handling all that and building ports one after the other... Paul -- http://paul-guyot.com/
participants (8)
-
Anders F Björklund
-
James Berry
-
Juan Manuel Palacios
-
Markus Weissmann
-
Paul Guyot
-
Randall Wood
-
Ryan Schmidt
-
Simon Ruderich