Hello again Simon! Top posting to make one general and quick comment on this topic. I am very very interested in this attempt of yours and will look into it in detail later on, so do expect many questions on what you did coming my way ;-) But in the mean time, would you care to read my entry on logging support for MacPorts "action targets" (read, those that drive a port toward its destroot)? http://trac.macports.org/projects/macports/wiki/PortLoggingProposal Following the website redesign, this is my number one priority and would indeed love to spark some interest from our developers base. It is one of the goals of my proposal to use the results of logs to create build result pages & summaries like yours on an automated fashion, as detailed in the Wiki doc, so I hope you'll find it interesting. Regards,... -jmpp On Oct 14, 2007, at 6:00 PM, Simon Ruderich wrote:

Hi all,

I tried creating a small "buildfarm" using my MacBook Pro. I already have some results, but it's just a small test, so please be not too strict with it ;-)

You can find the results here: http://ruderich.com/simon/macports/ But the results are some days old, so it may be possible a port is already fixed. I'm sorry but at the moment I don't have any dates.

It would be nice if some of you could use these results to fix broken ports. I separated fetch failures from build/other failures so they should be easy to fix.

If you have any suggestions/improvements/bugs or questions please tell me.

Thanks, Simon _______________________________________________ macports-dev mailing list macports-dev@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo/macports-dev

On Sun, Oct 14, 2007 at 10:32:20PM -0500, Ryan Schmidt wrote:

On Oct 14, 2007, at 17:00, Simon Ruderich wrote:

Very interesting and useful! Some random comments:

I hope you will regenerate this occasionally, at least those that are failing right now. How long did it take to generate that list?

It took my MacBook about 2-3 days which is quite long. I guess the whole port tree would need at least a week, maybe (much) more.

It would help to know the date/time when you encountered each failure, and/or the revision of the portfile that was used.

Yes, this would be indeed very nice.

You may want to add a table of contents at the top so we can jump to specific ports.

Good idea, I will add this in the future.

You may want to sort the ports in some way. It currently seems to be somewhat alphabetical, but with other ports in between (maybe dependencies?).

I thought it's already sorted alphabetical. Maybe the problem is that ports with a uppercase name are sorted in an own group at the top, then all lowercased ports.

In build.html you have gnome-session, gedit, gnu-crypto, and gnustep-base listed several times. Probably others as well.

Yes, this is a problem with the parser I wrote to get the ports. I hope I can improve it.

I fixed iksemel's fetch failure. Note that you had iksemel listed build.html. You have other fetch failures listed in build.html (DarwinPortsStartup, cook, p5-email-address, etc.).

Thanks, I removed it from build.html. This is also a parser problem, sorry.

Lots of ports complaining about /opt-devel/bin/python2.4 being missing. Aren't those ports in the python portgroup? Wonder what's going on there.

Not sure either. I will look into it.

Some of your failures are suspicious. For example:

php5 does not fail under normal circumstances, so I'd like to know how your build farm is operating.

You report a gettext/libintl build failure for grep that I am unable to reproduce. The portfile includes the --disable-nls flag so I cannot imagine why it would be trying to link against gettext/libintl.

The reported build failure for ufraw ("Image error: /opt-devel/bin/dcraw is being used by the active dcraw port") is curious since ufraw does not depend on dcraw (not even indirectly). Why was dcraw installed? I would think a build farm would need to build each port in a clean system.

As this was my first test I just executed "sudo port install all" and wrote a small script which parsed the output of this command. I know see how limited this approach was, but I'm already working on a better one. A big problem for example is that some fetches are not failing after a timeout even if the server is not responding. I'm not sure how to fix this as an own timeout in my buildfarm would stop long downloads. My current idea is to get a list of all ports ("port list") and then build each port after another. Before each new build all other ports are removed ("port uninstall installed"). The output of port is parsed and then based on this saved in a database which is then used to create the html files. It would also be good if maintainers could mark a port fixed so it's removed from the failure list. But the best thing would be if the buildfarm is linked to the svn tree and just tries to rebuild new or failed ports so we always know which ones are failing. I already have some ideas for this and will try to integrate it in my new approach I'm currently working on. The only problem is that some ports (like gtk2) need a long time to build so maybe they shouldn't be removed after each port build. What do you think of my ideas? Thanks for your help, Simon

On Oct 17, 2007, at 15:53, Simon Ruderich wrote:

On Sun, Oct 14, 2007 at 10:32:20PM -0500, Ryan Schmidt wrote:

...

Very interesting and useful! Some random comments:

I hope you will regenerate this occasionally, at least those that are failing right now. How long did it take to generate that list?

It took my MacBook about 2-3 days which is quite long. I guess the whole port tree would need at least a week, maybe (much) more.

I figured it would take quite awhile.

...

It would help to know the date/time when you encountered each failure, and/or the revision of the portfile that was used.

Yes, this would be indeed very nice.

...

You may want to add a table of contents at the top so we can jump to specific ports.

Good idea, I will add this in the future.

...

You may want to sort the ports in some way. It currently seems to be somewhat alphabetical, but with other ports in between (maybe dependencies?).

I thought it's already sorted alphabetical. Maybe the problem is that ports with a uppercase name are sorted in an own group at the top, then all lowercased ports.

Well, build.html currently lists failures in this order: ArpSpyX gnustep-base DesktopManager gnustep-base gnustep-base GNUMail-Aqua HandBrake gnustep-base ID3 gnustep-base qt3-mac NotificationWatcher gnustep-base You see my confusion.

...

In build.html you have gnome-session, gedit, gnu-crypto, and gnustep-base listed several times. Probably others as well.

Yes, this is a problem with the parser I wrote to get the ports. I hope I can improve it.

...

I fixed iksemel's fetch failure. Note that you had iksemel listed build.html. You have other fetch failures listed in build.html (DarwinPortsStartup, cook, p5-email-address, etc.).

Thanks, I removed it from build.html. This is also a parser problem, sorry.

...

Lots of ports complaining about /opt-devel/bin/python2.4 being missing. Aren't those ports in the python portgroup? Wonder what's going on there.

Not sure either. I will look into it.

...

Some of your failures are suspicious. For example:

php5 does not fail under normal circumstances, so I'd like to know how your build farm is operating.

You report a gettext/libintl build failure for grep that I am unable to reproduce. The portfile includes the --disable-nls flag so I cannot imagine why it would be trying to link against gettext/libintl.

The reported build failure for ufraw ("Image error: /opt-devel/bin/ dcraw is being used by the active dcraw port") is curious since ufraw does not depend on dcraw (not even indirectly). Why was dcraw installed? I would think a build farm would need to build each port in a clean system.

As this was my first test I just executed "sudo port install all" and wrote a small script which parsed the output of this command. I know see how limited this approach was, but I'm already working on a better one.

A big problem for example is that some fetches are not failing after a timeout even if the server is not responding. I'm not sure how to fix this as an own timeout in my buildfarm would stop long downloads.

MacPorts uses curl to download. Curl has options that can be used to consider the download failed if the download speed drops below some threshold for some period of time. If MacPorts is not currently using this option, perhaps it should. It would alleviate this problem. Here's how it's done on the command line (from "man curl"): --connect-timeout <seconds> Maximum time in seconds that you allow the connection to the server to take. This only limits the connection phase, once curl has connected this option is of no more use. See also the -m/--max-time option. -y/--speed-time <time> If a download is slower than speed-limit bytes per second during a speed-time period, the download gets aborted. If speed-time is used, the default speed-limit will be 1 unless set with -y. This option controls transfers and thus will not affect slow connects etc. If this is a concern for you, try the --connect- timeout option. -Y/--speed-limit <speed> If a download is slower than this given speed, in bytes per sec- ond, for speed-time seconds it gets aborted. speed- time is set with -Y and is 30 if not set.

My current idea is to get a list of all ports ("port list") and then build each port after another. Before each new build all other ports are removed ("port uninstall installed"). The output of port is parsed and then based on this saved in a database which is then used to create the html files.

It would also be good if maintainers could mark a port fixed so it's removed from the failure list.

But the best thing would be if the buildfarm is linked to the svn tree and just tries to rebuild new or failed ports so we always know which ones are failing. I already have some ideas for this and will try to integrate it in my new approach I'm currently working on.

Yes, that would be better.

The only problem is that some ports (like gtk2) need a long time to build so maybe they shouldn't be removed after each port build.

What do you think of my ideas?

Some ports do take a very long time, and some ports have very many dependencies. To start with, maybe it would be good to limit your build farm to ports that don't have so many dependencies (including indirect dependencies). Just so it doesn't take forever, and so that we start learning about the easier-to-fix failures.

On Mon, Oct 15, 2007 at 01:26:30PM -0400, Juan Manuel Palacios wrote:

On Oct 14, 2007, at 11:32 PM, Ryan Schmidt wrote:

Very useful indeed! Tabulation of build-run results is another one of the "services" MacPorts is sorely lacking. And I say "services" because I see this functionality more as a server side thing that can be fed day in and day out by logs created by, you guessed it, http://trac.macports.org/projects/macports/wiki/PortLoggingProposal ;-)

This would be indeed a very useful service. If we could create a (private) distributed build farm which each builds ports and reports the results back to the server it would be a very good addition.

[snip]

I would totally love it if any scripting that's put together to preprocess build result logs is done openly (that is, in our svn ;-) and in coordination with the login proposal above and the new website. Admittedly there's still a lot to discuss to even figure out some basic guidelines for the development of such script(s), so basically lets consider this message of mine a call to start tying some of those loose ends. For starters, I propose that, at least for starters:

*) the script(s) is (are) written in php and checked into some subdir of trunk/www (trunk/www/buildresults/ ?), so that the resulting web pages tie nicely into our new project homepage.

I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later. But the question is if we should "wait" for an integration of this logging feature to be integrated into macports or if we first try to create a (distributed) buildfarm which parses the port output and later uses the macports logging feature.

What I envision moving forward is a full blown section of our project website dedicated to the tabulation of build-run result logs once we get our infrastructure (both hardware and software wise) in shape to host the build-runs themselves. Links to this new section could be created anywhere, like on the sidebar (to an introductory web page for our build-run results) and off the "Available Ports" page (which could sport direct links to individual ports build result tables), among others.

This would be a real nice feature and so everybody who wants to help macports can just grab a port name and try to build and fix it on his own and then then the patch to us. It would also make it clear how much of our ports are not working.

[snip]

Much of the protocol for build-run results to be reliable still has to be defined (things like build frequency and creation of a "clean" environment, among many others), but getting this show on the road by taking some stabs at how results page might look like (the proposed scripts above) is by no means out of the question, in my opinion.

The question is how we want to design the logging/buildfarm features. I would suggest we use a line based logging as the first try, as it's very easy to parse/handle and also easy to generate. Maybe we can use xml later, which is by the way good integrated into ruby. The buildfarm is also another section. Should we create a real server based build system where each build computer polls the macports server for a port to build and then sends the results to the server, or should be make simpler by just making it based on a computer which tries to build everything and then sends it to the server. I like the first approach (distributed) better but it's a bit more complicated as we also have to create an own protocol to poll the server.

So... Simon and Ryan, did I manage to nourish your interest into putting some lines of php code together? ;-)

Of course ;-) It would like to help macports with this interesting problem. But before we can get into coding we should collect as much ideas as possible so we don't waste any energy. Ryan, Juan and everybody who likes to help with this or has any ideas. What do you think about our/my ideas/comments? Any ideas are welcome.

Regards,...

-jmpp

Thanks, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229

Randall Wood wrote:

...

I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later.

BTW, I use the built-in php that comes with OS X for scripting myself...

I would go for python since (cool sexy feature in 10.5) XCode 3 knows what it is, Apples supports Cocoa in Python (if I read it correctly) and it has shipped with OS X in one form or another for a while.

Why not Tcl ? (apart from the total lack of sex appeal, that is) --anders

On Oct 18, 2007, at 01:16, Anders F Björklund wrote:

Randall Wood wrote:

...

...

I like the idea of a server part in php as it's easy to code and maintain. For the client I would prefer ruby, because it's installed on each mac and is very powerful in parsing the port output and also easy to read. But that's a detail we can work out later.

BTW, I use the built-in php that comes with OS X for scripting myself...

I would go for python since (cool sexy feature in 10.5) XCode 3 knows what it is, Apples supports Cocoa in Python (if I read it correctly) and it has shipped with OS X in one form or another for a while.

Why not Tcl ? (apart from the total lack of sex appeal, that is)

Right. I don't know Ruby or Python and am not planning on learning them at this point. Would be unfortunate to introduce yet another language one has to learn to contribute to MacPorts. IMHO PHP would be ok, since we already use that for the web site. But my opinion may be influenced by the fact that I already know PHP very well and already use it for web sites and command-line scripts. But Tcl would also be a good choice given the rest of the project's source.

On Wed, Oct 17, 2007 at 02:48:43PM -0700, James Berry wrote:

I'd like to point out that mpwa, as deployed at http://db.macports.org has schema support for most of what is required for this, including problem reports and build output/logs from ports.

James

Hi, first I want to apologize for my late response. I was on vacation and busy again. Sorry. I just looked at it and it looks good. But I couldn't find any support for a buildfarm. Is this only hidden in the code? If so it would be nice if there could be svn access so we can look at it and use it. The rest looks really good. Thanks, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229

On Thu, Oct 18, 2007 at 08:21:55AM +0200, Anders F Björklund wrote:

Simon Ruderich wrote:

You will find some scripts that help with building ports "in order" and reusing results from previous builds in the portmgr section...

http://trac.macports.org/projects/macports/browser/trunk/base/portmgr/packag...

They recreate the entire chroot rather than `port uninstall installed` and of course they haven't been updated since DarwinPorts and 10.2/6.0.

--anders

This sounds good. But if they weren't updated some with some knowledge of the port base should check if they still work properly. I'm sorry, but my knowledge of Tcl and macports source is still about zero. Thanks for your help, Simon -- + privacy is necessary + using http://gnupg.org + public key id: 0x6115F804EFB33229

Le 4 nov. 07 à 18:51, Simon Ruderich a écrit :

On Wed, Oct 17, 2007 at 02:48:43PM -0700, James Berry wrote:

...

I'd like to point out that mpwa, as deployed at http:// db.macports.org has schema support for most of what is required for this, including problem reports and build output/logs from ports.

James

Hi,

first I want to apologize for my late response. I was on vacation and busy again. Sorry.

I just looked at it and it looks good. But I couldn't find any support for a buildfarm. Is this only hidden in the code? If so it would be nice if there could be svn access so we can look at it and use it. The rest looks really good.

Thanks, Simon

I vaguely recall seeing that thread, but I was too busy to reply then. We have a very nice support for building ports in a controlled environment called "virtual chroot" as it was implemented by Eugene as part of his GSoC internship. The code is there in svn. You need to configure your macports installation with --with-trace-sdk=... to specify which cross SDK to use for building ports, and then you need to use -t to enable the trace mode (which is called trace mode for historical reasons). I believe all we need is a script handling all that and building ports one after the other... Paul -- http://paul-guyot.com/