[MacPorts] #48609: El Capitan buildbot build server teneleven-slave.macports.org
#48609: El Capitan buildbot build server teneleven-slave.macports.org ----------------------------+--------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Keywords: | Port: ----------------------------+--------------------- OS X 10.11 El Capitan is coming, and I'd like to request the necessary resources to get a new buildbot build server (presumably virtual machine) set up for El Capitan, to run alongside the existing build servers. I'm Cc'ing Joshua who I believe was involved with setting up our buildbot installation. Since El Capitan is not released yet, we probably can't/shouldn't set this up with a beta version of 10.11, and should wait until the final public release, but I wanted to make sure you're aware of the need for additional server resources for this so that we can get those allocated on time. To match the other servers, this should get a static IP in the MacPorts VLAN and be given the hostname teneleven- slave.macports.org. As I recall, there were also a number of firewall exceptions that had to be requested which, if you don't have a list of them, would be the same exceptions that are granted for the other build servers; maybe IS&T can look them up from that. -- Ticket URL: <https://trac.macports.org/ticket/48609> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+--------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by ryandesign@…): Just checking, Keith: has anything been done about this yet? The rumor sites speculate that El Capitan will be released September 30, which is only two weeks away. I know IS&T can sometimes take time to respond to requests, so if you haven't yet requested an IP address and hostname from them and spoken with them about the firewall exceptions, please begin that dialog with them now. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:3> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+--------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by ryandesign@…): El Capitan has been released. Is there any progress update on getting an El Capitan buildbot builder set up? -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:4> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+--------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Changes (by ryandesign@…): * priority: Normal => High Comment: Any news on this? -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:5> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Changes (by ryandesign@…): * keywords: => elcapitan -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:7> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by casr+macports@…): Is there anything that can be donated to help with this? Time, money, machines, etc. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:9> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): No, I will work on it, there's just other existing systems that are broken that I'm trying to fix before I add a new system which is merely a nice- to-have and not something that's actually required. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:13> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by casr+macports@…): Okay and thank you for the update. I wish you luck! I'll very much appreciate it :) -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:14> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Progress so far: I cloned the Yosemite builder and gave it a new UUID and MAC address and IP address and computer name. I tried to create a new DNS name for it but I may not have restarted the DNS server properly because it's not resolving yet. I need to figure out how to change the buildbot configuration on it to give it a new buildbot identity, and on the buildbot master to let it know about this new builder. Obviously I still need to upgrade the OS, upgrade Xcode and the command line tools, reinstall MacPorts, and uninstall all ports and then have it build all ports. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:15> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by jmr@…): Replying to [comment:15 ryandesign@…]:
I tried to create a new DNS name for it but I may not have restarted the DNS server properly because it's not resolving yet.
What's the TTL?
I need to figure out how to change the buildbot configuration on it to give it a new buildbot identity, and on the buildbot master to let it know about this new builder.
On the slave machine, there should be a directory with all the files for the buildbot-slave instance, including buildbot.tac. In that file, you want the slavename parameter. On the master, you want master.cfg and slaves.json. You should probably sync up the former with svn, updating it with the changes from the repo and committing any local changes. Then you need to add the new platform to the build_platforms list. Also add an entry for the new slave to slaves.json. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:16> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Thanks Joshua. I'll look into that. I've upgraded the OS to El Capitan. I've put in a request to open up the required network ports. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:17> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Upgraded Xcode and command line tools. Reinstalled MacPorts. Uninstalled all ports prior to upgrading to El Capitan. Figured out how to properly reload the nameserver so the teneleven-slave hostname is resolving now. Changed the settings in the buildbot slave configuration. The ticket about opening the network ports has been escalated and I'm waiting to hear back. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:18> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Replying to [comment:18 ryandesign@…]:
The ticket about opening the network ports has been escalated and I'm waiting to hear back.
The ticket has been escalated to a different group today, who are hopefully the ones who can resolve it. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:22> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by dixiao@…): Would it be possible to host the buildbot somewhere else? 4 month after the release is a long waiting time, and compiling packages like llvm on an aging macbook air is not a fun task. Even though to use macports does not require binaries, I think providing binaries is essential for usability of macports. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:24> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Changes (by ryandesign@…): * cc: dixiao@… (added) Comment: Replying to [comment:24 dixiao@…]:
Would it be possible to host the buildbot somewhere else?
I am looking into all options. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:26> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Replying to [comment:18 ryandesign@…]:
The ticket about opening the network ports has been escalated and I'm waiting to hear back.
The network ports were opened, except the port which enables the master and build slave to talk to each other. I've notified them of this problem. I've upgraded Xcode and the command line tools to 7.2, also fixing ProblemHotlist#xcode7.2. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:27> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Replying to [comment:27 ryandesign@…]:
The network ports were opened, except the port which enables the master and build slave to talk to each other. I've notified them of this problem.
This port is now open as well. The next problem is that buildbot relies on the existence of /usr/bin/twistd, which Apple apparently removed from OS X in El Capitan. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:28> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by cal@…): MacPorts has it in the py27-twisted port, but I'm not sure if you actually want to install that using MacPorts. Otherwise a Python virtualenv could probably also install it using `pip install twisted`, but then `virtualenv`s are also not part of the OS X base install :/ -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:29> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): We can't install anything in the "main" MacPorts installation at /opt/local because that's controlled by the buildbot and everything in it gets deactivated after every build. We could have a second MacPorts installation elsewhere for software used by the buildbot, but for now I'm trying to install twisted manually. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:30> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): I downloaded Twisted-13.2.0.tar.bz2 because that's version included with Yosemite. I installed it with `sudo /usr/bin/python setup.py install`. I edited org.macports.build.base-slave.plist and org.macports.build.ports- slave.plist, replacing `/usr/bin/twistd` with `/usr/local/bin/twistd`. It appears to be working: the first base build completed, and the first ports build is running. Let me monitor that, and then kick off a few builds for the more popular ports, before we start the long build of all ports. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:31> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): I built the most-requested ports, based on the web server's error_log. Now I've started the build of all ports: https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/26 -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:32> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Hmm, it looks like the elcapitan ports builder stopped early: {{{ 2016-02-07 19:12:45-0800 [-] command timed out: 1200 seconds without output, attempting to kill 2016-02-07 19:12:45-0800 [-] trying to kill process group 71215 2016-02-07 19:12:45-0800 [-] signal 9 sent successfully 2016-02-07 19:12:45-0800 [-] command finished with signal 9, exit code None, elapsedTime: 420716.093723 2016-02-07 19:12:45-0800 [-] SlaveBuilder.commandComplete <buildslave.commands.shell.SlaveShellCommand instance at 0x10a87d680> }}} This may have been because of #49759, which we really need to fix. Then, it looks like the build master crashed. I think that's the order in which it happened. It could be that the master crashed while trying to do some processing on the builder's enormous log file or something. The builder has apparently re-used build number 26 now to build the ports that had changed after I started the original build 26. I'll start a new build of all ports after that's done. It should go quicker and produce a smaller log this time since it got through ~5000 out of ~20000 ports on the last attempt. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:34> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by jmr@…): Somehow minivmac is hanging the build after failing to configure: * https://build.macports.org/builders/buildports-elcapitan- x86_64/builds/38 * https://build.macports.org/builders/buildports-elcapitan- x86_64/builds/39 But why isn't it failing in pre-extract? Is there a ROM file on the buildslave? -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:35> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Noticed that... Yes, the ROM file is on the buildslaves. The build had been succeeding on the other builders. I had to make a change to make the port build on El Capitan. Maybe the buildbot isn't happy with that. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:36> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Oh wait... minivmac. That's the old version. minivmac-devel is where I fixed it. I moved the ROM file on the El Capitan builder so now it should fail in pre-extract and not hang. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:37> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): After that, Joshua started * https://build.macports.org/builders/buildports-elcapitan- x86_64/builds/44 It finished today, after 177 hours or so, but again it crashed the build master and build number 44 has now been reused to build subsequently modified ports. I think it got through all the ports though; it was a hundred or so ports shy of the end when I checked earlier today. One last attempt to build all ports should hopefully succeed, and upload the packages to the server. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:38> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by dixiao@…): It seems the buildbot stopped uploading compiled packages. For example, free type 2.6.3 is shown to be successfully built, but it's not available at packages.macports.org/freetype. Log (https://build.macports.org/builders/buildports-elcapitan- x86_64/builds/44/steps/deploy%20archives/logs/stdio) shows error at the very end. Same error occurred for build #45 too. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:39> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Replying to [comment:39 dixiao@…]:
It seems the buildbot stopped uploading compiled packages.
Sorry about that. It's an unrelated issue. I've filed #50668. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:40> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: new Priority: High | Milestone: Component: server/hosting | Version: Resolution: | Keywords: elcapitan Port: | -----------------------------+----------------------- Comment (by ryandesign@…): Replying to [comment:38 ryandesign@…]:
One last attempt to build all ports should hopefully succeed, and upload the packages to the server.
Here it goes: https://build.macports.org/builders/buildports-elcapitan- x86_64/builds/51 -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:41> MacPorts <https://www.macports.org/> Ports system for OS X
#48609: El Capitan buildbot build server teneleven-slave.macports.org -----------------------------+----------------------- Reporter: ryandesign@… | Owner: admin@… Type: enhancement | Status: closed Priority: High | Milestone: Component: server/hosting | Version: Resolution: fixed | Keywords: elcapitan Port: | -----------------------------+----------------------- Changes (by ryandesign@…): * status: new => closed * resolution: => fixed Comment: Finished. Made the buildbot master hang for about 13 minutes at the end, but it recovered on its own. -- Ticket URL: <https://trac.macports.org/ticket/48609#comment:42> MacPorts <https://www.macports.org/> Ports system for OS X
participants (1)
-
MacPorts