#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+-------------------------- Reporter: devans@… | Owner: ryandesign@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+-------------------------- Comment (by ryandesign@…): I'm not aware of the existence of any network logs. See also #49759 for a similar inexplicable git clone on the buildbot builders. I propose we fix the problem as explained there: slow network fetches should be permitted. MacPorts should print periodic output during slow fetches so that it is apparent something is happening and things are not stalled. -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:2> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+-------------------------- Reporter: devans@… | Owner: ryandesign@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+-------------------------- Comment (by jmr@…): The timeout is 20 minutes. -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:4> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+-------------------------- Reporter: devans@… | Owner: ryandesign@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+-------------------------- Comment (by ryandesign@…): Replying to [comment:3 devans@…]:

Are there specific network protocols that are blocked at Apple?

All ports that are not explicitly allowed are blocked. I understand bzr uses port 4155. That is not on the list of ports that are documented to be allowed, but I don't know if the documentation is up to date. If you give me the hostname or IP address of a bzr server I can check if it can be accessed from the servers. -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:6> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+-------------------------- Reporter: devans@… | Owner: ryandesign@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+-------------------------- Comment (by ryandesign@…): Port 80 is definitely open... -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:8> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by ryandesign@…): Replying to [comment:6 ryandesign@…]:

All ports that are not explicitly allowed are blocked. I understand bzr uses port 4155. That is not on the list of ports that are documented to be allowed, but I don't know if the documentation is up to date.

I think the documentation was out of date. Based on prior internal tickets filed, all buildbot builders should have ports 80 (http), 443 (https), 2401 (cvs), 3690 (svn), 4155 (bzr), 9418 (git) open. Replying to [comment:1 devans@…]:

BTW, I've tried doing an http tarball fetch from Launchpad but for a given bzr revision each fetch yields a different checksum.

This sounds like a launchpad bug. Can you report it to them? -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:10> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by devans@…): Replying to [comment:5 devans@…]:

Replying to [comment:4 jmr@…]:

...

The timeout is 20 minutes. This should be sufficient time for the bzr fetches to complete. Maximum fetch time on local machines is 5 minutes or less.

Actually, I need to update this statement as well. I recently ran a sequence of 10 consecutive inkscape-devel bzr fetches and the time required varied all over the map. Minimum was about 3 minutes with a maximum over 19 minutes. In addition, when running with tcpdump as mentioned above the fetch time was also very near 20 minutes. Given these results, it's quite possible that a fetch on a buildbot VM might well exceed the 20 minute limit. Is it possible to run one or more inkscape-devel fetch by hand on one of the builbots and see how much time it really takes? -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:12> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by ryandesign@…): If we could hook into the MacPorts progress bar API for all revision control system fetches (bzr, cvs, git, hg, svn) that would be super. -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:14> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by devans@…): Replying to [comment:13 ryandesign@…]:

I tested on a gigabit Internet connection in Austin and it fetched 198MB in under 2 minutes. I've taken the Snow Leopard builder offline to test fetching there with the `--verbose` flag, and so far it's fetching at around 50KB/s; at that rate, it should take around an hour to fetch it all.

I just did another fetch here and the download rate varied from a low of 25KB/s to a max around 256KB/s -- nothing more. So I'm assuming Launchpad has a fairly slow connection or their host is really loaded down. In addition, it looks like the bzr protocol is fairly inefficent -- lots of small packets with handshakes in between which doesn't help either.

So as I said in #49759 for git, we should modify MacPorts base so that bzr fetches print something while fetching. It looks like the `--verbose` flag only works in a terminal, so I'm not sure how to make bzr print progress information when called by MacPorts. Worst yet the bzr checkout command doesn't accept --verbose according to the docs. Although it does show a progress bar when connected to a terminal.

-- Ticket URL: <https://trac.macports.org/ticket/49812#comment:16> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by ryandesign@…): Replying to [comment:16 devans@…]:

Worst yet the bzr checkout command doesn't accept --verbose according to the docs.

The manpage says it does: {{{ bzr checkout [BRANCH_LOCATION] [TO_LOCATION] Options: --files-from ARG Get file contents from this tree. --hardlink Hard-link working tree files where possible. --help, -h Show help message. --lightweight Perform a lightweight checkout. Lightweight checkouts depend on access to the branch for every operation. Normal checkouts can perform common operations like diff and status without such access, and also support local commits. --quiet, -q Only display errors and warnings. --revision ARG, -r See "help revisionspec" for details. --usage Show usage message and options. --verbose, -v Display more information. }}}

Although it does show a progress bar when connected to a terminal.

Can we trick it into thinking it has a terminal? -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:18> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: new Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: | Keywords: Port: | -----------------------------+--------------------- Comment (by devans@…): And now inkscape-devel r143503 just fetched successfully on all active buildbots. -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:20> MacPorts <https://www.macports.org/> Ports system for OS X

#49812: buildbots: compile failures due to stalled bzr fetches -----------------------------+--------------------- Reporter: devans@… | Owner: admin@… Type: defect | Status: closed Priority: Normal | Milestone: Component: server/hosting | Version: Resolution: worksforme | Keywords: Port: | -----------------------------+--------------------- Changes (by devans@…): * status: new => closed * resolution: => worksforme Comment: No response. End to end network performance (root cause of this issue) seems to have improved significantly and is much less variable since this ticket was filed. Bzr fetches of imkscape-devel are no longer timing out and complete within a reasonable period of time. Closing as works for me. -- Ticket URL: <https://trac.macports.org/ticket/49812#comment:22> MacPorts <https://www.macports.org/> Ports system for the Mac operating system