[MacPorts] #25450: Ports fail to fetch from Japanese ftp mirro
#25450: Ports fail to fetch from Japanese ftp mirro ---------------------------------+------------------------------------------ Reporter: a.lathrop@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.9.1 Keywords: fetch GNU Japan | Port: gsl ---------------------------------+------------------------------------------ I live in Japan. When attempting to install/upgrade GNU packages/dependencies (ie. gsl, or gettext), I get: ---> Attempting to fetch gsl-1.14.tar.gz from ftp://ftp.dti.ad.jp/pub/GNU/gsl and the program hangs there, without timing out. I can go the the ftp site via Firefox, and manually download the package to /opt/local/var/macports/distfiles, and retry, and it works fine. Something must be wrong with the ftp site, since other sources work fine, but it must also be a problem with macports, since Firefox is able to access the ftp site without issue. -- Ticket URL: <http://trac.macports.org/ticket/25450> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ---------------------------------+------------------------------------------ Reporter: a.lathrop@… | Owner: jochen@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.9.1 Keywords: | Port: gsl ---------------------------------+------------------------------------------ Changes (by jmr@…): * owner: macports-tickets@… => jochen@… * keywords: fetch GNU Japan => Comment: Are you using a proxy? Are you able to fetch this file using curl with FTP_PROXY set? Snow Leopard's version of libcurl has a bug related to handling of proxy environment variables ([rdar://problem/7997030], [https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2913886&group_i... upstream]), so you may need to disable proxy use for FTP. -- Ticket URL: <http://trac.macports.org/ticket/25450#comment:1> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ---------------------------------+------------------------------------------ Reporter: a.lathrop@… | Owner: jochen@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.9.1 Keywords: | Port: gsl ---------------------------------+------------------------------------------ Comment(by a.lathrop@…): Not using a proxy. -- Ticket URL: <http://trac.macports.org/ticket/25450#comment:2> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ---------------------------------+------------------------------------------ Reporter: a.lathrop@… | Owner: macports-dev@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 1.9.1 Keywords: fetch | Port: gsl, gettext ---------------------------------+------------------------------------------ Changes (by jochen@…): * keywords: => fetch * owner: jochen@… => macports-dev@… * component: ports => base * port: gsl => gsl, gettext Comment: Not sure why this was assigned to me, looks more like a base issue to me. -- Ticket URL: <http://trac.macports.org/ticket/25450#comment:3> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ---------------------------------+------------------------------------------ Reporter: a.lathrop@… | Owner: macports-dev@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 1.9.1 Keywords: fetch | Port: gsl, gettext ---------------------------------+------------------------------------------ Comment(by jxzhu@…): Replying to [comment:1 jmr@…]:
Are you using a proxy? Are you able to fetch this file using curl with FTP_PROXY set? Snow Leopard's version of libcurl has a bug related to handling of proxy environment >variables ([rdar://problem/7997030], [https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2913886&group_i... upstream]), so you may need to disable >proxy use for FTP.
This should be related to the ticket (#25857) which was created by myself and is now closed. My system is behind the proxy. The Fetch via FTP always fails on my machine with Snow Leopard 10.6.4. But I don't have problem to open the ftp site by put the ftp link on a web browser. Since I am relatively newer to MacPorts, I would appreciate if you can tell me in more detail How to fetch the file using curl with FTP_PROXY set? [I mean a list of procedures (commands) to get, e.g., the file hdf5-1.8.5.tar.bz2 from ftp://ftp.hdfgroup.org/HDF5/current/src/ ]. Also how can I disable the proxy use for FTP? Right now, I have in sudoers file Defaults env_keep += "http_proxy HTTP_PROXY HTTPS_PROXY FTP_PROXY RSYNC_PROXY" Defaults env_keep += "ALL_PROXY NO_PROXY" and in my .cshrc setenv RSYNC_PROXY proxy.xxx.xxx:8080 #setenv FTP_PROXY proxy.xxx.xxx:8080 As you notice, I have commented out the line for FTP_PROXY environment variable. But it doesn't help. Thanks. -- Ticket URL: <http://trac.macports.org/ticket/25450#comment:4> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ----------------------------------+----------------------------------------- Reporter: a.lathrop@… | Owner: macports-dev@… Type: defect | Status: closed Priority: Normal | Milestone: Component: base | Version: 1.9.1 Resolution: worksforme | Keywords: fetch Port: gsl, gettext | ----------------------------------+----------------------------------------- Changes (by jmr@…): * status: new => closed * resolution: => worksforme Comment: Can't reproduce; MacPorts fetches fine from this site for me. -- Ticket URL: <https://trac.macports.org/ticket/25450#comment:5> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ----------------------------------+----------------------------------------- Reporter: a.lathrop@… | Owner: macports-dev@… Type: defect | Status: reopened Priority: Normal | Milestone: Component: base | Version: 1.9.1 Resolution: | Keywords: fetch Port: gsl, gettext | ----------------------------------+----------------------------------------- Changes (by florian@…): * status: closed => reopened * resolution: worksforme => Comment: I can confirm and reproduce the ftp hang by setting up a local squid proxy (port install squid): 1) run tcpdump: {{{ tcpdump -n -i lo0 host 127.0.0.1 and port 3128 }}} 2) in a 2nd terminal do this and observe tcpdump: {{{ http_proxy=0.0.0.0:0 FTP_PROXY=127.0.0.1:3128 port fetch openconnect # this should hang }}} 3) for comparison use curl: {{{ FTP_PROXY=127.0.0.1:3128 curl -O ftp://ftp.infradead.org/pub/openconnect/openconnect-3.14.tar.gz # this works }}} This is the output of tcpdump: A) port (2): {{{ 15:33:51.913101 IP 127.0.0.1.65288 > 127.0.0.1.3128: Flags [S], seq 3179921709, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862465 ecr 0,sackOK,eol], length 0 15:33:51.913163 IP 127.0.0.1.3128 > 127.0.0.1.65288: Flags [S.], seq 1404851936, ack 3179921710, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862465 ecr 114862465,sackOK,eol], length 0 15:33:51.913177 IP 127.0.0.1.65288 > 127.0.0.1.3128: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862465 ecr 114862465], length 0 15:33:51.913190 IP 127.0.0.1.3128 > 127.0.0.1.65288: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862465 ecr 114862465], length 0 # hangs here }}} B) curl (3): {{{ 15:33:12.245787 IP 127.0.0.1.65286 > 127.0.0.1.3128: Flags [S], seq 3483046350, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862069 ecr 0,sackOK,eol], length 0 15:33:12.245872 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [S.], seq 35148451, ack 3483046351, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862069 ecr 114862069,sackOK,eol], length 0 15:33:12.245888 IP 127.0.0.1.65286 > 127.0.0.1.3128: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 0 15:33:12.245902 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 0 # until now same as above but continues: 15:33:12.246458 IP 127.0.0.1.65286 > 127.0.0.1.3128: Flags [P.], seq 1:271, ack 1, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 270 15:33:12.246494 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [.], ack 271, win 56855, options [nop,nop,TS val 114862069 ecr 114862069], length 0 15:33:12.246794 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [P.], seq 1:340, ack 271, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 339 15:33:12.246837 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [P.], seq 340:4436, ack 271, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 4096 15:33:12.246863 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [P.], seq 4436:8532, ack 271, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 4096 [...] }}} Could this be a libcurl bug? ---- It is impossible to explicitly disable/set a proxy in macports.conf if proxies are set in sys preferences. Can this option be added? E.g.: {{{ proxy_ftp no }}} or {{{ proxy_ftp disable }}} Instead of proxy_override_env these methods would be much better: {{{ proxy_ftp env # use environment proxy_ftp sys # use sys preferences proxy_ftp host:port # set proxy here }}} Then we have better control of how a proxy is set. Thanks! -- Ticket URL: <https://trac.macports.org/ticket/25450#comment:6> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ----------------------------------+----------------------------------------- Reporter: a.lathrop@… | Owner: macports-dev@… Type: defect | Status: reopened Priority: Normal | Milestone: Component: base | Version: 1.9.1 Resolution: | Keywords: fetch Port: gsl, gettext | ----------------------------------+----------------------------------------- Comment(by florian@…): Here are some hints how to resolve this problem: 1) I traced down the problem to a buggy libcurl (7.19.7) which hangs on the call of select() in function Curl_socket_ready(). This is reproducible with the steps outlined in my previous post. This is the backtrace: {{{ #0 0x00007fff85e9d932 in select$DARWIN_EXTSN () #1 0x0000000100068bbd in Curl_socket_ready () #2 0x000000010003a9f2 in ftp_easy_statemach (conn=0x100800408) at ftp.c:3056 #3 0x000000010003ad28 in ftp_connect (conn=0x100800408, done=0x7fff5fbfecee) at ftp.c:3186 #4 0x0000000100044cb5 in Curl_protocol_connect (conn=0x100800408, protocol_done=0x7fff5fbfecee) at url.c:3070 #5 0x0000000100047ea8 in setup_conn (conn=0x100800408, hostaddr=0x100115338, protocol_done=0x7fff5fbfecee) at url.c:4711 #6 0x0000000100048024 in Curl_connect (data=0x100803c08, in_connect=0x7fff5fbfed20, asyncp=0x7fff5fbfecef, protocol_done=0x7fff5fbfecee) at url.c:4787 #7 0x0000000100059d5f in connect_host (data=0x100803c08, conn=0x7fff5fbfed20) at transfer.c:2488 #8 0x0000000100059ff7 in Curl_perform (data=0x100803c08) at transfer.c:2625 #9 0x000000010005aa9f in curl_easy_perform (curl=0x100803c08) at easy.c:557 #10 0x000000010000b2ee in operate (config=0x7fff5fbff350, argc=3, argv=0x7fff5fbff6a8) at main.c:5002 #11 0x000000010000bcbc in main (argc=3, argv=0x7fff5fbff6a8) at main.c:5317 }}} 2) Snow Leopard comes with libcurl 7.19.7 so this fails: {{{ http_proxy=0.0.0.0:0 FTP_PROXY=127.0.0.1:3128 /usr/bin/curl -v ftp://ftp.infradead.org/pub/openconnect/openconnect-3.15.tar.gz > /dev/null }}} 3) The bug is apparently fixed in the libcurl that the curl port installs. So the macport installed curl works as expected: {{{ http_proxy=0.0.0.0:0 FTP_PROXY=127.0.0.1:3128 /opt/local/bin/curl -v ftp://ftp.infradead.org/pub/openconnect/openconnect-3.15.tar.gz > /dev/null }}} 4) '''port fetch''' calls ''curl fetch'' from '''Pextlib''', however, Pextlib is linked against the buggy libcurl that comes with Snow Leopard: {{{ otool -L /opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib /opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib: Pextlib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libcurl.4.dylib (compatibility version 6.0.0, current version 6.1.0) /usr/lib/libedit.2.dylib (compatibility version 2.0.0, current version 2.11.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11) }}} 5) '''Conclusion''': The easiest fix would be to link Pextlib against the libcurl (currently 7.23.1) from macports and not rely on a 2 years old libcurl that comes with the system. The current version includes lots of bug fixes and probably several known security vulnerabilities. So I see no point in using the old one. -- Ticket URL: <https://trac.macports.org/ticket/25450#comment:7> MacPorts <http://www.macports.org/> Ports system for Mac OS
#25450: Ports fail to fetch from Japanese ftp mirror ---------------------------+---------------------------- Reporter: a.lathrop@… | Owner: macports-dev@… Type: defect | Status: closed Priority: Normal | Milestone: Component: base | Version: 1.9.1 Resolution: worksforme | Keywords: fetch Port: gsl, gettext | ---------------------------+---------------------------- Changes (by jmr@…): * status: reopened => closed * resolution: => worksforme -- Ticket URL: <https://trac.macports.org/ticket/25450#comment:8> MacPorts <http://www.macports.org/> Ports system for Mac OS
participants (1)
-
MacPorts