[MacPorts] #24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python ----------------------------------------------------+----------------------- Reporter: vkuznet@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: python crash, urllib2, multiprocessing | Port: ----------------------------------------------------+----------------------- Hi, I observed very tricky problem. I have application written in python which uses multiprocessing module. So using the python 2.6 from macports cause python die in a thread when urllib2 module is invoked. To test I installed stock python 2.6.5 from python.org and everything works. I also verify that the problem lies with urllib2/multiprocessing modules, since if I replace urllib2 to httplib, everything works too. I attach apple traceback which shows the problem. My guess that macports uses either special patch or enable flags which didn't come with stock python source and that cause a problem. Valentin. -- Ticket URL: <http://trac.macports.org/ticket/24421> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Changes (by jmr@…): * keywords: python crash, urllib2, multiprocessing => * cc: mcalhoun@… (added) * port: => python26 * owner: macports-tickets@… => blb@… Comment: Please remember to fill in the Port field and cc the maintainers. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:1> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by jmr@…): Looks like _scproxy.so is passing a null pointer to SystemConfiguration. I don't see anything in the patches that looks like it would cause this. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:2> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by jmr@…): Except SCDynamicStoreCopyProxies is meant to work with a NULL argument, which raises the possibility of this being a bug in SystemConfiguration or CF. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:3> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by blb@…): Do you have a simple example to reproduce this? -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:4> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by vkuznet@…): I tried to make stand-alone version, but so far I can't reproduce the problem with it. Is it possible to get configuration/patches which went into mac-port. If I can re-apply all configuration steps into the one I download from python.org I can have a look if I can reproduce the problem with manual build of python. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:5> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by vkuznet@…): I don't know if it helps, I found that _scproxy.so is linked differently for python from macports and the stock one. Here is the stock python _scproxy.so otool -L /Users/vk/Work/Python/Python-2.6.5//install/lib/python2.6/lib- dynload/_scproxy.so /Users/vk/Work/Python/Python-2.6.5//install/lib/python2.6/lib- dynload/_scproxy.so: /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 293.4.0) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.19.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 246.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1) while the macports version is otool -L $PWD/lib-dynload/_scproxy.so /opt/local/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6 /lib-dynload/_scproxy.so: /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 293.4.0) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.19.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1) Actually all _XXX.so from macport build are missing /usr/lib/libgcc_s.1.dylib link. Also I commented out the if sys.platform == 'darwin': ... in urllib.py in macport version and my code works just fine when there is no _scproxy.so objects are loaded, while it crash if _scproxy.so is there. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:6> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by blb@…): The way MacPorts builds python26 (and the patches applied) can be see in the trac browser for [browser:trunk/dports/lang/python26 the python26 port] (patches are in the files/ subdir). Note that not linking to libgcc_s.1.dylib should be fine since, at least on 10.6, it is just a symlink to libSystem anyway: {{{ $ ls -l /usr/lib/libgcc_s.1.dylib lrwxr-xr-x 1 root wheel 17 Apr 16 18:54 /usr/lib/libgcc_s.1.dylib -> libSystem.B.dylib }}} -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:7> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by blb@…): Looking at your crash log, I see {{{ ... 0x1007f2000 - 0x1007f6ff7 +_ssl.so ??? (???) <BC4937DC-E257-4B12-88BA-58678A48F931> /Users/vk/CMS/COMP/DAS/mylib/_ssl.so ... 0x1012c9000 - 0x101308fff +libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <98F9160D-1D42-9189-CE14-204FDB2B62E7> /Users/vk/Work/openssl-0.9.8k/lib/libssl.0.9.8.dylib 0x10131c000 - 0x10143bfe7 +libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <4BB4F468-D80A-B3A1-AE43-DB544191E975> /Users/vk/Work/openssl-0.9.8k/lib/libcrypto.0.9.8.dylib }}} Not sure if that would be causing the crash since it's in SSL and not where the crash is happening but it might be an interesting experiment to see if using the MacPorts-provided SSL module crashes as well. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:8> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by vkuznet@…): The openssl module was compiled intentionally, since in our environment we still use renegotiation feature on front-end, which is disabled by default since 0.9.8m release of OpenSSL. In addition, both python (stock one and from mac ports) used my local module, so it should not be a problem. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:9> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by scott.moynes@…): I am also experiencing this bug. Downgrading to python26 version 2.6.4 has corrected it for me. I will work on a simple reproduction recipe. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:10> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: blb@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by craig.hawco@…): As a workaround setting any environment variable ending with "_proxy" to a non-empty value prevents urllib from calling SystemConfiguration, and thus prevents this particular crash. That does not resolve this central issue. -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:13> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: mcalhoun@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Changes (by blb@…): * owner: blb@… => mcalhoun@… * cc: mcalhoun@… (removed) -- Ticket URL: <http://trac.macports.org/ticket/24421#comment:14> MacPorts <http://www.macports.org/> Ports system for Mac OS
#24421: python dies using multiprocessing/urllib2 from mac ports, but works with stock python -------------------------------+-------------------------------------------- Reporter: vkuznet@… | Owner: mcalhoun@… Type: defect | Status: new Priority: Normal | Milestone: Component: ports | Version: 1.8.2 Keywords: | Port: python26 -------------------------------+-------------------------------------------- Comment(by jmr@…): Does 2.6.6 still have this problem? In any case, it's pretty hard to do anything about it without a test case. -- Ticket URL: <https://trac.macports.org/ticket/24421#comment:15> MacPorts <http://www.macports.org/> Ports system for Mac OS
participants (1)
-
MacPorts