[MacPorts] #18259: Improve mportsearch performance
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Attached is a patch which changes how mportsearch works: - now it always reads in all of each PortIndex available in ${sources} - goes through the cached data from that read to find possible matches From a speedup point-of-view, it appears to increase the speed of multiple mportsearches in one run of port by 20%+. For example, running a dry-run install of gnome (port-test is a trunk-based installed with no ports installed): {{{ $ time port-test -dy install gnome > gnome_orig.txt 2>&1 real 1m10.154s user 1m0.102s sys 0m9.089s }}} After the attached patch: {{{ $ time port-test -dy install gnome > gnome_patch.txt 2>&1 real 0m50.689s user 0m43.276s sys 0m6.508s }}} The disadvantage is that memory usage increases; watching top for both of the above tests shows the original peaking at about 344M, the patched at about 419M (so about a 20% increase in memory usage). The debug output from each is identical. As the dependency tree has been increasing lately (use_* adding the proper deps, various X11-based changes) speeding this up is becoming more important. -- Ticket URL: <http://trac.macports.org/ticket/18259> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by jmr@…): There's another approach that I'd like to compare against this. Stay tuned. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:1> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Changes (by jmr@…): * cc: jmr@… (added) Comment: Here's a patch that adds a new "quick index" and a new mportlookup proc, which maps a single port name to its PortIndex entry. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:2> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by jmr@…): Performance comparison: ||Original|| || ||`port outdated`||52.61|| ||`sudo port -y install gnome`||1:19.46|| || || || ||Quick index|| || ||`port outdated`||6.12|| ||`sudo port -y install gnome`||19.31|| -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:3> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by blb@…): To compare against my earlier results, quickindex-array.diff shows: {{{ $ time port-test -dy install gnome > gnome_patch2.txt 2>&1 real 0m26.863s user 0m18.970s sys 0m6.854s }}} using a max of 340M of memory. So faster than my update without the memory cost. Note that this was after the PortIndex.quick file had already been generated, though that definitely doesn't take much time either. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:4> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by blb@…): Added new patch to change handling of quick_index variable and work better with multiple repositories. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:5> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by jmr@…): Updated again: now being smart about not reading the quick index when it's just been generated, added some comments, renamed a couple of variables for clarity. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:6> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by blb@…): Update my version again so it works with multiple repos, basically, quicklist in _mports_load_quickindex needs to be unset if it exists, otherwise the first repo's quicklist is used for the second repo. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:7> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by jmr@…): Update: decompose generation of the quick index into its own proc, and call the new proc from the portindex script. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:8> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance --------------------------------------------+------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: new Priority: Normal | Milestone: MacPorts Future Component: base | Version: 1.8.0 Keywords: search mportsearch performance | Port: --------------------------------------------+------------------------------- Comment(by jmr@…): Update: change PortIndex.quick to have one entry per line, and update PortIndexRegen.sh to include PortIndex.quick. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:9> MacPorts <http://www.macports.org/> Ports system for Mac OS
#18259: Improve mportsearch performance -------------------------------+-------------------------------------------- Reporter: blb@… | Owner: macports-tickets@… Type: enhancement | Status: closed Priority: Normal | Milestone: MacPorts 1.8.0 Component: base | Version: 1.8.0 Resolution: fixed | Keywords: search mportsearch performance Port: | -------------------------------+-------------------------------------------- Changes (by jmr@…): * status: new => closed * resolution: => fixed * milestone: MacPorts Future => MacPorts 1.8.0 Comment: Committed in r47110. -- Ticket URL: <http://trac.macports.org/ticket/18259#comment:10> MacPorts <http://www.macports.org/> Ports system for Mac OS
participants (1)
-
MacPorts