<div dir="ltr"><div style="font-family:arial,sans-serif;font-size:13px">I've definitely had success; the main thing to realize is that pbzip2 can only speed up decompression on a file created by pbzip2. (pbzip2 files are still valid bzip2 files.)</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Compression (at least for bzip2) is still slow relative to disk speeds, so parallelizing works very well. As bzip2 is a block compressor (900k chunks by default) it maps to a parallel process very nicely.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Here's an example with a locally compiled (and therefore run through pbzip2 as that is how I have MP set) MacPorts archive.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">MacPro:software$ time pbzip2 -dc clang-3.4/clang-3.4-3.4_0+analyzer+python27.darwin_10.x86_64.tbz2 > /dev/null</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">real<span style="white-space:pre-wrap"> </span>0m12.084s</div><div style="font-family:arial,sans-serif;font-size:13px">
user<span style="white-space:pre-wrap"> </span>0m45.713s</div><div style="font-family:arial,sans-serif;font-size:13px">sys<span style="white-space:pre-wrap"> </span>0m0.534s</div><div style="font-family:arial,sans-serif;font-size:13px">
MacPro:software$ time bzip2 -dc clang-3.4/clang-3.4-3.4_0+analyzer+python27.darwin_10.x86_64.tbz2 > /dev/null</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
real<span style="white-space:pre-wrap"> </span>0m37.223s</div><div style="font-family:arial,sans-serif;font-size:13px">user<span style="white-space:pre-wrap"> </span>0m36.872s</div><div style="font-family:arial,sans-serif;font-size:13px">
sys<span style="white-space:pre-wrap"> </span>0m0.343s</div><div style="font-family:arial,sans-serif;font-size:13px">MacPro:software$ uname -a<br></div><div style="font-family:arial,sans-serif;font-size:13px">Darwin <host> 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386</div>
<div style="font-family:arial,sans-serif;font-size:13px"> </div><div style="font-family:arial,sans-serif;font-size:13px">Note the user time is up (no free lunch; there is some overhead) but real or "wall" time is down by a factor of 3. (I have a 4 core 2007 machine.)</div>
<div style="font-family:arial,sans-serif;font-size:13px">The software autodetects (by default) how many cores to use by default, but you can set with -pN. Do you see all cores being used during compression?</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px"> -- Eric</div><div style="font-family:arial,sans-serif;font-size:13px">[re-sent to list with correct address]</div><div class="" style="font-family:arial,sans-serif;font-size:13px">
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Apr 4, 2014 at 4:49 AM, Ryan Schmidt <span dir="ltr"><<a href="mailto:ryandesign@macports.org" target="_blank">ryandesign@macports.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=""><br>
On Apr 4, 2014, at 03:33, René J.V. Bertin wrote:<br>
<br>
> On Apr 04, 2014, at 10:19, Ryan Schmidt wrote:<br>
><br>
>> While waiting minutes for clang-3.5 to install (compress) and then activate (decompress) a 600MB archive, I wondered why I was sitting here waiting for a single-threaded process to complete when I have a multi-core Mac.<br>
><br>
> Is that 600MB raw, or 600MB when compressed?<br>
<br>
</div>610 MB compressed. clang is enormous.<br>
<div class=""><br>
<br>
>> Has anybody successfully achieved the promised parallel operation of pbzip2 on OS X? If so, I wonder if it depends on the OS X version or the compiler used. I’m on OS X 10.9.2 with Xcode 5.1’s Apple LLVM version 5.1 (clang-503.0.38) (based on LLVM 3.4svn).<br>
><br>
> The correct question to ask is for what cases pbzip2 is faster, if any ... A compressed file is essentially a 1D string that's not segmented like multimedia data (how common is it to use multiple threads to [de]compress audio?). I may be wrong, but for now I'm not at all amazed that parallelisation of uncompressing such data entails a lot of overhead, esp. if it also means letting the disk seek so many times more<br>
<br>
</div>The homepage says "PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines."<br>
<br>
If this didn’t actually work, there would be no reason for the program to exist, but it has, for over a decade.<br>
<div class=""><br>
> (have you tried to compare to [de]compress from one disk to another, or using an SSD?)<br>
<br>
</div>I am using an SSD. I have also tried decompressing to /dev/null, with no change in speed.<br>
<div class="im HOEnZb"><br>
> Also, decompression tends to be so much cheaper than compressing that the parallel overhead will count even more.<br>
<br>
</div><div class="HOEnZb"><div class="h5">_______________________________________________<br>
macports-users mailing list<br>
<a href="mailto:macports-users@lists.macosforge.org">macports-users@lists.macosforge.org</a><br>
<a href="https://lists.macosforge.org/mailman/listinfo/macports-users" target="_blank">https://lists.macosforge.org/mailman/listinfo/macports-users</a><br>
</div></div></blockquote></div><br></div>