[MacPorts] #50969: Provide a generic extract (and compress) commands in base
#50969: Provide a generic extract (and compress) commands in base -------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Keywords: | Port: -------------------------+---------------------- It would be extremely useful to introduce a command like {{{ file_extract foo.tar.xz }}} in the core, so that ports could even "manually" extract (or compress) additional files when needed. At the moment MacPorts supports fetching distfiles from different mirrors, but it doesn't support extracting different filetypes, so all the "hard work" of extracting the proper file has to be repeated manually in the Portfile. It might also be useful to be able to extract additional tarballs when fetching from git/svn, so the command would be helpful in many different context. MacPorts could also have the ability to automatically use different extract methods based on filenames (that is: when file extension is specified; `use_xz` would then only influence the default type that could theoretically be overwritten). (The other nice-to-have functionality would be the ability to create (reproducible) tarballs on demand. But maybe that's for another ticket.) -- Ticket URL: <https://trac.macports.org/ticket/50969> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Description changed by mojca@…: Old description:
It would be extremely useful to introduce a command like {{{ file_extract foo.tar.xz }}} in the core, so that ports could even "manually" extract (or compress) additional files when needed.
At the moment MacPorts supports fetching distfiles from different mirrors, but it doesn't support extracting different filetypes, so all the "hard work" of extracting the proper file has to be repeated manually in the Portfile. It might also be useful to be able to extract additional tarballs when fetching from git/svn, so the command would be helpful in many different context.
MacPorts could also have the ability to automatically use different extract methods based on filenames (that is: when file extension is specified; `use_xz` would then only influence the default type that could theoretically be overwritten).
(The other nice-to-have functionality would be the ability to create (reproducible) tarballs on demand. But maybe that's for another ticket.)
New description: It would be extremely useful to introduce a command like {{{ file_extract foo.tar.xz }}} in the core, so that ports could even "manually" extract (or compress) additional files when needed. At the moment MacPorts supports fetching distfiles from different mirrors, but it doesn't support extracting different filetypes, so all the "hard work" of extracting the proper file has to be repeated manually in the Portfile. It might also be useful to be able to extract additional tarballs when fetching from git/svn, so the command would be helpful in many different context. MacPorts could also have the ability to automatically use different extract methods based on filenames (that is: when file extension is specified; `use_xz` would then only influence the default type that could theoretically be overwritten). (The other nice-to-have functionality would be the ability to create (reproducible) tarballs on demand as requested in #50896.) -- -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:1> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by rjvbertin@…):
At the moment MacPorts supports fetching distfiles from different mirrors, but it doesn't support extracting different filetypes, so all the "hard work" of extracting the proper file has to be repeated manually in the Portfile.
Shouldn't that read "would have to be repeated manually in the Portfile if an error wasn't raised by the automatic extraction process" and you don't rewrite the extract block yourself? I'd add a nice-to-have feature, at least until "base" provides an integrated way of fetching files in multiple formats: `file_fetch <name> [site]` that would do an on-demand fetch of the named file, looking it up on the `master_sites` or else from the given `site`. The site argument might be redundant if adding unused sites to master_sites has no negative side- effects. As far as I can see that would mostly require refactoring portfetch::fetchfiles from `portfetch.tcl` slightly, and exporting the procedure for extracting a single file. Here's an example from a work-in-progress Portfile of mine where such a function would come in handy: {{{ post-fetch { if {![file exists ${distpath}/${qtwkdistfile}]} { ui_msg "---> Fetching qtwebkit" # find a better way to fetch the archive with some form of optional progress # report that doesn't scroll system -W ${distpath} "wget --progress=bar:force:noscroll --quiet --show-progress https://github.com/qtproject/qtwebkit/archive/${qtwebkit_commit}/qtwebkit- snapshot.tar.gz" # give the file a more evocative name file rename ${distpath}/qtwebkit-snapshot.tar.gz ${distpath}/${qtwkdistfile} } } }}} I don't see how the automatic fetch/extract mechanism could achieve this - it would require syntax to tell fetch to save a given distfile under a different name, and extract to use that new name. I'd still need a manual extract, but that's probably required anyway (and not just because using hfsCompression really makes a difference here): {{{ post-extract { if {![file exists ${distpath}/${qtwkdistfile}]} { ui_error "This shouldn't happen" } elseif {![file exists ${worksrcpath}/qtwebkit/.git] && [file exists ${distpath}/${qtwkdistfile}]} { ui_msg "---> Extracting ${qtwkdistfile}" if {[file exists ${prefix}/bin/bsdtar]} { system -W ${worksrcpath} "bsdtar -x -k --hfsCompression -f ${distpath}/${qtwkdistfile}" } else { system -W ${worksrcpath} "tar -xf ${distpath}/${qtwkdistfile}" } file rename ${worksrcpath}/qtwebkit-${qtwebkit_commit} ${worksrcpath}/qtwebkit file mkdir ${worksrcpath}/qtwebkit/.git } } }}} -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:2> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by rjvbertin@…): Ahem, after squinting some more at portfetch.tcl I realised that there is already a 'curl' command that's available directly in Tcl, so I can replace my wget above with {{{ if {[llength [info commands ui_progress_download]] > 0 } { curl fetch --remote-time --progress ui_progress_download \ https://github.com/qtproject/qtwebkit/archive/${qtwebkit_commit}/qtwebkit- snapshot.tar.gz \ ${distpath}/${qtwkdistfile} } else { curl fetch --remote-time --progress builtin \ https://github.com/qtproject/qtwebkit/archive/${qtwebkit_commit}/qtwebkit- snapshot.tar.gz \ ${distpath}/${qtwkdistfile} } }}} Still, I'd say that something like this would be nice to have: {{{ proc file_fetch {url dest} { if {[llength [info commands ui_progress_download]] > 0 } { curl fetch --remote-time --progress ui_progress_download \ ${url} ${dest} } else { curl fetch --remote-time --progress builtin \ ${url} ${dest} } } }}} -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:4> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by mojca@…): I agree that if we add `file_extract`, `file_compress`, ... adding `file_fetch` would be a a natural follower. But if I understand correctly the only reason why you need to rewrite the fetching phase yourself in this particular example is because you want to rename the file afterwards to prevent clashes? I usually worked around that by using `dist_subdir`, but again `dist_subdir` is global to the port, so if your ports need one huge file that doesn't change (but is properly versioned) and another small file that doesn't change the name (perhaps it changes just the URL) and keeps changing on regular basis, using `dist_subdir` is very suboptimal. Supporting renaming of files is not a completely unreasonable request. In particular not in the context of adding more flexibility like different suffixes / extract methods, files with the same name from different URLs, very unfortunate filenames, fetching files to different dist_subdir-s, ... who knows what else. But it would require a different syntax. -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:5> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by rjvbertin@…): No, in my current case there is also the issue that the main file uses a different compression scheme, so putting the additional file in distfiles will cause the extraction phase to fail. However, one can think of other scenarios where one would want to download additional file(s) outside of the regular mechanism mostly designed for source files. Configuration, templates, data, there are many kinds of additional things a port could want to provide automatically when it is (re)installed, that are kept up-to-date upstreams but for which the installed software doesn't provide an integrated updating mechanism. I admit, it's still a bit far-fetched. OTOH, almost all the work has been done, cf. my example `file_fetch` procedure above. -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:6> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by mojca@…): I have another example of a need for "`post-fetch`". A [http://bugzilla- geant4.kek.jp/show_bug.cgi?id=1849 project] forgot to compress the files (the files have a correct name `foo-version.tar.gz`, but they are in fact tar files). A solution would be to either adapt the extract for that single file or to compress the file after it has been fetched and store the compressed version only (under the same name as the file that has just been fetched). This is somewhat similar to René's case with the need to rename file after it has been fetched. -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:7> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract (and compress) commands in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by rjvbertin@…): Replying to [comment:7 mojca@…]:
This is somewhat similar to René's case with the need to rename file after it has been fetched.
Yes, and my case used to be even more similar. Before I discovered the github mirror where one can fetch a tarball, I actually had to do a manual git clone in the post-fetch. And to avoid the associated overhead I archived that check-out so that a rebuild would be able to unpack a tarball in the post-extract phase. -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:8> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract command in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Changes (by raimue@…): * cc: raimue@… (added) -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:9> MacPorts <https://www.macports.org/> Ports system for OS X
#50969: Provide a generic extract command in base --------------------------+---------------------- Reporter: mojca@… | Owner: raimue@… Type: enhancement | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | --------------------------+---------------------- Comment (by raimue@…): Example usage of the simplified distfiles option on macports-dev: https://lists.macosforge.org/pipermail/macports-dev/2016-March/032671.html -- Ticket URL: <https://trac.macports.org/ticket/50969#comment:10> MacPorts <https://www.macports.org/> Ports system for OS X
participants (1)
-
MacPorts