[MacPorts] #50708: Base should support fetching git submodules (git submodule update)
#50708: Base should support fetching git submodules (git submodule update) ---------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Keywords: | Port: ---------------------+-------------------------------- If ports fetched via git (or via the github portgroup) have submodules, those submodules are left out (and should usually be fetched via `git submodule update`). It would be great if the base would provide a straightforward way to also fetch the submodules automatically. Examples: `multimarkdown` and `git-flow` ports use {{{ # https://github.com/fletcher/MultiMarkdown-4/issues/7 fetch.type git post-fetch { system -W ${worksrcpath} "git submodule update --init" } }}} and {{{ post-fetch { # Does the github portgroup support 'clone --recursive'? # This will be removed once the python re-write is complete. system -W ${worksrcpath} "git submodule update --init" } }}} [https://github.com/RJVB/macstrop/blob/master/devel/qt5-creator/Portfile René] used an even more complex approach. See also #16373. -- Ticket URL: <https://trac.macports.org/ticket/50708> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by rjvbertin@…): I cannot remember exactly why I ended up with the complex approach, and whether it had to do with peculiarities of the remote git repo. I've already encountered several occasions where the latest submodule commit wasn't compatible with the latest Qt-Creator commit and I had to use an earlier commit. I do remember though that I couldn't determine with certainty what revision you get when doing a `submodule update --init`. The whole approach does appear to be geared towards getting the latest commit. Which makes sense, but not so much so for MacPorts. If we use git to fetch sources for a port, the assumption is that we grab a known version ("commit") which we have tested sufficiently to be reasonably sure it work at least as good as the latest official release. That should also apply to submodule checkouts, which is why in the end I went with an approach which more or less treats the submodule as an independent repo that just happens to be checkout out under another working copy. -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:1> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by mojca@…): Indeed. Submodules should be specified together with the shasum. -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:3> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by ryandesign@…): Sounds like the current instructions for how to use submodules in the github portgroup are insufficient, because they don't mention using a shasum for submodule updates. If so, please update the instructions. Many ports follow those instructions and will need updates too. -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:4> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by mojca@…): Do you man the following free text in `github-1.0.tcl`? {{{ # Some projects' tag- or commit-based distfiles will not contain all the # necessary files, if the project uses git submodules. If available, use a # distfile from "releases" or "downloads" instead, as described above. If the # project does not provide those, encourage the project's developers to provide # releases. Until they do, fetch from git instead of from a distfile, and add a # post-fetch block to fetch the submodules: # # fetch.type git # # post-fetch { # system -W ${worksrcpath} "git submodule update --init" # } }}} I would need to study this a bit more. After reading the first answer from http://stackoverflow.com/questions/1777854/git-submodules-specify-a -branch-tag I'm even more confused about whether or not the commit is fixed or not and whether that depends on the version of git being used by developers. (And perhaps also double-check the situation in SVN.) -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:5> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by rjvbertin@…): Understand now why I use the approach I use? Also, if you have to do part of the fetching explicitly anyway, using a git(hub) PortGroup uses part of its interest (but leaves the additional parsing overhead). -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:6> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by mojca@…): I keep reading the git documentation and it seems to me that submodules are well-defined rather than arbitrary commits from the included third- party repository. (The question is: what if some port maintainers urgently need a different commit than what upstream envisioned?) Now it seems that once a user has a sufficiently recent `git` available (which is already the case for MacPorts), it could be enough to use {{{ git clone --recursive ${git.url} }}} or rather: I don't currently see a problem with `"git submodule update --init"`. The question is then: are the any reasons why these steps shouldn't be executed by default? Or perhaps with something like {{{ git.submodules yes }}} could trigger fetching the submodules. I tried to read your code, René, but I don't yet completely understand what you tried to do / which problems you were facing. -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:7> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by cal@…): Submodules are always referenced using a well-defined commit. Checking out a different commit in a submodule isn't supported. For Github, we usually prefer HTTP downloads instead of fetching with git. It would be nice if that also worked for repositories with submodules. See Homebrew's approach of resources and "install resource" calls for the target directories (this example is with git trees, but this also works with HTTP tarballs): https://github.com/Homebrew/homebrew/blob/master/Library/Formula/v8.rb#L86-L... -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:8> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by rjvbertin@…): The problem I've faced at several occasions is that a simple checkout of a known commit followed by the standard submodule fetch left me with a mismatch between the main and the sub modules. So what I do now is a `git clone --recursive` followed by a `git checkout -q ${git.branch}` in the working copy. That takes care of the main source tree. In order to get the known & tested submodule version I need to do some extra work in the post- fetch. If we ignore the reinplace in the .git/config file which is there for other reasons, I do `git submodule update --init ; git submodule update` in the toplevel working copy followed by a series of git commands in the submodule working copy that leave me with the known commit. It might be possible to do that in less than the 3 commands I'm using, but if so I haven't yet found out how. Part of this complexity may stem from the fact that I've been trying to support the use of a local working copy as the git, something I clearly dropped when it became impossible. I probably ought to revisit the submodule post-fetch code to see if it cannot be simplified, but that's got a bit too much of a fixing something that ain't broke thing (and the repo is big so takes a bit too much time to check out repeatedly). -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:9> MacPorts <https://www.macports.org/> Ports system for OS X
#50708: Base should support fetching git submodules (git submodule update) ----------------------+-------------------------------- Reporter: mojca@… | Owner: macports-tickets@… Type: defect | Status: new Priority: Normal | Milestone: Component: base | Version: 2.3.4 Resolution: | Keywords: Port: | ----------------------+-------------------------------- Comment (by cal@…): Whenever you change the version of the base repository, you need to run `git submodule update`, preferably with the `--recursive` and `--init` flags, just in case there are nested submodules, or submodules have been added/removed. So yes, it is expected behavior that you get a mismatch if you do {{{ git checkout $branch git submodule fetch }}} because you never did a `submodule update`. Fetch operations never change the state of a working copy in git. So for a proper implementation in MacPorts, we need: 1. git clone 2. git checkout $yourbranch_hash_or_tag 3. git submodule update --init --recursive `git clone --recursive` does not buy as anything, because the state of the submodules at our target commit `$yourbranch_hash_or_tag` can be different from origin/master, which is what the recursive clone would give us. We cannot use `git clone -b $branch_or_tag --recursive` either, because that doesn't work for commit hashes. -- Ticket URL: <https://trac.macports.org/ticket/50708#comment:10> MacPorts <https://www.macports.org/> Ports system for OS X
participants (1)
-
MacPorts