Proposal for a new repository layout

[Juan Manuel Palacios]

	Hello Ryan, good to see some feedback! ;-)

On Apr 24, 2007, at 8:02 PM, Ryan Schmidt wrote:

>
>
> What is the deal with the PortSystem value? :) When does that get  
> incremented? What happens if/when it does?


	The PortSystem declaration is (was) meant to be a floating point  
value stating what "version" of the "port API", the portal to the  
library that parses and executes the Portfiles, is needed to do  
exactly that, parse and execute a given Portfile. However, the only  
value that was ever offered (when DarwinPorts first started) is 1.0  
(I think at some point, maybe before going public, a 0.5 value also  
existed), which is why you see every single Portfile with that value  
and a base/src/port1.0 directory in svn. Another proposal I've  
discussed on some opportunities with people like Kevin, James and  
Landon is putting that functionality to work to effectively start  
using the PortSystem declaration, designing a strategy to bump its  
value as needed (in tandem with the library version) and catching it  
when first parsing a Portfile to determine if the installed version  
of MacPorts can handle it. The base/doc/INTERNALS file also explains  
some of what I've written here.

>
>> 	Users and committers would be free to suggest the creation of  
>> whatever justifiable new tree they find appropriate. Making all of  
>> them available through svn and sync (rsync) and whatever else  
>> would be immediately beneficial to all, I'm sure it's easy to  
>> appreciate.
>
> I can't think of any other trees we would need at this point, and I  
> would even discourage the creation of arbitrary trees at anyone's  
> discretion, without some compelling reason. But at least the  
> directory structure you propose would certainly enable an arbitrary  
> number of trees, should we find we need them.


	Agreed on all of this. We would not be creating arbitrary trees at  
will, not at all, but actually only after compelling reasons for them  
are provided and wide consensus is reached. I only said users would  
be free to "suggest" a given tree ;-)

>
>> 	Such rearrangements to our repo would of course require changes  
>> in our source code to upgrade existing installations to leverage  
>> them (for instance, the selfupdate procedure hardcodes the path to  
>> the sources and therefore would break if we change that under the  
>> hood c.f. branches/release_1_4 initially missing the base  
>> directory level), but I believe the results will be well worth the  
>> effort. What I would like to do here is reach consensus on the  
>> best possible repo redesign (which I'm hoping does take place!) so  
>> that we see ourselves in the need to upgrade users only once.
>
> Does selfupdate get the new sources directly from the Subversion  
> repository?


	Nope, both sync and selfupdate get their feeds from an rsync repo.

> Somehow I thought rsync was used for that too, and that there was  
> therefore some level of abstraction between the repo and the  
> selfupdate command such that reorganizations of the repository  
> would not pose a problem.


	Yes and no. An explanation is in order, I feel. Rsync repos used by  
MacPorts still live on OpenDarwin servers, sampson I believe, where  
the base/portmgr/mprsyncup script is installed and working day in and  
day out (it's a rudimentary script that could see many improvements,  
agreed, but I would only delve into improving it if we do go forward  
with all these reorganizations I'm proposing). Said script checks out  
all of trunk and puts it in a given path that the rsync server makes  
available as the dpupdate module; it then does some sh & svn magic  
and checks out the tag listed in base/config/RELEASE_URL (listing our  
official release) and makes available at another path that the rsync  
server publishes as the dpupdate1 module. That's why we ship  
"dpupdate1/base" as the default value of the 'rsync_dir' key in  
ports.conf (used by the selfupdate target) and "dpupdate/dport" as  
the default value in sources.conf (used by both sync and selfupdate).  
Switching to "dpupdate/base" for 'rsync_dir' will update you to trunk  
through selfupdate, but that's as much flexibility as we currently  
have. For one, changing these values server side in any way would  
surely mandate changing our default conf values to follow the new  
modules, and of course providing a migration path for existing  
installations. Secondly, the "seflupdate" target is rather fragile in  
its coding; from base/src/darwinports1.0/darwinports.tcl, line No. 1501:

if { [catch { system "cd $dp_base_path && ./configure --prefix= 
$prefix --with-install-user=$owner --with-install-group=$group --with- 
tclpackag
e=$tclpackage && make && make install" } result] }

	Where $dp_base_path is set to (line No. 1438):

set dp_base_path [file join $prefix var/db/dports/sources/rsync.$ 
{rsync_server}_${rsync_dir}/]

	Therefore, on a default MacPorts installation, $dp_base_path maps  
to /opt/local (${prefix}) + var/db/dports/sources/ +  
rsync.rsync.darwinports.org (rsync.${rsync_server}) + _dpupdate1/base  
(_${rsync_dir}), "/opt/local/var/db/dports/sources/ 
rsync.rsync.darwinports.org_dpupdate1/base/" in all. The selfupdate  
procedure cd's into that dir and looks for the configure script to do  
its magic, meaning that if the exact path changes in any way the  
whole thing breaks irrecoverably (I here again refer to how the  
release_1_4 branch had to be recreated for selfupdate to work after I  
omitted the "base" directory level by mistake when first creating  
it). With the redesigned repo we would no longer have trunk/base but  
actually base/{trunk,branches,tags}, so paths could become mangled in  
various ways we might not even be aware of (we're not even  
considering how the rsync server on sampson exports the modules,  
since we don't have access to that conf file). Other than migrating  
users to new paths and rsync moduels, I would like to have this  
inherent fragility corrected in an upgraded selfupdate procedure.


> If it currently does come directly out of the repo, it might be  
> beneficial to separate that conceptually, so that reorganizations  
> of the repository do not break any future selfupdate procedure.


	Addressed above.

>
> Do we currently have the ability to configure our MacPorts  
> installations such that selfupdate will download and build the  
> latest version from trunk, rather than from a released branch?  
> That's a capability that I think would be good to have (or keep)  
> for developers.


	Addressed above.

>
>> 	Lastly, the doc and www dirs currently inside trunk could be  
>> moved to a "legacy" (detention!) top level dir too (and pulled out  
>> of it once brought into the light again ;-)
>>
>> repository/legacy/docs/{www,faq,guide}
>
> Moving www, faq and guide to a legacy directory sounds like a good  
> idea to me, since it makes it clear that these parts are no longer  
> in active development nor do they represent the current state of,  
> for example, the web site.


	A good point I wasn't too clear on when first posting, thanks for  
expanding on it.

>
> I don't think there should be a "docs" directory, however. I  
> recommend:
>
> repository/legacy/faq
> repository/legacy/guide
> repository/legacy/www


	There already exists a docs dir (trunk/docs), but I made a mistake  
when making my proposal for this. I wrote "repository/legacy/docs/ 
{www,faq,guide}" when it should have been "repository/legacy/ 
{docs,www,faq,guide}". Thanks for pointing me to my mistake!


>
>> PS: Yes, I'm also emphasizing the new ports directory should be  
>> called just that, "ports", and not "dports" nor "mports" :-P
>
> Agreed.


	Again, thanks for the feedback! Regards,...


-jmpp