Juan, I agree with pretty much everything you've written here. On Apr 24, 2007, at 18:15, Juan Manuel Palacios wrote:

First and foremost, I propose we make our use of trunk & branches & tags for base only as explicit as it can be, and therefore reorder the existing hierarchy as follows:

repository/base/ repository/base/trunk repository/base/branches repository/base/tags

Other than base now being a top level dir (emphasized by listing "repository/" in my explanation) and having only MacPorts sources in the directories therein, their usage would still be the self- explanatory svn standard, so nothing would change on that respect.

Excellent.

The ports tree would also become a top level dir in our repo, "repository/ports", packing yet another surprise we could leverage as we see fit and need: multiple ports trees! Some examples:

repository/ports/released/{categories,...} repository/ports/testing/{categories,...}

[snip]

I propose we start with those two trees and fill them up as necessary, the first one being the existing trunk/dports which would *have* to stay in sync with base/branches/release_x_y and the second being a testbed for Portfiles that need unreleased features and syntax to function (ports would be added to it as needed, no need to mirror every single category and port in ports/released if the corresponding Portfile doesn't pack anything different). I would like to make clear that our released Vs. testing split would be with respect to our *Portfiles* and not to the projects we port; that is, emacs and emacs-devel would still be in the released tree as long as their Portfiles are in sync with MacPorts released code. This is contrary to the a la Fink "stable" Vs. "unstable" model, which we all know condemns stable to always be incredibly out of date and pushes everyone to use the unstable tree, while alerting users their computers could catch fire.

Having "released" and "testing" ports trees sound like a good idea.

Getting a bit ahead of myself, Portfiles in the testing tree could be free to declare a different PortSystem value, to signal they need a special MacPorts release to function... but said functionality is still not in our code (Eridius? Landon? ;-)

What is the deal with the PortSystem value? :) When does that get incremented? What happens if/when it does?

Users and committers would be free to suggest the creation of whatever justifiable new tree they find appropriate. Making all of them available through svn and sync (rsync) and whatever else would be immediately beneficial to all, I'm sure it's easy to appreciate.

I can't think of any other trees we would need at this point, and I would even discourage the creation of arbitrary trees at anyone's discretion, without some compelling reason. But at least the directory structure you propose would certainly enable an arbitrary number of trees, should we find we need them.

Such rearrangements to our repo would of course require changes in our source code to upgrade existing installations to leverage them (for instance, the selfupdate procedure hardcodes the path to the sources and therefore would break if we change that under the hood c.f. branches/release_1_4 initially missing the base directory level), but I believe the results will be well worth the effort. What I would like to do here is reach consensus on the best possible repo redesign (which I'm hoping does take place!) so that we see ourselves in the need to upgrade users only once.

Does selfupdate get the new sources directly from the Subversion repository? Somehow I thought rsync was used for that too, and that there was therefore some level of abstraction between the repo and the selfupdate command such that reorganizations of the repository would not pose a problem. If it currently does come directly out of the repo, it might be beneficial to separate that conceptually, so that reorganizations of the repository do not break any future selfupdate procedure. Do we currently have the ability to configure our MacPorts installations such that selfupdate will download and build the latest version from trunk, rather than from a released branch? That's a capability that I think would be good to have (or keep) for developers.

Lastly, the doc and www dirs currently inside trunk could be moved to a "legacy" (detention!) top level dir too (and pulled out of it once brought into the light again ;-)

repository/legacy/docs/{www,faq,guide}

Moving www, faq and guide to a legacy directory sounds like a good idea to me, since it makes it clear that these parts are no longer in active development nor do they represent the current state of, for example, the web site. I don't think there should be a "docs" directory, however. I recommend: repository/legacy/faq repository/legacy/guide repository/legacy/www

PS: Yes, I'm also emphasizing the new ports directory should be called just that, "ports", and not "dports" nor "mports" :-P

Agreed.

On Apr 25, 2007, at 9:02 AM, Ryan Schmidt wrote:

...

I propose we start with those two trees and fill them up as necessary, the first one being the existing trunk/dports which would *have* to stay in sync with base/branches/release_x_y and the second being a testbed for Portfiles that need unreleased features and syntax to function (ports would be added to it as needed, no need to mirror every single category and port in ports/ released if the corresponding Portfile doesn't pack anything different). I would like to make clear that our released Vs. testing split would be with respect to our *Portfiles* and not to the projects we port; that is, emacs and emacs-devel would still be in the released tree as long as their Portfiles are in sync with MacPorts released code. This is contrary to the a la Fink "stable" Vs. "unstable" model, which we all know condemns stable to always be incredibly out of date and pushes everyone to use the unstable tree, while alerting users their computers could catch fire.

Having "released" and "testing" ports trees sound like a good idea.

I disagree. What's the point of putting portfiles in testing? Who's going to move them to release when they are ready? How installation will pick them? Releases are already a big pain, which explains why we have them so infrequently. Making this more pain will only make things worse. I think we should postpone the versioning of the Portfile APIs and condition it to the existence of a remote repository (through mpwa for example). And until we have continuous integration, I'm in favor of simply dropping the very idea of releases. The 1.4.x series show that base/ is globally extremely stable, that major failures are introduced by active developers who are able to fix them very quickly. It also proved how wrong the release candidate workflow is. Active users who might test release candidates and trunk/ do not run 10.3 (this means that testing time is infinite), while we are able to fix bugs in base/ and ship fixes within 48 hours (this means the update time is epsilon). And I definitely do not understand the interest of changing the repository layout, except for the joy of requiring every developer who has uncommitted code to move their files and fix their code manually to adapt to the new layout and names. Juan, I realize that I'm just proposing to simply cancel most of your job within the project. And I realize this is partly why you are so reluctant from dropping releases, you consider your job as the member of portmgr responsible for releases is to be conservative against us, the liberal developers. Well, from my liberal developer's point of view, your contribution to the project is mostly beyond this perceived job and your GSoC proposal seemed much more useful for the project than this new repository layout idea. But this is just my opinion, and I do not mean to dictate anyone's behavior. Paul

On Apr 25, 2007, at 7:25 AM, Paul Guyot wrote:

On Apr 25, 2007, at 9:02 AM, Ryan Schmidt wrote:

...

...

I propose we start with those two trees and fill them up as necessary, the first one being the existing trunk/dports which would *have* to stay in sync with base/branches/release_x_y and the second being a testbed for Portfiles that need unreleased features and syntax to function (ports would be added to it as needed, no need to mirror every single category and port in ports/ released if the corresponding Portfile doesn't pack anything different). I would like to make clear that our released Vs. testing split would be with respect to our *Portfiles* and not to the projects we port; that is, emacs and emacs-devel would still be in the released tree as long as their Portfiles are in sync with MacPorts released code. This is contrary to the a la Fink "stable" Vs. "unstable" model, which we all know condemns stable to always be incredibly out of date and pushes everyone to use the unstable tree, while alerting users their computers could catch fire.

Having "released" and "testing" ports trees sound like a good idea.

I disagree. What's the point of putting portfiles in testing? Who's going to move them to release when they are ready? How installation will pick them?

The testing ports tree would just be a facility offered by the project to developers who, like you, develop new features in trunk and/or bug fixes and, naturally, want and need to test them in Portfiles. Such Portfiles break an MP installation based on released code as we've seen one too many times already, so we can't have them in the shipping ("released") ports tree as they are useless there. Here's where my "testing" tree proposal comes in, it's name (or whatever is more descriptive of its purpose) being simply a label claiming "these portfiles are meant to work with trunk, so don't complain if you get errors". As I explained in my original message, this tree would not be some kind of Secret Service Agent keeping a detailed eye on each of our Portfiles (that is, "not another road block to get things into users hands"), some kind of prerequisite every single Portfile would have to meet and go through before being considered "stable" and moved to the "released" tree after some approval is granted.... no, not at all, not at all what I meant. On the contrary, consider "testing" as a tree open to our entire developer community for testing MacPorts itself, putting in Portfiles therein whatever you want and need, thus lifting every single restriction you may ever encounter with Portfiles in "released". We would not ship the "testing" tree by default, just make it available for those knowledgeable enough to configure their sources to fetch it. We would not (or at least that's not my intent) hold Portfiles in "testing" with different (newer) versions for the project they install with respect to the corresponding Portfile in "released", the difference between the trees simply being the features each Portfile sports: emacs and emacs-devel would both be in "released" as long as they both work with a released version of MP; one of them, whichever, could be duplicated into the "testing" tree to test a new *MacPorts* feature. I can't stress this enough, this is not an a la Fink "stable" Vs. "unstable" trees proposal. Also, the "testing" tree would not mirror "released" either with respect to every single category and portdir in the latter, they would simply be created in the former as needed. If you want to test a new MP feature in one of your Portfiles you would be free to do whatever you need in the "testing", and otherwise there would be no need to add it there, it would simply live in "released", as every single other Portfile. In short, "released" is our existing dports tree renamed, "testing" is where anyone would be free to play in an open fashion. If you keep a local tree to play with new features in trunk, your testing would be limited to you and anyone you send your development Portfiles to; if you put them in dports things break, as they already have; if you make them available to everyone through an alternate tree where you can feel free to play as you like, wouldn't that, at least potentially, translate into wider testing? To put it in other words, nothing of what we now have would change..... I'm just proposing we *also* offer something new, an alternate and public tree for the purpose of testing new MP features and syntax.

Releases are already a big pain, which explains why we have them so infrequently.

Not recently,we've done 4 releases in less than a month. I am aware I personally wasn't too convinced about it in the first place, but I didn't oppose it either and your point that we could/should do more frequent releases was proven.

Making this more pain will only make things worse.

Nothing of what I am proposing here slows down the release process in any way. As already explained, the "testing" tree is not a requirement our Portfiles have to go through to only then cascade down into "released", so why would it make things worse?

I think we should postpone the versioning of the Portfile APIs and condition it to the existence of a remote repository (through mpwa for example).

The versioning of the API is something that could help us in many ways, I believe, but I agree this could be a different thread and that it could be postponed for various reasons (though I don't understand the ones you explain here). As for mpwa, I admit that I haven't done my homework in looking into it, but other than that I don't have a single idea about its roadmap. Tomorrow? Next month? MacPorts 2.0? What I'm proposing is something simple and easy that can help us *now* solve a problem we've had for long.

And until we have continuous integration, I'm in favor of simply dropping the very idea of releases.

As I said at one point already, I get the feeling you forget what a pain the days of users using ToT were, how reduced our user base was because of that among other reasons and how difficult providing support was. I have to be adamant about this, I am completely opposed to the idea of dropping releases. Do we need to streamline our release process even further....? Yes, I agree and I would like to see it happen. Should someone quicker than me step in to do releases? I wouldn't be opposed to that... and, in fact, wasn't, as James already did. Dropping releases as some kind of panacea for our problems? Couldn't disagree more!

The 1.4.x series show that base/ is globally extremely stable, that major failures are introduced by active developers who are able to fix them very quickly. It also proved how wrong the release candidate workflow is.

I disagree that the release candidate workflow is "wrong". That it doesn't serve us in the best way possible because we don't have a large enough testing community? I agree to that, but that does not mean the workflow is inherently flawed. Hey, testing MacPorts is in itself a difficult task, as it implies to some extent testing as many ports as possible, and that's a problem we're still to solve, albeit a somewhat different one.

Active users who might test release candidates and trunk/ do not run 10.3 (this means that testing time is infinite), while we are able to fix bugs in base/ and ship fixes within 48 hours (this means the update time is epsilon).

Testing on Panther is difficult and a pain, yes. I was practically the only 10.3 committer and tester and, unfortunately, not any longer because I recently upgraded to Tiger. I'm getting the feeling we'll need to start thinking about dropping Panther support sooner than latero, as many Open Source projects already have. This is, however, a different problem, I believe.

And I definitely do not understand the interest of changing the repository layout, except for the joy of requiring every developer who has uncommitted code to move their files and fix their code manually to adapt to the new layout and names.

trunk/base makes no sense for us, that's an svn adaptation of something we inherited from the old cvs layout that does not apply any longer. We're only versioning base, not our ports nor www nor docs, so why have all that in trunk? It's confusing at best. I say pull 'em out of there and put them elsewhere, as already suggested. We'd end up having only trunk/base, but then again, we're only versioning base! So why have {trunk,branches,tags} only for base? Makes more sense to have base/{trunk,branches,tags}, thus emphasizing base is the only thing we version. Am I an order freak? Yes, I am, that's no secret. But then again, I think we can all agree order is only beneficial. Can we envision the need of ever versioning anything else? Yes? OK, lets keep trunk/{base,something_else} and {branches,tags} parallel to trunk. If not... what's the use of the now somewhat gratuitous extra (and loner) "base" directory inside trunk to get to our sources? I don't see it. No, my motivation is not disturbing developers simply for the joy of it, not by far.

Juan, I realize that I'm just proposing to simply cancel most of your job within the project. And I realize this is partly why you are so reluctant from dropping releases

You are assuming that, I did not say nor imply that anywhere, and you're wrong. I am reluctant to dropping releases for a myriad of other, valid, reasons.

you consider your job as the member of portmgr responsible for releases is to be conservative against us, the liberal developers.

One of my responsibilities is to insure our shipping product meets at least some quality standard, not to counterbalance anyone per se.

Well, from my liberal developer's point of view, your contribution to the project is mostly beyond this perceived job and your GSoC proposal seemed much more useful for the project than this new repository layout idea. But this is just my opinion, and I do not mean to dictate anyone's behavior.

I do not either, I appreciate highly all the work you do for the project and would never try to block or oppose it. On the contrary, what I'm trying here is to find better ways to streamline the workflow for our entire developer community, within certain parameters (as open as possible) that allow us to ensure necessary traits like quality control, versioning base being one of them.

Paul

-jmpp PS: Indexing the "testing" tree is a secondary problem that can be solved easily with minor tweaks to the IndexRegen.sh script.