Re: [27043] trunk/base/portmgr
macports-dev@lists.macosforge.org on Monday, July 16, 2007 at 10:59 AM -0800 wrote:
--- trunk/base/portmgr/mprsyncup 2007-07-16 15:35:59 UTC (rev 27042) +++ trunk/base/portmgr/mprsyncup 2007-07-16 17:59:44 UTC (rev 27043) @@ -5,11 +5,25 @@ # release tag (as determined by the base/config/RELEASE_URL file) and a ports # tree from trunk (ToT), and then export and sync all of them to the # ${REPOROOT} location, wherefrom the rsync modules are fed to the `sync' -# and `selfupdate' routines in port(1). -# Read the base/portmgr/rsync.repos file for more information on both the -# necessary rsync modules and filesystem level paths, which this script -# bootstraps. +# and `selfupdate' routines in port(1). Read the base/portmgr/rsync.repos +# file for more information on both the necessary rsync modules and filesystem +# level paths, which this script bootstraps. # +# Whatever server uses this script to mirror the MacPorts rsync repositories +# should simply adapt the ${REPOROOT} path variable as necessary (keeping it +# in sync with the equally named variable in base/portmgr/rsync.repos) and +# install it on cron/launchd with a suitable periodicity, previously discussed +# with the portmgr@ team (macports-mgr@lists.macosforge.org). Repositories +# themselves are detailed & served by base/portmgr/rsync.repos, as stated above +# (that is, no manual intervention what-so-ever is needed, other than installing +# this script and adding the repositories detailed in base/portmgr/rsync.repos +# to a local rsyncd.conf file). +# +# Lastly, it is required of every 3rd party mirrors to keep track of this script +# and the base/portmgr/rsync.repos file and always maintain local copies in as +# close sync as possible.
If I understood this better perhaps I could write it up in the new guide. When did this function become available? What do we call it? Remote repository? Is it basically an internal macports repository for the enterprise? Are the comments in mprsyncup and rsync.repos the only documentation we have? Mark
On Jul 16, 2007, at 2:24 PM, markd@macports.org wrote:
If I understood this better perhaps I could write it up in the new guide. When did this function become available? What do we call it? Remote repository? Is it basically an internal macports repository for the enterprise? Are the comments in mprsyncup and rsync.repos the only documentation we have?
Mark
Hi Mark! Thanks for inquiring about this, I was just about to write to you to request inclusion in the new guide ;-) The base/portmgr/mprsyncup and base/portmgr/rsync.repos files explain in their comments all that's needed to replicate our rsync mirrors locally, for instance on a server that wishes to supply local clients with faster sync/ selfupdate functionality off its own rsync server (not having to travel all the way to California if you're around the globe). Note that this is not exactly mirroring, as the example server would be doing the exact same thing rsync.macports.org is doing when setup with those two files, but independently of it (therefore "replicating"). Vlad recently mailed portmgr about the "mirroring" topic, which is what prompted me to better explain the procedure in the files. The two thing I forgot to say to him in my reply is, one, that clients wanting to use the replicating server should be properly configured to do so: -) rsync_server in macports.conf & rsync:// url in sources.conf: reset to point to your local server, not rsync.macports.org -) rsync_dir in macports.conf & path component in rsync:// url in sources.conf: these shouldn't need any modifications if you follow the default and official repositories the MacPorts project offers on its own rsync server (coded in both base/portmgr/mprsyncup and base/ portmgr/rsync.repos); adapt accordingly otherwise (we assume you know what you're doing if you go to the trouble of offering other rsync repositories). And two, anyone wanting to replicate our configuration should consult with portmgr@ on an appropriate periodicity, we don't want to hammer the svn server too bad (every replicating server would be pulling from there). We refresh every 30 minutes at each x:00 & x:30, so we would appreciate it if 3rd parties didn't replicate more often than that and a bit off those two marks in any case. It would be great if all this information (including the comments in the two files in svn) were incorporated into the new guide! Lastly and with respect to the new guide's source: could you, Mark, please split it up in logical xml files (as you asked a while ago and as the current guide exists in svn) rather than having it in just one big monolithic file? Seems to me like it's much easier to handle like that. Regards,... -jmpp PS: Another topic that's up for inclusion in the new guide are the new ticketing guidelines that we're currently discussing on -devel@. I'll try to reply ASAP to the feedback I've received (thanks a bunch!) and will also setup a new Wiki doc (NewTracTicketing) to hold a mockup of the final design and consensus (currently smeared among a few mail posts, hard to track). Once that's done I'd like to see that text fully expanded into the new guide, please ;-), and NewTracTicketing removed, while the existing one (TracTicketing) should be turned into either a URL forward to the guide docs (once they're setup) or simply slimmed down with a single "Read all about it here <url>" line. Thanks!
Juan Manuel Palacios <jmpp@macports.org> on Monday, July 16, 2007 at 1:29 PM -0800 wrote:
Thanks for inquiring about this, I was just about to write to you to request inclusion in the new guide ;-) The base/portmgr/mprsyncup and base/portmgr/rsync.repos files explain in their comments all that's needed to replicate our rsync mirrors locally, for instance on a server that wishes to supply local clients with faster sync/ selfupdate functionality off its own rsync server (not having to travel all the way to California if you're around the globe). Note that this is not exactly mirroring, as the example server would be doing the exact same thing rsync.macports.org is doing when setup with those two files, but independently of it (therefore "replicating").
Got it. Here is my description. Sound good? Are the terms I use ok? "You may setup an rsync replication server on your local network to minimize internet delay and bandwidth when performing MacPorts selfupdates. An rsync replication server pulls the latest rsync data (MacPorts base and port sources) from the remote MacPorts rsync mirrors via selfupdate, and then serves as the rsync source when rsync replication clients on the local network perform selfupdates." 1) What are the functional roles of the scripts mprsyncup and rsync.repos? As far as the server, I don't get REPOROOT as far the scripts. 2) Where do I set REPOROOT? mprsynup only has it in descriptive comments, and it isn't in "Paths we'll work on:", and rsync.repos has no single occurance where it can be set that I can find. Shouldn't there be one line somewhere with REPOROOT = /path/to/default/reporoot to uncomment? Ah wait! Is REPOROOT supposed to be named RSYNCROOT? 3) Does one set rsync.repos to run in cron also, along with mprsyncup? It says I need an rsyncd.conf? Where, how?
See what I have so far in "Using MacPorts" section 3.4. Mark
Hi Mark! Thanks for getting down to this! My amendments: On Jul 17, 2007, at 1:46 AM, markd@macports.org wrote:
Juan Manuel Palacios <jmpp@macports.org> on Monday, July 16, 2007 at 1:29 PM -0800 wrote:
Thanks for inquiring about this, I was just about to write to you to request inclusion in the new guide ;-) The base/portmgr/mprsyncup and base/portmgr/rsync.repos files explain in their comments all that's needed to replicate our rsync mirrors locally, for instance on a server that wishes to supply local clients with faster sync/ selfupdate functionality off its own rsync server (not having to travel all the way to California if you're around the globe). Note that this is not exactly mirroring, as the example server would be doing the exact same thing rsync.macports.org is doing when setup with those two files, but independently of it (therefore "replicating").
Got it. Here is my description. Sound good? Are the terms I use ok?
"You may setup an rsync replication server on your local network to minimize internet delay and bandwidth when performing MacPorts selfupdates. An rsync replication server pulls the latest rsync data (MacPorts base and port sources) from the remote MacPorts rsync mirrors via selfupdate, and then serves as the rsync source when rsync replication clients on the local network perform selfupdates."
"You may setup an rsync replication server on your local network to minimize internet delay and bandwidth when performing MacPorts sync and selfupdate operations. An rsync replication server pulls the latest necessary data (MacPorts release & development sources and the ports tree) from the remote MacPorts svn server via the base/portmgr/mprsyncup script and then serves up the results as the rsync modules used by the replication clients on the local network that perform the sync and selfupdate operations, per the configurations on the base/ portmgr/rsync.repos file." That was both correcting the technicalities and trying to stick to your writing style ;-) Keep the technicalities but feel free to reword it if necessary. One thing that's not included there is my note about periodicity and my urge to discuss it with portmgr prior to setting up a replicant server.
1) What are the functional roles of the scripts mprsyncup and rsync.repos?
mprsycnup is the script that bootstraps the rsync modules off the MacPorts svn server; rsync.repos is an rsyncd.conf mockup configuration file explaining how the replicant rsync server should be setup (actually, rsync.repos details exactly how the main macports rsync server is configured).
As far as the server, I don't get REPOROOT as far the scripts.
2) Where do I set REPOROOT? mprsynup only has it in descriptive comments, and it isn't in "Paths we'll work on:", and rsync.repos has no single occurance where it can be set that I can find. Shouldn't there be one line somewhere with REPOROOT = /path/to/default/reporoot to uncomment? Ah wait! Is REPOROOT supposed to be named RSYNCROOT?
Totally! Just corrected that in my latest commit to the files (r27065), thanks for noticing ;-) mprsyncup does use the variable name for runtime substitution based on the initial definition (line 43), as any bash script would do. But rsync.repos only mentions it in its comments as the rsyncd.conf file does not employ keyword substitution that I know of (would be lovely!); therefore the path is explicit and hardcoded throughout the file. The variable is mentioned, though, in the file comments to emphasize that the hardcoded path in rsync.repos has to be the same as the value of RSYNCROOT in mprsyncup, or otherwise the rsync server will be feeding something different from what the script prepared.
3) Does one set rsync.repos to run in cron also, along with mprsyncup? It says I need an rsyncd.conf? Where, how?
mprsyncup has to run off cron/launchd with a suitable frequency, otherwise the rsync repos become stale rapidly (and we get an unnecessary amount of support requests). rsync.repos provides a working base for a suggested rsyncd.conf file, to which the rsync server has to be pointed (where ever it's installed, whether that's / etc, /usr/local/etc, /opt/local/etc, etc.. ;-)
See what I have so far in "Using MacPorts" section 3.4.
Will do, but maybe tomorrow as it's alraedy 3am here and I got bills to pay, Sir! ;-)
Mark
Thanks again for your hard work on the guide! Regards,... -jmpp
Juan Manuel Palacios <jmpp@macports.org> on Tuesday, July 17, 2007 at 12:14 AM -0800 wrote:
Got it. Here is my description. Sound good? Are the terms I use ok?
"You may setup an rsync replication server on your local network to minimize internet delay and bandwidth when performing MacPorts selfupdates. An rsync replication server pulls the latest rsync data (MacPorts base and port sources) from the remote MacPorts rsync mirrors via selfupdate, and then serves as the rsync source when rsync replication clients on the local network perform selfupdates."
"You may setup an rsync replication server on your local network to minimize internet delay and bandwidth when performing MacPorts sync and selfupdate operations. An rsync replication server pulls the latest necessary data (MacPorts release & development sources and the ports tree) from the remote MacPorts svn server via the base/portmgr/mprsyncup script and then serves up the results as the rsync modules used by the replication clients on the local network that perform the sync and selfupdate operations, per the configurations on the base/ portmgr/rsync.repos file."
I don't like the term rsync modules". Is that standard and/or useful terminology? Is it synonymous with directory? I think the term either needs to be defined or dropped. It is quite confusing. And I think we tend to throw out the term "sync" once in awhile and confuse people. I don't think I've ever used it. I wonder if we should not be emphasizing "port sync" like this. Perhaps it should be relegated the obscurity of a reference section somewhere. Or am I wrong?
That was both correcting the technicalities and trying to stick to your writing style ;-) Keep the technicalities but feel free to reword it if necessary. One thing that's not included there is my note about periodicity and my urge to discuss it with portmgr prior to setting up a replicant server.
I put it in a note: http://homepage.mac.com/duling/macports/guide.html#d0e712
1) What are the functional roles of the scripts mprsyncup and rsync.repos?
mprsycnup is the script that bootstraps the rsync modules off the MacPorts svn server; rsync.repos is an rsyncd.conf mockup configuration file explaining how the replicant rsync server should be setup (actually, rsync.repos details exactly how the main macports rsync server is configured).
Is this a non-traditional use of "bootstrap"? If rsync modules are just directories, I would think there are more common and intuitive words for this. I wonder if "downloads" is ok. I'll have to look at the rest and try to finish the section tomorrow. I haven't finished reading the information you've given me so far. I'd like to get this section free of jargon unless it really has some significant meaning. Thanks for all your help. Mark
Juan, After reading a bit further, two things I would need if I'm going to document this section even in the most minimal form: 1) A list of rsync and subversion terms used therein, or removal of unneccesary special terms. 2) A Howto of how to setup an rsync replication server listing exact steps and not worrying too much about explanation. #1 Do this; #2 do that; etc. It would probably be easier for you to document this section yourself, but before you put too much effort into answering my previous emails I just wanted to let you know that I don't think I'm getting anywhere so far and I'm running out of time to spend on it. Mark
participants (2)
-
Juan Manuel Palacios
-
markd＠macports.org