On Jun 11, 2007, at 02:05, paul beard wrote:
Still wondering why my distfiles have to come from so far away.
Because nobody has programmed (or discussed) an alternative. I'm Cc'ing the -dev list, where discussions of this nature (developing MacPorts) should probably take place. When you reply, please edit -users out of the addressees and let's continue this discussion only on -dev.
I would create a more formal feature request but I can't access anything at MacOSForge.
Clearly that's a separate issue, but we should address that too. Please tell us (on -users) what problems you're having. Before we file a feature request in Trac, though, we should discuss on the mailing list what the feature should do.
Is there some way for the lists of mirrors to be associated with the user's timezone (gdate +%Z value) so that the load can be distributed around more evenly, users get distfiles faster, and mirror sites at the top of the list don't get pounded?
{prefix}/share/darwinports/resources/port1.0/fetch/mirror_sites.tcl Actually for some of these, it could be done by hemisphere or continent ;-)
set portfetch::mirror_sites::sites(gnome) { http://mandril.creatis.insa-lyon.fr/linux/gnome.org/ http://mirror.aarnet.edu.au/pub/GNOME/ http://ftp.unina.it/pub/linux/GNOME http://fr.rpmfind.net/linux/gnome.org/ http://fr2.rpmfind.net/linux/gnome.org/ http://ftp.acc.umu.se/pub/GNOME/ http://ftp.belnet.be/mirror/ftp.gnome.org/ http://ftp.linux.org.uk/mirrors/ftp.gnome.org/ http://ftp.nara.wide.ad.jp/pub/X11/GNOME/ http://ftp.gnome.org/pub/GNOME/ ftp://mirror.pacific.net.au/linux/GNOME ftp://ftp.dataplus.se/pub/GNOME/ ftp://ftp.dit.upm.es/pub/GNOME/ ftp://ftp.no.gnome.org/pub/GNOME/ ftp://ftp.isu.edu.tw/pub/Unix/Desktop/GNOME/ ftp://ftp.nara.wide.ad.jp/pub/X11/GNOME/ ftp://ftp.chg.ru/pub/X11/gnome/ ftp://ftp.kddlabs.co.jp/pub/GNOME/ ftp://ftp.dti.ad.jp/pub/X/gnome/ }
I think the suggestion has merit in principle, though I wouldn't necessary do it by time zone. SourceForge has implemented something like this, where you access one central URL to download anything, and it redirects you to a nearby mirror, based on your IP. I don't know if they do it by country or what, but I know there are databases (free ones, even) that will tell you what country an IP address is in, so that's a reasonable way to do it. One solution would be to have a script (like a PHP script*) available on www.macports.org which we could call to download files from our various master_sites groups (like the gnome group you've quoted above) with a URL like: http://example.macports.org/download/from/gnome/sources/pango/1.16/ pango-1.16.4.tar.bz2 http://example.macports.org/download/from is the download script gnome is the group sources/pango/1.16/pango-1.16.4.tar.bz2 is the path used on servers in that group In this scenario, the list of gnome mirrors (and other groups from mirror_sites.tcl) would migrate to this "download" script, which would also know (or automatically find out) what country each mirror is in. And for each incoming request, it would find out the visitor's IP and direct them to one of the mirrors in their country. We obviously also need a solution for when there is no mirror in the visitor's country. We probably need a table that says which countries are near which other countries. Maybe such data can also be downloaded for free. If we can even just get lat/long coordinates for each country, MySQL 5 has support for spacial queries which may help us find other nearby servers. I guess this would be one aspect where doing it by time zone might simplify things... Finding out the visitor's time zone is easy (on the client side), and finding out what time zones are near other time zones is also easy. Finding out which time zone a given server is in might also be possible... though I can't off the top of my head think of an automated way... The headers the servers return seem to always be GMT... Also, I'm not sure how much load this would add to www.macports.org. And I'm not sure if it would be seen as a bad thing that the mirror list becomes a server-side thing, not a client-side thing. Though to me it seems like a reasonable good idea. As expressed above, the idea would only handle the common master_sites groups. It wouldn't deal with other ports that define several master_sites. A different form of the above idea would be a download URL that just says which port and which file to download, with a URL like: http://example.macports.org/download/pango/pango-1.16.4.tar.bz2 The "download" script would read the portfile and get its list of master_sites, so this would work both for portfiles that use master_sites groups and those that define the sites individually. For each master_site, determine its IP, and from that, its country (and cache this). Get the visitor's IP, and thus country, compare, find match, redirect for download. Same caveat as above regarding visitors that don't have a mirror in their country. There's probably a way to turn this around and do it a more "MacPorts"-like way where the country data is either embedded into the PortIndex or into another file that like the PortIndex gets automatically updated at some interval. This way clients aren't constantly pinging the MacPorts web site to get each file's download location; they'd already know. A less-automated solution could be that for each master_site defined in master_sites.tcl and for each master_site defined in each portfile, one must specify the country code. The port command can then determine what country the user is in (e.g. by pinging a simple script on the server which returns the country code -- e.g. http:// example.macports.org/country.php -- and this can be cached on the client side for some interval -- or since we only support Mac OS now, we can find a way to look up wherever the Date & Time system preferences stores its data) and see which mirror to download from. I think I don't like this option because it places too great a burden on the port authors to determine and specify each master_site's country. Since we can figure it out, we should do so, and not make the port author do it. I'm just throwing ideas out there. Any others? Any comments on my ideas? What's good, what's bad? * I just say PHP script because I like to write PHP scripts. It could of course be in any language.