I'm having real problems with up2date since upgrading from FC2 to FC3t2.
There was the usual flood of new packages after the release of FC3t2 and up2date seemed to be totally incapable of downloading them and installing. Worse, up2date would show that packages needed to be installed, but couldn't find them when asked to update them, or would show nothing needing to be updated when there were over 175 packages needing updating.
The icon in the notification area is currently displaying as red and shows that 55 packages need updating, but if I click on the icon and then click Launch up2date (which shows 55 packages needing updates) and then click through, up2date says my system is up to date (it isn't)
It's trying to use yum channel fedora-core-rawhide from http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/ for the updates.
Rodd
On Sep 22, 2004 at 22:28, Rodd Clarkson in a soothing rage wrote:
I'm having real problems with up2date since upgrading from FC2 to FC3t2.
[...]
It's trying to use yum channel fedora-core-rawhide from http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/ for the updates.
When it tries to dl from http://download.fedora.redhat.com stop and start it again after a while so that it chooses a different mirror. Else change your /etc/sysconfig/rhn/sources to use different mirrors.
N.Emile...
On Wed, 22 Sep 2004 22:28:35 +1000, Rodd Clarkson rodd@redfishbluefish.com.au wrote:
The icon in the notification area is currently displaying as red and shows that 55 packages need updating, but if I click on the icon and then click Launch up2date (which shows 55 packages needing updates) and then click through, up2date says my system is up to date (it isn't)
It's trying to use yum channel fedora-core-rawhide from http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/ for the updates.
I bet you also have an active yum-mirror fedora-core-rawhide line as well. Assuming you do have a yum-mirror line active, what are are most likely seeing is what happens mirrors are out of sync with the master download server. The panel applet( rhn-applet-gui) sees there are updates available from the main server i believe. But when up2date is run, up2date does some sort of selection among the mirrors in the mirrorlist ( http://fedora.redhat.com/download/up2date-mirrors/fedora-core-rawhide ) chooses a mirror to try to download packages from and then attempts to download the packages.
So what i think you are seeing is a side-affect of the fact that different mirrors sync with the main download server at different rates and times. If this is a case, this is just one example of how hard it is to do reasonable mirror selection without requiring information to flow from a mirror back to a centralized location about the sync status of each mirror, or by requiring the end-user to make a selection on the client-side to choose mirrors based on past experience.
We could have many, long and fruitless discussion (and I have) about how to do mirror-selection better given the current constraints on how the mirrors are setup. So before someone decides to run off on a tagent about how to do it better than up2date does it right now. I'm going to set down the constraints that define the current mirror structure. If you start a discussion that does not live with in the boundaries of these constraints you're fantasizing about an unimplementable utopia that does not match the current political and technical realities which you can't change. And i will publicly point and laugh at you and call you names if you attempt to do so.
1) the mirror networks are NOT peer-to-peer nodes. Anything that requires a mirror to run a specialized daemon is not going to be acceptable. One of http/ftp/rsync are the only services mirrors need to choose to run.
2) No mirror-side scripts to communicate back to a centralized location to report that the mirror has started or is done syncing to the master server. No information is "pushed" from a mirror back to the master mirror or to another centralized location. Any statistics gathering you might think of doing to try to determine if mirrors are in sync with the master mirror needs to be done by a seperate server and not services implemented at each mirror.
3) No scheme can rely on red hat specifically implementing a new centralized service or providing the infrastructure for a new mirror-selection service. If you do come up with a clever way to provide accurate list of the mirrors that are in sync with the master mirror, build it on a community controlled site and get people testing it with up2date. There is no reason up2date's yum-mirror has to take a redhat.com address as an argument.
-jef
Once upon a time, Jeff Spaleta jspaleta@gmail.com said:
But when up2date is run, up2date does some sort of selection among the mirrors in the mirrorlist ( http://fedora.redhat.com/download/up2date-mirrors/fedora-core-rawhide ) chooses a mirror to try to download packages from and then attempts to download the packages.
How is this mirror list created? Looking at it right now, there are only 6 servers listed (the Red Hat master plus 5 mirrors); aren't there more mirrors than that that carry the rawhide tree? Checking the other up2date-mirrors files, there are a few more mirrors listed but not many.
How does up2date (or yum) handle a mirror not having what is being looked for? For example, mirror.hiwaay.net is in the list, but I don't mirror all the rawhide architectures (just i386 and x86_64). Does up2date recognize a 404 and try another mirror? Is this action remembered?
One problem with syncing rawhide to mirrors is that the daily build and push of rawhide to the master servers takes a different amount of time (and so finishes at a different time) each day. Some days, my sync finishes before the push is complete, so until my mirror syncs again (once a day), it will be effectively fubared. One thing I've considered doing is to change my mirroring scripts to watch for actual file syncs and loop (with a small delay) until the sync runs without changing any files. That would help, but there's still a period where my mirror of the tree is not in sync. I guess if I synced the header.info and repodata files last and then deleted, I should always have a consistent tree, but scripting that will be somewhat of a PITA.
On Wed, 29 Sep 2004 09:56:27 -0500, Chris Adams cmadams@hiwaay.net wrote:
How is this mirror list created?
I honestly don't know how the default mirror lists are created. I suspect the developer just created them by hand using some personal experience associate with which mirrors were reliable.
How does up2date (or yum) handle a mirror not having what is being looked for?
I think they both handle lack of a tree somewhat gracefully and failover. I think neither handle a tree being out of sync with the master very well.
So if a mirror doesn't have a specific tree i think they failover to the next mirror in the list. What "next" means i think is different between yum and up2date. yum im pretty sure marches down the list. up2date might "pick" another mirror from the list somehow,
For example, mirror.hiwaay.net is in the list, but I don't mirror all the rawhide architectures (just i386 and x86_64). Does up2date recognize a 404 and try another mirror? Is this action remembered?
I don't think there is any cleverness about remembering if mirrors have gone sour and removing them locally from the client. You have to be careful with this sort of logic. A specific mirror could fail for a temporary reason, so trying to be clever and remember that it failed could prevent you from ever using that mirror again just because it had a short period downtime event once.
One problem with syncing rawhide to mirrors is that the daily build and push of rawhide to the master servers takes a different amount of time (and so finishes at a different time) each day. Some days, my sync finishes before the push is complete, so until my mirror syncs again (once a day), it will be effectively fubared.
Okay this sort of problem could maybe be preventable with small changes on the master server, if there was a way to implement a start/stop file on the master servers that mirrors could check to see if a rawhide push was still in process and delay starting a sync attempt until the rawhide push on the master server was done. But even if the small technical change on the master server was made, all the mirrors would have to write/use scripts to take advantage of the change and there is no way you can enforce mirror admins to do that across the board. Its the politics of individual mirror admins. So while you might use this to make your mirror more reliable... i don't think you can count on all admins to do it.
One thing I've considered doing is to change my mirroring scripts to watch for actual file syncs and loop (with a small delay) until the sync runs without changing any files. That would help, but there's still a period where my mirror of the tree is not in sync. I guess if I synced the header.info and repodata files last and then deleted, I should always have a consistent tree, but scripting that will be somewhat of a PITA.
The heart of the problem... anything that is going to make mirrors significantly more reliable is going to end up being a solution that puts a heavy burden on the mirror admins. PITA scripts or extra services that communicate back or whatever. The maximum reliability in terms of how out of sync an invidiual mirror can get basically comes down to how much care and feeding individual mirror maintainers are willing to do. So even if you get scripts in place on your mirror that work to keep your mirror synced with the master mirror, its not clear to me that you can get equal effort from the other mirrors who are volunteering their bandwidth to host the packages.
And if we can't get all the mirrors to be reliable, then the only other way to do it, is to have some scheme that dynamically checks for out of synced mirrors and ranks the reliability of mirrors so that over time, clients who are connecting to mirrors get a preferred set and never set a set that is known to be out of sync. But, i have not seen a scheme that doesn't require an admin to run extra services or scripts, that is garunteed to catch a mirror being out of sync and prevent a client computer from contacting a mirror that is out of sync.
-jef
Folks,
Forgive my jumping into what looks like an increasingly unproductive discussion but I have an idea that might help all involved:
Problems that need to be solved: 1) Not enough community mirrors. 2) Tester demand for up to the minute Rawhide updates.
Potential two part answer: a) Point yum to a local repository. b) Fill that repository via a daily torrent of upgraded Rawhide RPMs. Change the .torrent document daily. Probably a straightforward scripting problem.
Benefits to Redhat: 1) Torrents are, probably, a lower load on your server. 2) Users contribute to solving the problem instead of whining about dead mirrors. 3) Eventually move all Fedora users to this model. Make it the default.
Benefits to Fedora users: 1) Get relatively rapid access to new RPMs. 2) Simple but valuable contribution of bandwidth to the Fedora cause. I know I would feel better getting my updates via a torrent. 3) Yum delays from overloaded servers are a thing of the past.
Thoughts?
Best Regards, Andrew ____________________________________ Andrew W. Donoho awd@DDG.com, PGP Key ID: 0x81D0F250 +1 (512) 453-6652 (o), +1 (512) 750-7596 (m)
Potential two part answer: a) Point yum to a local repository. b) Fill that repository via a daily torrent of upgraded Rawhide RPMs.Change the .torrent document daily. Probably a straightforward scripting problem.
A torrent of ALL of rawhide is 50+GB of disk space.
You'll never be able to sync it all and most people will be fairly pissed about eating up that much space.
Benefits to Redhat:
- Torrents are, probably, a lower load on your server.
- Users contribute to solving the problem instead of whining
aboutdead mirrors. 3) Eventually move all Fedora users to this model. Make it the default.
Benefits to Fedora users:
- Get relatively rapid access to new RPMs.
once you get passed all the overhead involved.
- Simple but valuable contribution of bandwidth to the Fedora cause.I
know I would feel better getting my updates via a torrent.
How do you separate out your updates from all the other crap?
-sv
On Sep 30, 2004, at 12:20, seth vidal wrote:
Potential two part answer: a) Point yum to a local repository. b) Fill that repository via a daily torrent of upgraded Rawhide RPMs.Change the .torrent document daily. Probably a straightforward scripting problem.
A torrent of ALL of rawhide is 50+GB of disk space.
Please read what I wrote - "a daily torrent of upgraded Rawhide RPMs". Nothing in that statement says anyone has to keep them beyond the day they came out. Nor does it say that every user needs to keep a full archive of Rawhide.
You'll never be able to sync it all and most people will be fairly pissed about eating up that much space.
See the above comment.
Benefits to Redhat:
- Torrents are, probably, a lower load on your server.
- Users contribute to solving the problem instead of whining
aboutdead mirrors. 3) Eventually move all Fedora users to this model. Make it the default.
Benefits to Fedora users:
- Get relatively rapid access to new RPMs.
once you get passed all the overhead involved.
This is no different than the delay getting to download updated RPMs today. Or are you still misreading my above statements and assuming that I proposed a full backup of Rawhide to each Fedora user?
- Simple but valuable contribution of bandwidth to the Fedora cause.I
know I would feel better getting my updates via a torrent.
How do you separate out your updates from all the other crap?
This will require some scripting on the Redhat side of the repository but it is something they fully control - yum, up2date, Fedora Distribution, Rawhide update policies. I think this meets all or most of Mr. Spaleta's constraints and provides benefits to Redhat and users.
Andrew
____________________________________ Andrew W. Donoho awd@DDG.com, PGP Key ID: 0x81D0F250 +1 (512) 453-6652 (o), +1 (512) 750-7596 (m)
Please read what I wrote - "a daily torrent of upgraded Rawhide RPMs".Nothing in that statement says anyone has to keep them beyond the daythey came out. Nor does it say that every user needs to keep a fullarchive of Rawhide.
A daily torrent only works if everyone is in sync on a day-to-day basis. Otherwise they'll need to get in sync with any missed days.
Moreover they'll need ALL the packages as an example:
day1: pkgx is shipped day2: pkgy is shipped day3: pkgy is updated with a requirement on pkgx
now if you're only getting updates and you're not keeping them where do you get pkgx from?
This is no different than the delay getting to download updated RPMstoday. Or are you still misreading my above statements and assumingthat I proposed a full backup of Rawhide to each Fedora user?
This will require some scripting on the Redhat side of the repositorybut it is something they fully control - yum, up2date, FedoraDistribution, Rawhide update policies.
I beg to differ with you. Red Hat does NOT control yum development and with no offense to the people at red hat, they've shown remarkably little mirror guidance or leadership.
I think this meets all or mostof Mr. Spaleta's constraints and provides benefits to Redhat and users.
Except for the incredible overhead of running a torrent seed on each client. People always forget that part.
-sv
On Thu, 30 Sep 2004 12:46:15 -0500, Andrew W. Donoho awd@ddg.com wrote:
This will require some scripting on the Redhat side of the repository but it is something they fully control - yum, up2date, Fedora Distribution, Rawhide update policies. I think this meets all or most of Mr. Spaleta's constraints and provides benefits to Redhat and users.
<point and laugh mode> HeHe you said torrent HeHe </point and laugh mode>
Constraint one... the mirror system is not a p2p network. That constraint was aimed directly at anyone suggesting bittorrent as a way forward.
I frankly don't care much about how well rawhide works. I believe everything about rawhide from how packages are pushed to the main server to how people access mirrors is inherently unstable, fragile and prone to breaking. Testers using rawhide should expect breakage and be prepared for it. Building any service meant primarily to satify the demands of access to rawhide is a waste of development manhours.
I care far far more about finding a way to give each client computer a way to get a reasonable accurate list of mirrors that works well and reliably for access to released updates and base packages for full releases. If discussion isn't framed around this being the central goal, I'm not overly interested in it.
-jef"hehe rawhide torrent hehe..thats a good one"spaleta
I care far far more about finding a way to give each client computer a way to get a reasonable accurate list of mirrors that works well and reliably for access to released updates and base packages for full releases. If discussion isn't framed around this being the central goal, I'm not overly interested in it.
IMHO much of the weirdness with up2date and to a lesser degree with yum stems from mirror selection issues. The best solution currently seems to require that each OP hand edit his/her relevant configuration file(s) to point to more "desirable" mirrors. I've found up2date to work poorly at best since FC1; note that I don't so much mind breakage in the test releases since there are typically in excess of 50 package updates a day.
Perhaps a solution similar to the one fedora.us has implemented with their version of apt could be applied to up2date/yum namely the first time the app is run the OP gets a TUI that allows for mirror selections. Painless & effective.
Hoping this suggestion falls within Jef's ground rules else I'll be taunted as well ;)
--- Bests, Jon ----
<((((º>`·.¸¸.·´¯`·.¸.·´¯`·...¸><((((º>¸.
·´¯`·.¸. , . .·´¯`·.. ><((((º>`·.¸¸.·´¯`·.¸.·´¯`·...¸><((((º>>
On Thu, 2004-09-30 at 15:03, Jon Savage wrote:
I care far far more about finding a way to give each client computer a way to get a reasonable accurate list of mirrors that works well and reliably for access to released updates and base packages for full releases. If discussion isn't framed around this being the central goal, I'm not overly interested in it.
IMHO much of the weirdness with up2date and to a lesser degree with yum stems from mirror selection issues. The best solution currently seems to require that each OP hand edit his/her relevant configuration file(s) to point to more "desirable" mirrors. I've found up2date to work poorly at best since FC1; note that I don't so much mind breakage in the test releases since there are typically in excess of 50 package updates a day.
Perhaps a solution similar to the one fedora.us has implemented with their version of apt could be applied to up2date/yum namely the first time the app is run the OP gets a TUI that allows for mirror selections. Painless & effective.
except when the person kickstarts the system and expects it to be updated.
chkconfig yum on
and then it just uses whatever is the default mirror and you're back to square one
-sv
Em Qua, 2004-09-22 às 09:28, Rodd Clarkson escreveu:
I'm having real problems with up2date since upgrading from FC2 to FC3t2.
Since fedora 1, I've never could get up2date to work nicely. It is damn slow, specially if you compare it with synaptic and apt-get. Needless to say it is not nearly as pratical as synaptic.
Not to say that synaptic have the options of seeking for new/unknown (like when adding a new repo) packages, something you cannot do with ease with default fedora tools.
On Wed, Sep 22, 2004 at 10:28:35PM +1000, Rodd Clarkson wrote:
I'm having real problems with up2date since upgrading from FC2 to FC3t2.
Don't bother with up2date. Its unfortunate it wasnt dropped for FC2 if not earlier. Use yum.
installing. Worse, up2date would show that packages needed to be installed, but couldn't find them when asked to update them, or would show nothing needing to be updated when there were over 175 packages needing updating.
Some of this is out of sync mirrors I suspect. It isn't clear in the case of mirrors the right way to handle this - yum can be given multiple mirrors and will do retries up2date seems to just explode.