Rawhide serves several purposes in Fedora and yet I don't think we clearly delineate what they are. It is often said in Fedoraland that the best way to test the Feodora is to "run rawhide." There is a difference between installing and running rawhide to experience the absolute latest and greatest Fedora has to offer and intentionally testing it. I just think we need to be clearer about our approach and use of rawhide for testing (specific QA) because sometimes it doesn't make sense.
Take a step back and consider what a strange testing target rawhide is:
1) An "officially" available version for reproducing bugs or test results only exists for one day--contrast that with our build environment that was created specifically with the intention of always being able to recreate binaries in their original environment.
2) For most people, the economical way to get rawhide is a mirror--there is no simple definitive way to determine that you have "the originally composed version" of rawhide for that day. There are a few methods that can give you "reasonable certainty", but nothing like a single check sum or a way to confirm that the tree you've downloaded is exactly the same as what was composed.
3) There is no "last known good version" when things completely get messed up and you need to reinstall from scratch except the Alpha, Beta, and Preview Release.
4) You never know when it will install or not. It isn't smoke tested--we leave that for *everyone* to do themselves based on their own install attempt. Even if Fedora hosted an automated smoke test to determine if it installs you still have the problem of mirrors being out of sync and not knowing if what you have is the exact same group of packages that passed the smoke test.
5) It is the community's only access point for obtaining the (what is close to) the Release Candidate for testing. I know several people will disagree with me immediately here--we've had the argument several times on IRC. I still think the underlying assumptions that it is "close enough" and "most likely the same thing" are not good enough. I think we have been lucky so far and there was a situation with F9 where the RC contained a different package version than what was in rawhide.
6) We often place a higher value on daily rawhide than the Alpha and Beta releases by proclaiming that they "don't really matter that much because they are 'simply snapshots' of rawhide." The community at large seems to focus more on the Alpha, Beta, and Preview releases as evidenced by spikes in traffic on fedora-test-list after these releases.
7) We consider rawhide our primary testing target yet there is no practical way to create a test matrix around it because it changes every day. Instead we create test matrices for the Alpha, Beta, and Preview releases which..... see the previous point. How do we know when we have completed a full test run? How can you thoroughly test a moving target?
I know I will hear the stock replies of "but it is rawhide" or "this is Fedora, not an enterprise distro like RHEL" and while I agree that "rawhide is rawhide" (a circular, but well understood argument for long term rawhide veterans) and that Fedora is not striving to be an "enterprise distro" that doesn't mean one of Fedora's goals has to be emphasizing a testing process that is flawed and could be better if we all put our collective brains together to come up with something better :-) We innovate in so many other areas... why not innovate here?
I would like to advocate that we reconsider the value we place on rawhide and the emphasis we place around the Alpha, Beta, and Preview Release. I think a good place to start would be document in our test plan where using rawhide for test results makes sense and where it does not. I believe rawhide does have its place, but I think we are trying to use it to cover too many bases and could do more effective testing with a more refined approach which in the end makes Fedora better!
What do other people think? Is there something here worth throwing around here on this list with a following up discussing at FUDCon later in the week?
On Mon, Jun 16, 2008 at 10:16:46AM -0400, John Poelstra wrote:
Rawhide serves several purposes in Fedora and yet I don't think we clearly delineate what they are. It is often said in Fedoraland that the best way to test the Feodora is to "run rawhide." There is a difference between installing and running rawhide to experience the absolute latest and greatest Fedora has to offer and intentionally testing it. I just think we need to be clearer about our approach and use of rawhide for testing (specific QA) because sometimes it doesn't make sense.
[...]
- You never know when it will install or not. It isn't smoke tested--we
leave that for *everyone* to do themselves based on their own install attempt. Even if Fedora hosted an automated smoke test to determine if it installs you still have the problem of mirrors being out of sync and not knowing if what you have is the exact same group of packages that passed the smoke test.
Still, doing automated installs is something that's possible to do and a good testing tool.
Regards, R.
On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote:
Rawhide serves several purposes in Fedora and yet I don't think we clearly delineate what they are. Take a step back and consider what a strange testing target rawhide is:
Rawhide is a good place (along with update-testing) to check the progress of bugs oneself has submitted.
6) The community at large
seems to focus more on the Alpha, Beta, and Preview releases as evidenced by spikes in traffic on fedora-test-list after these releases.
It's tangible you can put your hands on it, and run it (livecd) without damaging stable system. Which for most users is a good thing.
- We consider rawhide our primary testing target yet there is no
practical way to create a test matrix around it because it changes every day. Instead we create test matrices for the Alpha, Beta, and Preview releases which..... see the previous point. How do we know when we have completed a full test run? How can you thoroughly test a moving target?
Maybe reduce rawhide release(s) to weekly and date? eg: yum-3.2.16-2.fc9.061608.noarch.rpm
So more testing can be accomplished
Frank
On Tue, Jun 17, 2008 at 10:29 AM, Frank Murphy frankly3d@gmail.com wrote:
Maybe reduce rawhide release(s) to weekly and date? eg: yum-3.2.16-2.fc9.061608.noarch.rpm
So more testing can be accomplished
That unfortunately increases the amount of delay before new fixes can be brought in, which is the other side of the coin. The plain problem is that the Fedora package set is /huge/, and I don't think there is any reasonable way we can try to coordinate all the various pieces of it into a cohesive testable unit outside of the major effort that goes into our alpha/beta/prerelease parts, and even that doesn't get the whole thing.
I don't disagree that this is a tough problem, or that there is even a problem here. I just don't know of a good way to solve this problem that doesn't introduce lots of others, such as the above mentioned lag on getting fixes into the hands of the masses.
I've always felt that the alpha/beta/pre releases provide good well known starting points for the Rawhide adventure. You can be reasonably certain that those points will install on your system. From there you can update yourself in part or in whole to Rawhide de jour in order to verify operation of a particular piece of software or subsystem. It is true that bugs contained within the alpha/beta don't really matter much past the rawhide day that these were snapshot from. If the bug persists into the current rawhide version of the given package then it does matter, but there is often a chance that it's been fixed along the way.
Anyway, I'd love to have more of this discussion sometime around FUDCon, and more on list too.
-- Jes
On Tue, 2008-06-17 at 10:44 -0400, Jesse Keating wrote:
On Tue, Jun 17, 2008 at 10:29 AM, Frank Murphy frankly3d@gmail.com wrote:
Maybe reduce rawhide release(s) to weekly and date? eg: yum-3.2.16-2.fc9.061608.noarch.rpm So more testing can be accomplished
Rawhide definitely meets the need of providing latest and greatest package content by way of mirroring the yum repo daily. If you find a busted package, report the bug, it gets fixed ... you only have to wait at most 23:59. Talk about a tight feedback loop, that's great!
The pain point for me has been integrating the latest and greatest content, while attempting to stabilize components which have significant dependencies (e.g. installation). We typically get stuck in a cycle where a bug blocks installation ... and during the blocked period ... development continues. This isn't a bad thing for development, but can often leads to a perfect storm where:
1) packageX introduces a blocker for installation testing 2) new code lands in anaconda-devel that introduces/exposes a bug 3) packageX bug is fixed 4) packageY introduces a blocker for installation testing 5) new code lands in anaconda-devel ...
This tends to be the case between milestones. All engineering energy is focused at removing the blockers, but many more serious issues remain under the covers.
I've seen several solutions on the table in the past ... each with their own pros/cons. I'll toss a few that are on my mind now (and please fill in the blanks if there are others) ...
1) Greater visibility on test blockers
Perhaps just raising awareness that testing is blocked on a particular component or sub-system is sufficient.
2) dist-f10-anaconda koji tag
Continue providing rawhide, but also allow for baking/testing of anaconda related changes in a specific koji tag. This tag would be used to build anaconda install/boot images in order to test anaconda changes and features. The same packages could be either tagged for general rawhide consumption at build time, or after testing has completed.
3) Host several days of rawhide
Continue mirroring just 'rawhide' latest ... no change there. However, begin hosting timestamp'd daily copies of rawhide.
rawhide -> rawhide-20080604 rawhide-20080604 rawhide-20080603 rawhide-20080602 rawhide-20080601
The idea here being we have a working rawhide for at least several days in order to walk through a test matrix.
Jesse: You know the internals of the mirroring, you probably have better insight to say why this isn't done, and perhaps why we don't want this.
4) Make "rawhide" just a yum repo
Only partly serious here ... drop the install images from rawhide.
Doesn't mean that the rawhide package set cannot be tested during development, only that the boot/install images won't necessarily be built against rawhide.
I'm only partly serious here. But what appeals to me about this is it's very clear what we are providing. Just a repo ... do with it what you like. One use case of the repo would be building/hosting install images elsewhere. Another might be liveCD generation. We'd certainly need to work that out install image creation as to not impede community installation testing.
Any other thoughts/ideas out there that might help clear up what the intended use for rawhide is?
That unfortunately increases the amount of delay before new fixes can be brought in, which is the other side of the coin. The plain problem is that the Fedora package set is /huge/, and I don't think there is any reasonable way we can try to coordinate all the various pieces of it into a cohesive testable unit outside of the major effort that goes into our alpha/beta/prerelease parts, and even that doesn't get the whole thing.
I don't disagree that this is a tough problem, or that there is even a problem here. I just don't know of a good way to solve this problem that doesn't introduce lots of others, such as the above mentioned lag on getting fixes into the hands of the masses.
I've always felt that the alpha/beta/pre releases provide good well known starting points for the Rawhide adventure. You can be reasonably certain that those points will install on your system. From there you can update yourself in part or in whole to Rawhide de jour in order to verify operation of a particular piece of software or subsystem. It is true that bugs contained within the alpha/beta don't really matter much past the rawhide day that these were snapshot from. If the bug persists into the current rawhide version of the given package then it does matter, but there is often a chance that it's been fixed along the way.
Anyway, I'd love to have more of this discussion sometime around FUDCon, and more on list too.
Yeah, I'd be up for further discussion @ FUDCon.
If nothing else, any discussion might serve to strengthen+support why things are the way they are.
Thanks, James
-- fedora-test-list mailing list fedora-test-list@redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-test-list
On Tue, 2008-06-17 at 13:31 -0400, James Laska wrote:
- Host several days of rawhide
Continue mirroring just 'rawhide' latest ... no change there. However, begin hosting timestamp'd daily copies of rawhide.
rawhide -> rawhide-20080604 rawhide-20080604 rawhide-20080603 rawhide-20080602 rawhide-20080601
The idea here being we have a working rawhide for at least several days in order to walk through a test matrix.
Jesse: You know the internals of the mirroring, you probably have better insight to say why this isn't done, and perhaps why we don't want this.
Mostly that this can balloon into several hundred gigs worth of extra data quite quickly depending on the changes. Bear in mind that a single Openoffice.org build consumes 5.1 gigs of disk space. 5.1gigs. A single build. As I look through the build history, I see rawhide builds on the second, third, fourth, fifth, sixth, etc... That's over 25gigs just in one package. Obviously oo.org is an extreme example but there is a very real size issue to be concerned with, and much more mirror churn to deal with and a larger barrier of entry for new mirrors.
Now somebody is going to say something about deltas, and that's great for the rpm consumer, but realistically we're going to have to store the full rpms somewhere and mirror those around.
Now, storing multiple copies of just boot.iso somewhere may be just as valuable (since you're really after the stable installer image) and much easier to accomplish. It's only a few hundred megs of extra data to contend with.
On Tue, 2008-06-17 at 13:49 -0400, Jesse Keating wrote:
Now, storing multiple copies of just boot.iso somewhere may be just as valuable (since you're really after the stable installer image) and much easier to accomplish. It's only a few hundred megs of extra data to contend with.
Oooh that's a clever idea. Continue providing the daily rawhide repo's, but save the install/boot images (images/*) for several days?
Might this be a tie-in with the recent anaconda stage2= support?
Thanks, James
On Tue, 2008-06-17 at 13:55 -0400, James Laska wrote:
Oooh that's a clever idea. Continue providing the daily rawhide repo's, but save the install/boot images (images/*) for several days?
Might this be a tie-in with the recent anaconda stage2= support?
Well, if you're saving boot.iso there is no need to specify stage2= since it's already on the iso. But yes, for those just using vmlinuz/initrd they can specify a different stage2 location.
On Tue, 2008-06-17 at 13:55 -0400, James Laska wrote:
On Tue, 2008-06-17 at 13:49 -0400, Jesse Keating wrote:
Now, storing multiple copies of just boot.iso somewhere may be just as valuable (since you're really after the stable installer image) and much easier to accomplish. It's only a few hundred megs of extra data to contend with.
Oooh that's a clever idea. Continue providing the daily rawhide repo's, but save the install/boot images (images/*) for several days?
Might this be a tie-in with the recent anaconda stage2= support?
This is one of the reasons behind all of the changes that we[1]'ve been making to anaconda in this area
Jeremy
[1] Where by "we", I mean mostly "clumens".
--- On Tue, 6/17/08, Jesse Keating jkeating@j2solutions.net wrote:
From: Jesse Keating jkeating@j2solutions.net Subject: Re: What is rawhide for? To: "For testers of Fedora Core development releases" fedora-test-list@redhat.com Date: Tuesday, June 17, 2008, 7:44 AM On Tue, Jun 17, 2008 at 10:29 AM, Frank Murphy frankly3d@gmail.com wrote:
Maybe reduce rawhide release(s) to weekly and date? eg: yum-3.2.16-2.fc9.061608.noarch.rpm
So more testing can be accomplished
I've always felt that the alpha/beta/pre releases provide good well known starting points for the Rawhide adventure. You can be reasonably certain that those points will install on your system. From there you can update yourself in part or in whole to Rawhide de jour in order to verify operation of a particular piece of software or subsystem. It is true that bugs contained within the alpha/beta don't really matter much past the rawhide day that these were snapshot from. If the bug persists into the current rawhide version of the given package then it does matter, but there is often a chance that it's been fixed along the way.
--
I think that daily snapshots are overkill, but to wait for an installable iso(s) only to be (alpha/beta/pre releases) does not help much, the boot iso is the only one that is being updated and have network installs in which the mirrors do not collaborate.
A well documented install from the mirrors with just boot iso, will be a good thing. I would try this when I get back to work in late August/early September when that time comes and someone is nice enough to provide precise information to do an install that way forgetting about the traditional releases and the new alpha/beta/pre release stuff.
Also, instead of daily snapshots of an installable fedora rawhide tree, how about weekly snapshots? It might help people install rawhide after updates that did not allow previous installations to take place due to bugs that did not allow the install program(anaconda) to finish its work.
I think that the beauty of rawhide is release fast, release often, report bugs, fix X, Y breaks, fix Y, X breaks again, so back to X and so on. I am not running rawhide now as I would like to, I do not have a fast connection like I do at work and will wait to continue my rawhide adventure. I only had two machines running rawhide and before the release of Fedora 9, I moved an aging Fedora 7 release to rawhide (with some troubles) but it worked out nicely in the end, and a Fedora 8 machine that froze all the time, moved it to Pre-Release and it started working fine (not freezing like it did on Fedora 8) so I am a happy camper and I believe in rawhide. I got in the boat because I really like Fedora and believe in its goals, *not everything* I must clarify, but the people involved on this lists are fantastic. They are most of the time willing to give solutions/workarounds to problems that arise when bad things do happen while running rawhide.
One strange thing that happened to me was when Fedora 7 was released, the programs pointed to a Fedora 7 straight release and deviated from Rawhide. Rawhide repo became disabled and I proceeded. Then my hard drive died out, so I reinstalled pre 7 test release and updated straight into rawhide again and when Fedora releases a final product, I change back to rawhide on that machine I am very comfortable running it that I did not felt a need to run the regular Fedora on that machine. At home though I do run a regular Fedora, currently a Fedora 9 x86_64 machine, but since I have slow connection, I have to pick and choose which updates I install. But I do miss the fun of running rawhide during the summer months.
Regards,
Antonio
On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote:
- An "officially" available version for reproducing bugs or test
results only exists for one day--contrast that with our build environment that was created specifically with the intention of always being able to recreate binaries in their original environment.
The difference between rawhide on any two days is, on average, somewhere between 0.1% and 0.5%. If this *blazing* development pace is too much for you, you're welcomed and encouraged to keep multiple copies of the rawhide repos.
It'd definitely be nice if we had some good scripts for maintaining multiple local copies of rawhide, with hardlinks to save space.
- For most people, the economical way to get rawhide is a mirror--there
is no simple definitive way to determine that you have "the originally composed version" of rawhide for that day. There are a few methods that can give you "reasonable certainty", but nothing like a single check sum or a way to confirm that the tree you've downloaded is exactly the same as what was composed.
This is just plain wrong. The timestamp in .treeinfo is unique per-tree.
Compare the master mirror: http://download.fedora.redhat.com/pub/fedora/linux/development/i386/os/.tree... With your local mirror, say: http://limestone.uoregon.edu/ftp/fedora/development/i386/os/.treeinfo If the timestamps match, you have the most current rawhide.
You can also use 'snake' to do this: [wwoods@metroid ~]$ snake-tree info MIRROR_URL Name : Fedora development i386 (20080617) Arch : i386 Id : 1213693723.22 Version : development Family : Fedora Variant : Time : 2008-06-17 05:08 EDT URI's : Images : i386 (kernel, family, timestamp, variant, boot.iso, initrd, version, arch), xen (kernel, family, timestamp, variant, initrd, version, arch)
- There is no "last known good version" when things completely get
messed up and you need to reinstall from scratch except the Alpha, Beta, and Preview Release.
This is definitely a problem, and a tricky one at that. It deserves further discussion in its own mail thread and/or at FUDCon.
- You never know when it will install or not. It isn't smoke
tested--we leave that for *everyone* to do themselves based on their own install attempt. Even if Fedora hosted an automated smoke test to determine if it installs you still have the problem of mirrors being out of sync and not knowing if what you have is the exact same group of packages that passed the smoke test.
This is the problem that the rawhide dashboard page is supposed to solve. We're automating SNAKE to do smoke tests on rawhide every night, and post results on the dashboard. So you *will* know whether it installs or not.
We've got a prototype rawhide dashboard page that shows whether boot images are present, at least: http://wwoods.fedorapeople.org/rawhide.html
As for the mirrors being out of sync - see above.
- It is the community's only access point for obtaining the (what is
close to) the Release Candidate for testing. I know several people will disagree with me immediately here--we've had the argument several times on IRC. I still think the underlying assumptions that it is "close enough" and "most likely the same thing" are not good enough.
I think you're wrong. I also think you underestimate the time it requires to put out an RC versus the turnaround time on testing it.
Given the amount of time it adds to the schedule and lengthens the development freeze, plus the amount of work it takes from every part of the release team, what would we actually *gain* from doing all that work?
- We often place a higher value on daily rawhide than the Alpha and
Beta releases by proclaiming that they "don't really matter that much because they are 'simply snapshots' of rawhide." The community at large seems to focus more on the Alpha, Beta, and Preview releases as evidenced by spikes in traffic on fedora-test-list after these releases.
I think you're mixing up cause and effect here.
People don't focus on the milestones because they just inherently *love* milestones. They focus on the milestones because we go to the effort of making them easy to consume, and we publicize them and test them to ensure they'll be installable and put up ISOs on torrents and the mirrors. So *obviously* more people use them.
- We consider rawhide our primary testing target yet there is no
practical way to create a test matrix around it because it changes every day. Instead we create test matrices for the Alpha, Beta, and Preview releases which..... see the previous point. How do we know when we have completed a full test run? How can you thoroughly test a moving target?
I completely disagree with the assertion that "there is no practical way to create a test matrix for Rawhide".
First of all, the only reason we don't create test matrices for every day's rawhide is that it's time-consuming to create the matrix itself. The old wiki's pretty slow, remember?
Better test run tracking (with Testopia, for instance) will make it trivial to create a matrix for every day's rawhide. Further, automating our test cases will let us fill in most of that matrix automatically.
The test plan for rawhide is, obviously going to be somewhat less exhaustive than the test plans for milestone releases or the final release. So we can complete a "full test run" by looking at the (mostly-full, auto-created) matrix for that day's rawhide and.. filling in the blanks.
that doesn't mean one of Fedora's goals has to be emphasizing a testing process that is flawed and could be better if we all put our collective brains together to come up with something better :-) We innovate in so many other areas... why not innovate here?
I would like to advocate that we reconsider the value we place on rawhide and the emphasis we place around the Alpha, Beta, and Preview Release.
Innovate.. by emphasizing the traditional milestones? I think you'll find that "innovation" and "tradition" are actually *opposites*.
Rawhide might be a strange testing target from the point of view of someone coming from RHEL or proprietary systems or other traditional milestone-focused development, but I don't think Rawhide is that strange in the Open Source world.
You can apply nearly all of your arguments to, say, the way the kernel is developed - it changes too quickly! Not enough freeze time! We need more milestones! You can't possibly test it!
You know what would *really* be innovative? Engineering our test efforts to match the pace and reality of typical Open Source development, rather than working the other way around.
I think a good place to start would be document in our test plan where using rawhide for test results makes sense and where it does not. I believe rawhide does have its place, but I think we are trying to use it to cover too many bases and could do more effective testing with a more refined approach which in the end makes Fedora better!
Okay. So start documenting where you think it does and doesn't make sense and we'll discuss *that*. But so far all you've done is rehashed a bunch of complaints and offered no solutions.
What do other people think? Is there something here worth throwing around here on this list with a following up discussing at FUDCon later in the week?
Sure, if we go to the effort of actually *defining* problems and discussing *solutions*, it's totally worth it.
-w
Will Woods said the following on 06/17/2008 11:12 AM Pacific Time:
On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote:
- An "officially" available version for reproducing bugs or test
results only exists for one day--contrast that with our build environment that was created specifically with the intention of always being able to recreate binaries in their original environment.
The difference between rawhide on any two days is, on average, somewhere between 0.1% and 0.5%. If this *blazing* development pace is too much for you, you're welcomed and encouraged to keep multiple copies of the rawhide repos.
Rapid amount of change is not the issue. I'm speaking more to the situation of encountering a problem, reporting a bug, being asked for more information on that bug, but not being able to troubleshoot further because the exact same tree used to install is gone... there is no history.
It'd definitely be nice if we had some good scripts for maintaining multiple local copies of rawhide, with hardlinks to save space.
- For most people, the economical way to get rawhide is a mirror--there
is no simple definitive way to determine that you have "the originally composed version" of rawhide for that day. There are a few methods that can give you "reasonable certainty", but nothing like a single check sum or a way to confirm that the tree you've downloaded is exactly the same as what was composed.
This is just plain wrong. The timestamp in .treeinfo is unique per-tree.
Just because .treeinfo is present doesn't mean all the other packages associated with it from that day are present on the mirror.
<<CUT>>
- You never know when it will install or not. It isn't smoke
tested--we leave that for *everyone* to do themselves based on their own install attempt. Even if Fedora hosted an automated smoke test to determine if it installs you still have the problem of mirrors being out of sync and not knowing if what you have is the exact same group of packages that passed the smoke test.
This is the problem that the rawhide dashboard page is supposed to solve. We're automating SNAKE to do smoke tests on rawhide every night, and post results on the dashboard. So you *will* know whether it installs or not.
Excellent. I didn't realize that was in process. Could the dashboard also contain a historical record of which days install and which days don't? I think this would be useful information to have.
We've got a prototype rawhide dashboard page that shows whether boot images are present, at least: http://wwoods.fedorapeople.org/rawhide.html
snake-treeinfo can tell this too, right?
As for the mirrors being out of sync - see above.
- It is the community's only access point for obtaining the (what is
close to) the Release Candidate for testing. I know several people will disagree with me immediately here--we've had the argument several times on IRC. I still think the underlying assumptions that it is "close enough" and "most likely the same thing" are not good enough.
I think you're wrong. I also think you underestimate the time it requires to put out an RC versus the turnaround time on testing it.
Given the amount of time it adds to the schedule and lengthens the development freeze, plus the amount of work it takes from every part of the release team, what would we actually *gain* from doing all that work?
I think we can create a more polished release with less critical items on the "known issues" wiki page for that release. I realize the "criteria" and "how much time is too much" is somewhat subjective and something we do not agree on.
- We often place a higher value on daily rawhide than the Alpha and
Beta releases by proclaiming that they "don't really matter that much because they are 'simply snapshots' of rawhide." The community at large seems to focus more on the Alpha, Beta, and Preview releases as evidenced by spikes in traffic on fedora-test-list after these releases.
I think you're mixing up cause and effect here.
People don't focus on the milestones because they just inherently *love* milestones. They focus on the milestones because we go to the effort of making them easy to consume, and we publicize them and test them to ensure they'll be installable and put up ISOs on torrents and the mirrors. So *obviously* more people use them.
I was attempting to respond to past situations when the alpha and beta haven't been in that great of shape and it has been said that it doesn't matter that much because "beta == snapshot of rawhide".
- We consider rawhide our primary testing target yet there is no
practical way to create a test matrix around it because it changes every day. Instead we create test matrices for the Alpha, Beta, and Preview releases which..... see the previous point. How do we know when we have completed a full test run? How can you thoroughly test a moving target?
I completely disagree with the assertion that "there is no practical way to create a test matrix for Rawhide".
First of all, the only reason we don't create test matrices for every day's rawhide is that it's time-consuming to create the matrix itself. The old wiki's pretty slow, remember?
Better test run tracking (with Testopia, for instance) will make it trivial to create a matrix for every day's rawhide. Further, automating our test cases will let us fill in most of that matrix automatically.
The test plan for rawhide is, obviously going to be somewhat less exhaustive than the test plans for milestone releases or the final release. So we can complete a "full test run" by looking at the (mostly-full, auto-created) matrix for that day's rawhide and.. filling in the blanks.
that doesn't mean one of Fedora's goals has to be emphasizing a testing process that is flawed and could be better if we all put our collective brains together to come up with something better :-) We innovate in so many other areas... why not innovate here?
I would like to advocate that we reconsider the value we place on rawhide and the emphasis we place around the Alpha, Beta, and Preview Release.
Innovate.. by emphasizing the traditional milestones? I think you'll find that "innovation" and "tradition" are actually *opposites*.
Innovate by taking a different approach we haven't considered before. I didn't suggest that "traditional milestones" are the answer.
Rawhide might be a strange testing target from the point of view of someone coming from RHEL or proprietary systems or other traditional milestone-focused development, but I don't think Rawhide is that strange in the Open Source world.
I'll give that some more thought.
You can apply nearly all of your arguments to, say, the way the kernel is developed - it changes too quickly! Not enough freeze time! We need more milestones! You can't possibly test it!
You know what would *really* be innovative? Engineering our test efforts to match the pace and reality of typical Open Source development, rather than working the other way around.
I think a good place to start would be document in our test plan where using rawhide for test results makes sense and where it does not. I believe rawhide does have its place, but I think we are trying to use it to cover too many bases and could do more effective testing with a more refined approach which in the end makes Fedora better!
Okay. So start documenting where you think it does and doesn't make sense and we'll discuss *that*. But so far all you've done is rehashed a bunch of complaints and offered no solutions.
I guess that is the tricky part. I'd like to think that being a contributor to Fedora does not mean you always have to have the solution to raise what you perceive as something that is wrong. Hopefully my contributions in other areas Fedora make up for it here.
What do other people think? Is there something here worth throwing around here on this list with a following up discussing at FUDCon later in the week?
Sure, if we go to the effort of actually *defining* problems and discussing *solutions*, it's totally worth it.
I agree.
Or if the problems are misunderstood then we need to do a better explaining why our current approach really does make sense--I am not the only one who has echoed these same questions. Maybe someone else can explain the issues I'm trying to get at in a better way?
John
John Poelstra said the following on 06/16/2008 07:16 AM Pacific Time:
Rawhide serves several purposes in Fedora and yet I don't think we clearly delineate what they are. It is often said in Fedoraland that the best way to test the Feodora is to "run rawhide." There is a difference between installing and running rawhide to experience the absolute latest and greatest Fedora has to offer and intentionally testing it. I just think we need to be clearer about our approach and use of rawhide for testing (specific QA) because sometimes it doesn't make sense.
Follow-up. We had a great discussion at FUDCon which is documented here: https://fedoraproject.org/wiki/JohnPoelstra/ImproveRawhideF10
John
-
Antonio Olivares -
Dominik 'Rathann' Mierzejewski -
Frank Murphy -
James Laska -
Jeremy Katz -
Jesse Keating -
Jesse Keating -
John Poelstra -
Will Woods