During FUDCon, we've been working on revising the Fedora release criteria. John Poelstra had already fleshed out a structure and much of the final content, and we've been revising and tweaking it in conjunction with QA (myself, Will Woods and James Laska), release engineering (Jesse Keating), anaconda team (especially Denise Dumas and Peter Jones) and desktop team (Christopher Aillon and Matthias Clasen, who provided suggestions at an earlier stage).
The new structure is based around a general page and specific pages for the Fedora 13 Alpha, Beta and Final releases (which have been written generically so they can easily be converted into pages for F14 and all future releases just by copying and pasting). You can find the criteria here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
they should contain everything you need to know. We based most of the criteria around testing that was already being carried out but with no formal policy basis, with additional suggestions from the anaconda and desktop teams.
We will follow these criteria for the Fedora 13 release process. So if you can see any problems or potential trouble with any of this, please do reply and let us know!
Desktop team - can you please let us know of any additional things that you would expect to be working at each point during the release cycle? Note that only things that *must* be working at each point should be listed on these pages, not nice-to-haves. You must be able to commit to the idea that, if any criterion on the page is not met, we would slip the release in question.
On Tue, Dec 8, 2009 at 3:55 AM, Adam Williamson awilliam@redhat.com wrote:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
My comments:
"10. # The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation of the previous stable Fedora release, either via preupgrade or by booting to the installer manually "
It should be from at least ten other rawhide stages too. Why is it so weak?
Criteria 11 of beta should figure in alpha. The rescue feature should be absolutely robust.
Criteria 15 can be more specific. Which desktops?.. all of Kde, Xfce and Gnome
Software Compilation from source and important rpm-src packages should be ok at beta stage.
Best
A. Mani
On Tue, 2009-12-08 at 06:38 +0500, A. Mani wrote:
On Tue, Dec 8, 2009 at 3:55 AM, Adam Williamson awilliam@redhat.com wrote:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
My comments:
"10. # The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation of the previous stable Fedora release, either via preupgrade or by booting to the installer manually "
It should be from at least ten other rawhide stages too. Why is it so weak?
That's all we can commit to test at present. Also, upgrading from Rawhide is something we care far less about; after all, all Rawhide installations should be disposable, so one not being upgradeable really isn't a big deal.
Criteria 11 of beta should figure in alpha. The rescue feature should be absolutely robust.
That's a possibility - James, John, what do you guys think of this? Anyone else?
Criteria 15 can be more specific. Which desktops?.. all of Kde, Xfce and Gnome
It specifically says 'default desktop' and that is all it's intended to cover. See John Poelstra's post to -devel-list about the difficulties of covering multiple desktops in the criteria.
Software Compilation from source and important rpm-src packages should be ok at beta stage.
that sounds like a good idea too - again, any other opinions on this?
On Wed, 2009-12-09 at 12:55 -0800, Adam Williamson wrote:
On Tue, 2009-12-08 at 06:38 +0500, A. Mani wrote:
On Tue, Dec 8, 2009 at 3:55 AM, Adam Williamson awilliam@redhat.com wrote:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
My comments:
"10. # The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation of the previous stable Fedora release, either via preupgrade or by booting to the installer manually "
It should be from at least ten other rawhide stages too. Why is it so weak?
That's all we can commit to test at present. Also, upgrading from Rawhide is something we care far less about; after all, all Rawhide installations should be disposable, so one not being upgradeable really isn't a big deal.
Criteria 11 of beta should figure in alpha. The rescue feature should be absolutely robust.
That's a possibility - James, John, what do you guys think of this? Anyone else?
Good catch, I don't have any objections. I've moved the rescue mode criteria to the previous milestone. So for Alpha, rescue mode will detect and mount default installations. For Beta, rescue mode will detect and mount more complex installation scenarios.
Criteria 15 can be more specific. Which desktops?.. all of Kde, Xfce and Gnome
It specifically says 'default desktop' and that is all it's intended to cover. See John Poelstra's post to -devel-list about the difficulties of covering multiple desktops in the criteria.
+1 to John's thread ... we'll tackle this in a future update.
Software Compilation from source and important rpm-src packages should be ok at beta stage.
that sounds like a good idea too - again, any other opinions on this?
Hmm, no strong opinions from me. If such a criteria existed, what would be the verification steps to ensure it is upheld?
On 12/07/2009 10:55 PM, Adam Williamson wrote:
During FUDCon, we've been working on revising the Fedora release criteria. John Poelstra had already fleshed out a structure and much of the final content, and we've been revising and tweaking it in conjunction with QA (myself, Will Woods and James Laska), release engineering (Jesse Keating), anaconda team (especially Denise Dumas and Peter Jones) and desktop team (Christopher Aillon and Matthias Clasen, who provided suggestions at an earlier stage).
Is this discussion available somewhere online?
The new structure is based around a general page and specific pages for the Fedora 13 Alpha, Beta and Final releases (which have been written generically so they can easily be converted into pages for F14 and all future releases just by copying and pasting). You can find the criteria here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
Category 10 in the Beta Release Criteria.
"The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation of the previous stable Fedora release, either via preupgrade or by booting to the installer manually"
I think we first need to get established if Fedora officially "supports" upgrades now. If it does we need create and add several upgrade test cases and to do so we need to know what "default installation" is. What packages that installation contains and since Releng needs to provide us with that list it would be good if they document and explain at the same time how and why they choose to make that selection the default. ( Default dvd install from my perspective should just be secure base no auto selection for the end user in the installer. )
https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
they should contain everything you need to know. We based most of the criteria around testing that was already being carried out but with no formal policy basis, with additional suggestions from the anaconda and desktop teams.
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR MOBLIN ( I would not be surprise if I'm forgetting some ) present and chimed in on this? What where these suggestion that Christopher and Matthias and others made encase any one wants to chime in and provide additional feedback?
Desktop team - can you please let us know of any additional things that you would expect to be working at each point during the release cycle? Note that only things that *must* be working at each point should be listed on these pages, not nice-to-haves. You must be able to commit to the idea that, if any criterion on the page is not met, we would slip the release in question.
Was this sent to all the *DE lists? ( Noticed that only Gnome was cc on this ).
JBG
On Tue, 2009-12-08 at 08:16 +0000, "Jóhann B. Guðmundsson" wrote:
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR MOBLIN ( I would not be surprise if I'm forgetting some ) present and chimed in on this? What where these suggestion that Christopher and Matthias and others made encase any one wants to chime in and provide additional feedback?
My comments were all on this list, so you should have no difficulty finding them.
On 12/08/2009 01:21 PM, Matthias Clasen wrote:
On Tue, 2009-12-08 at 08:16 +0000, "Jóhann B. Guðmundsson" wrote:
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR MOBLIN ( I would not be surprise if I'm forgetting some ) present and chimed in on this? What where these suggestion that Christopher and Matthias and others made encase any one wants to chime in and provide additional feedback?
My comments were all on this list, so you should have no difficulty finding them.
Interesting reply however I'm only interested in the ones that they took as your suggestions. As soon as I know that I can look through your comments on the list to find them :)
If you are unaware of which one(s) it was it's perhaps best to get Adam, Will or James to chime that into this thread.
JBG
On Tue, 2009-12-08 at 14:00 +0000, "Jóhann B. Guðmundsson" wrote:
Interesting reply however I'm only interested in the ones that they took as your suggestions. As soon as I know that I can look through your comments on the list to find them :)
John actually added Matthias' suggestions to the page in bullet point form before FUDCon happened; we only fleshed them out into better language during FUDCon. If you look at the state of the pages from _before_ FUDCon in the Wiki history, you should be able to see quite easily the quick-n-dirty list of Matthias' points :)
On Tue, 2009-12-08 at 08:16 +0000, "Jóhann B. Guðmundsson" wrote:
On 12/07/2009 10:55 PM, Adam Williamson wrote:
During FUDCon, we've been working on revising the Fedora release criteria. John Poelstra had already fleshed out a structure and much of the final content, and we've been revising and tweaking it in conjunction with QA (myself, Will Woods and James Laska), release engineering (Jesse Keating), anaconda team (especially Denise Dumas and Peter Jones) and desktop team (Christopher Aillon and Matthias Clasen, who provided suggestions at an earlier stage).
Is this discussion available somewhere online?
It was a hackfest, not a presentation, so it wasn't recorded - we were moving around and talking to different people and things, it took a day and a half, so it wouldn't have been practical. Remember it was discussed for quite a while before FUDCon too.
The new structure is based around a general page and specific pages for the Fedora 13 Alpha, Beta and Final releases (which have been written generically so they can easily be converted into pages for F14 and all future releases just by copying and pasting). You can find the criteria here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
Category 10 in the Beta Release Criteria.
"The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation of the previous stable Fedora release, either via preupgrade or by booting to the installer manually"
I think we first need to get established if Fedora officially "supports" upgrades now. If it does we need create and add several upgrade test cases and to do so we need to know what "default installation" is. What packages that installation contains and since Releng needs to provide us with that list it would be good if they document and explain at the same time how and why they choose to make that selection the default. ( Default dvd install from my perspective should just be secure base no auto selection for the end user in the installer. )
'Supports' is kind of a vague term and not really useful. We don't 'support' anything in the sense of guaranteeing that it'll work - it's not like we give you a refund if your system breaks :) So I try to avoid the word when talking formally about this sort of thing.
What it basically means is that at beta stage, we test that if you just install the previous release into a VM with all default options, update it, and then run an upgrade, it works. This isn't the same as 'supporting' upgrades, because it only tests a single very simple case. It's more of a basic check that the fundamental bits of upgrade code do their job, and there's no really huge bug that everyone who tries to upgrade is going to hit. We're not committing to testing and supporting every conceivable combination of packages for upgrading.
This isn't really a change - we've always been doing this test anyway, it's part of the installation test matrix, and if it had failed, we would have considered that a blocker bug. A lot of what this page does is just properly document and justify the tests that we've already been running in QA, with no official policy document to define what tests we _should_ run.
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR MOBLIN ( I would not be surprise if I'm forgetting some ) present and chimed in on this? What where these suggestion that Christopher and Matthias and others made encase any one wants to chime in and provide additional feedback?
The initial session was mostly poelcat, me and jlaska working the tests we have into the policy and cleaning up the high-level explanation stuff. At that point it had all the anaconda stuff in it, plus a few desktop things Matthias had suggested before FUDCon. We then took it to the anaconda team for their feedback, and to Christopher Aillon for a desktop team perspective, and added their suggestions. There were KDE people at FUDCon and the criteria hackfest was announced so they should have known about it, but they didn't come to offer suggestions and I didn't run into any of them while I was working on it to ask for their suggestions. Adam Miller was around and knows about this stuff from an XFCE position, but again he hasn't given anything the XFCE team would like to list. Moblin and Sugar are different as they don't get released as part of the Fedora release, so they probably don't go into these criteria.
You can see the suggestions made by Chris and Matthias pretty easily: all the desktop-related criteria in the pages come from them. All the stuff about what has to work in the 'default desktop' (so currently GNOME) is feedback from them.
Desktop team - can you please let us know of any additional things that you would expect to be working at each point during the release cycle? Note that only things that *must* be working at each point should be listed on these pages, not nice-to-haves. You must be able to commit to the idea that, if any criterion on the page is not met, we would slip the release in question.
Was this sent to all the *DE lists? ( Noticed that only Gnome was cc on this ).
Unfortunately not yet, just because I'm not subscribed to them and I was sending that message from sucktastic FUDCon wireless so it would have been pain to subscribe to them to send it :) I will definitely get in touch with each spin SIG to see what they want to feed back into the criteria, though. The pages as they currently exist aren't set in stone, though there is an issue about the relative importance of the spins which we really can't answer in a discussion simply about the release criteria, because it's a much bigger issue than that.
On Mon, 2009-12-07 at 14:55 -0800, Adam Williamson wrote:
During FUDCon, we've been working on revising the Fedora release criteria. John Poelstra had already fleshed out a structure and much of the final content, and we've been revising and tweaking it in conjunction with QA (myself, Will Woods and James Laska), release engineering (Jesse Keating), anaconda team (especially Denise Dumas and Peter Jones) and desktop team (Christopher Aillon and Matthias Clasen, who provided suggestions at an earlier stage).
The new structure is based around a general page and specific pages for the Fedora 13 Alpha, Beta and Final releases (which have been written generically so they can easily be converted into pages for F14 and all future releases just by copying and pasting). You can find the criteria here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
they should contain everything you need to know. We based most of the criteria around testing that was already being carried out but with no formal policy basis, with additional suggestions from the anaconda and desktop teams.
We will follow these criteria for the Fedora 13 release process. So if you can see any problems or potential trouble with any of this, please do reply and let us know!
Desktop team - can you please let us know of any additional things that you would expect to be working at each point during the release cycle? Note that only things that *must* be working at each point should be listed on these pages, not nice-to-haves. You must be able to commit to the idea that, if any criterion on the page is not met, we would slip the release in question.
Not sure if this has been raised yet, but are we specifying when in the release that packages should be signed with a valid signature? I believe packages are signed at all release milestones, but I'd like to clear up that assumption.
Thanks, James
On Fri, Dec 11, 2009 at 10:53:40 -0500, James Laska jlaska@redhat.com wrote:
Not sure if this has been raised yet, but are we specifying when in the release that packages should be signed with a valid signature? I believe packages are signed at all release milestones, but I'd like to clear up that assumption.
I belive the plan is that all official koji builds are going to be signed with the same key. The key will just provide assurance that the rpms were official builds from our koji and not that they are tied to a particular release or release type.
On Fri, 2009-12-11 at 10:53 -0500, James Laska wrote:
Not sure if this has been raised yet, but are we specifying when in the release that packages should be signed with a valid signature? I believe packages are signed at all release milestones, but I'd like to clear up that assumption.
Do you think that's a criteria issue, i.e. something to which there's an innate correct answer which can be defined and which shouldn't change? I'd think of it more as a process issue, but IMBW.
On Fri, 2009-12-11 at 08:20 -0800, Adam Williamson wrote:
On Fri, 2009-12-11 at 10:53 -0500, James Laska wrote:
Not sure if this has been raised yet, but are we specifying when in the release that packages should be signed with a valid signature? I believe packages are signed at all release milestones, but I'd like to clear up that assumption.
Do you think that's a criteria issue, i.e. something to which there's an innate correct answer which can be defined and which shouldn't change? I'd think of it more as a process issue, but IMBW.
Yeah, that's my question ... is there an assumption that all packages will be signed? Does this assumption need to be validated?
Looking at our current test plans for the release, I don't see anything where we confirm that packages are properly signed. Should we be testing this, and if so ... does it map back to a specific release criteria?
On 12/11/2009 08:52 AM, James Laska wrote:
On Fri, 2009-12-11 at 08:20 -0800, Adam Williamson wrote:
On Fri, 2009-12-11 at 10:53 -0500, James Laska wrote:
Not sure if this has been raised yet, but are we specifying when in the release that packages should be signed with a valid signature? I believe packages are signed at all release milestones, but I'd like to clear up that assumption.
Do you think that's a criteria issue, i.e. something to which there's an innate correct answer which can be defined and which shouldn't change? I'd think of it more as a process issue, but IMBW.
Yeah, that's my question ... is there an assumption that all packages will be signed? Does this assumption need to be validated?
Looking at our current test plans for the release, I don't see anything where we confirm that packages are properly signed. Should we be testing this, and if so ... does it map back to a specific release criteria?
The way we've approached the Release Criteria is that we are only capturing things that *must* be present to ship.
If the answer to the question: "If for whatever reason, packages were not signed with the correct key (or at all), would we delay shipping the release until they were signed correctly?"... is "yes" then I think it should be added to the criteria.
I would propose the answer should be "yes, all packages must be properly signed." I'm sure people will let me know if they disagree ;-)
John
-
"Jóhann B. Guðmundsson" -
A. Mani -
Adam Williamson -
Bruno Wolff III -
James Laska -
John Poelstra -
Matthias Clasen