Hi, folks! So I'm following up again on the request from the Council that we send someone to report on the QA project's status. The proposed date was 2015-06-08.
At the last QA meeting (on 2015-05-04) I volunteered to do it, then realized I won't be able to on 2015-06-08 as I'll be on a plane. So, we have two choices: either someone else volunteers to do the report, or we ask the Council if we can change the date. Is anyone else interested in doing the report? Please reply if so!
The other thing to settle is what we actually want to report. Matt provided the following guidelines:
- the current state of the subproject - future plans - things the team needs from the rest of the project - any blockers we can help unblock - big resource requests?
I have some initial thoughts, and it'd be good if anyone else can chip in: whoever reports to the Council, I think they should be acting as a representative for all of us as a whole, so it'd be good to gather everyone's thoughts together for the representative to pass along.
Current state -------------
I'd say we have a fairly solid process in place now for release validation, and everyone's probably more or less familiar with it. We also have the update testing process nailed down, though I still think we can improve it in some ways, and we *still* want Bodhi 2.0 :)
We have taskotron running for some package tests and openQA for some installation validation tests. openQA is I think still running on openSUSE boxes, but we have some work done on containerization to potentially move them to Fedora hosts (and I've also been working on packaging the openQA bits for Fedora).
I think we might still be lacking a bit in terms of Change testing; there's a personpower issue there, though. We usually wind up only cherrypicking the most obviously potentially disruptive Changes for testing, and usually from the perspective of 'does this break the release'; we rarely manage to find the time to test Changes *in themselves* and help make sure they work well.
Future plans ------------
We have a fairly well-defined roadmap for Taskotron development, I believe; we could broaden its actual test coverage, but I think we still have some infrastructure work that probably needs nailing down before we can focus on that, particularly in terms of the disposable test client work, and also I believe in terms of results management.
I'd definitely like to see us extending openQA coverage in the post-F22 'quiet time'. There are still many validation tests that could successfully be automated, which gives us much better coverage and reduces the manual testing burden. I'm certainly intending to help work on this. We could also consider soliciting wider contributions of openQA tests; I think the openQA experiment has been fairly successful, and I could certainly envisage some cases where maintainers might want to have tests related to their packages in openQA.
Something I've been kicking around with release engineering a bit, and which will probably come up at Flock, is the possibility of revising the whole TC/RC approach to release composing. It's getting fairly old, and is built on some assumptions that are becoming somewhat outdated: that building and consuming composes are inherently expensive operations and hence need to be 'rationed' and carefully, manually handled. In an ideal world, I think it would be nice to take a more modern, CI-ish approach: we should be doing the whole compose process daily (not just boot.isos and lives) and running automated tests on the daily composes (openQA is already capable of this), and improving the efficiency of the process whereby updates get into composes. The 'special package requests' in TC/RC compose requests have become kinda a permanent feature, but they were never meant to be. I'd prefer if we could make the process whereby updates move to stable vastly more efficient, and perhaps have a slightly different process for handling builds we want to put in TCs/TCs but which we don't yet want to push stable (the most common case being the packages that are actually involved in *doing composes*, so in order to test them, we need to build composes with them) - something more visible and less clunky than the current 'list them in TC/RC requests and they go into a manually maintained bleed repo' approach.
Things we need --------------
Kinda tied in to the above: Bodhi 2.0, and more efficient releng processes, are the big two I can think of - faster update pushes and composes. I think tflink and kparal are in a better position than I to know about any major resources we need.
Hope that kicks off a few thoughts from other folks! Please don't be shy, and pitch in with any thoughts you have :)
Hi, folks! So I'm following up again on the request from the Council that we send someone to report on the QA project's status. The proposed date was 2015-06-08.
At the last QA meeting (on 2015-05-04) I volunteered to do it, then realized I won't be able to on 2015-06-08 as I'll be on a plane. So, we have two choices: either someone else volunteers to do the report, or we ask the Council if we can change the date. Is anyone else interested in doing the report? Please reply if so!
Thanks, Adam, for looking into this. You'll probably already on vacation, so I assume you won't be reading this until the actual council meeting. But if you did...
If needed, I'm happy to fill in for you. But if you wanted to be actively part of the discussion, I can also ask the council to move the date. You seem to have given quite some thought into those issues. I won't be able to look into it properly before we push F22 out of the door, but that should hopefully happen way before the council meeting, so then I can try to expand the points you mentioned below.
If there are some other volunteers to take care of this, I'm even more happy to pass this onto you :-)
Kamil
----- Original Message ----- | From: "Kamil Paral" kparal@redhat.com | To: "For testing and quality assurance of Fedora releases" test@lists.fedoraproject.org | Sent: Monday, May 18, 2015 6:27:26 PM | Subject: Re: Fedora Council report | | > Hi, folks! So I'm following up again on the request from the Council | > that we send someone to report on the QA project's status. The proposed | > date was 2015-06-08. | > | > At the last QA meeting (on 2015-05-04) I volunteered to do it, then | > realized I won't be able to on 2015-06-08 as I'll be on a plane. So, we | > have two choices: either someone else volunteers to do the report, or | > we ask the Council if we can change the date. Is anyone else interested | > in doing the report? Please reply if so! | | Thanks, Adam, for looking into this. You'll probably already on vacation, so | I assume you won't be reading this until the actual council meeting. But if | you did... | | If needed, I'm happy to fill in for you. But if you wanted to be actively | part of the discussion, I can also ask the council to move the date. You | seem to have given quite some thought into those issues. I won't be able to | look into it properly before we push F22 out of the door, but that should | hopefully happen way before the council meeting, so then I can try to expand | the points you mentioned below.
Hi Kamil,
Glad to see you are filling in for Adam. You can sync up with him once he is back from vacation :)
Cheers, Sudhir
| | If there are some other volunteers to take care of this, I'm even more happy | to pass this onto you :-) | | Kamil | | -- | test mailing list | test@lists.fedoraproject.org | To unsubscribe: | https://admin.fedoraproject.org/mailman/listinfo/test
On Mon, May 18, 2015 at 08:57:26AM -0400, Kamil Paral wrote:
Hi, folks! So I'm following up again on the request from the Council that we send someone to report on the QA project's status. The proposed date was 2015-06-08.
At the last QA meeting (on 2015-05-04) I volunteered to do it, then realized I won't be able to on 2015-06-08 as I'll be on a plane. So, we have two choices: either someone else volunteers to do the report, or we ask the Council if we can change the date. Is anyone else interested in doing the report? Please reply if so!
Thanks, Adam, for looking into this. You'll probably already on vacation, so I assume you won't be reading this until the actual council meeting. But if you did...
If needed, I'm happy to fill in for you. But if you wanted to be actively part of the discussion, I can also ask the council to move the date. You seem to have given quite some thought into those issues. I won't be able to look into it properly before we push F22 out of the door, but that should hopefully happen way before the council meeting, so then I can try to expand the points you mentioned below.
If there are some other volunteers to take care of this, I'm even more happy to pass this onto you :-)
Kamil
Since we decided right after the QA meeting that I'd be filling in as the QA rep for this meeting - I figured I'd ping the list to see if anyone had anything else they wanted to include?
Kamil will be sending out his additions (probably tomorrow), and then I'll be making a slide deck for the hangout. It would also be good if we all watched the meeting as it was taking place; I'm sure the council will have questions for us as a group.
Thanks!
On Mon, Jun 01, 2015 at 01:33:19PM -0600, Mike Ruckman wrote:
Since we decided right after the QA meeting that I'd be filling in as the QA rep for this meeting - I figured I'd ping the list to see if anyone had anything else they wanted to include?
Kamil will be sending out his additions (probably tomorrow), and then I'll be making a slide deck for the hangout. It would also be good if we all watched the meeting as it was taking place; I'm sure the council will have questions for us as a group.
Thanks!
Thanks for the feedback! I've created and uploaded a slide deck for the report on the wiki [0]. It doesn't go into much detail in the slides, but please let me know if it looks like I missed anything.
Thanks!
[0] https://fedoraproject.org/wiki/User:Roshi/QA/CouncilReportSlides
Hello,
I have added some notes below.
The other thing to settle is what we actually want to report. Matt provided the following guidelines:
- the current state of the subproject
- future plans
- things the team needs from the rest of the project
- any blockers we can help unblock
- big resource requests?
Current state
I'd say we have a fairly solid process in place now for release validation, and everyone's probably more or less familiar with it. We also have the update testing process nailed down, though I still think we can improve it in some ways, and we *still* want Bodhi 2.0 :)
We have taskotron running for some package tests and openQA for some installation validation tests. openQA is I think still running on openSUSE boxes, but we have some work done on containerization to potentially move them to Fedora hosts (and I've also been working on packaging the openQA bits for Fedora).
I think we might still be lacking a bit in terms of Change testing; there's a personpower issue there, though. We usually wind up only cherrypicking the most obviously potentially disruptive Changes for testing, and usually from the perspective of 'does this break the release'; we rarely manage to find the time to test Changes *in themselves* and help make sure they work well.
I think that is a good summary.
To provide more background for those not familiar with what has been happening lately, we used openQA (an openSUSE project) in Fedora 22 cycle to test about 20-25 different installation test cases, using the tester name 'coconut'. We don't unfortunately have this system running in public (even though Adam's development machine was), but the pass results were sent into our wiki, and the failures were examined manually. I think this was quite a success and it saved us a lot of manual work. It also detected some issues, even though most of those issues were detected manually as well. But the main benefit here is, I believe, that we can skip a lot of repetitive manual testing (because we know it's covered by openQA) and focus on handling with real bugs, do more exploratory testing, etc.
On the Taskotron front, led mainly by Tim and Martin in the recent days, there has been some improvements and a lot of bug fixes. They are mostly hidden under the hood, but one thing I would mention was enabling tasks to store "task artifacts" - basically any files useful for later review. This is currently not yet exposed in the ResultsDB UI, but the rest of the code is there. Another invisible area which has been improved is that we've been working our way towards disposable test clients. Those will allow us to run destructive checks, and potentially any third-party checks/tasks in the future. This is by no means finished yet.
Future plans
We have a fairly well-defined roadmap for Taskotron development, I believe; we could broaden its actual test coverage, but I think we still have some infrastructure work that probably needs nailing down before we can focus on that, particularly in terms of the disposable test client work, and also I believe in terms of results management.
Disposable test clients are now our current priority in Taskotron. We also work on emitting fedmsg notifications, we need that for integration with Bodhi2. Another middle-term feature could be, I think, finishing up task artifacts to show up in ResultsDB and then using that functionality to displaying only relevant parts of test logs to maintainers (e.g. pidgin maintainer should only see test output relevant for pidgin, and not also everything else) - the last part is actually already implemented, but the dots are not connected yet.
I'd definitely like to see us extending openQA coverage in the post-F22 'quiet time'. There are still many validation tests that could successfully be automated, which gives us much better coverage and reduces the manual testing burden. I'm certainly intending to help work on this. We could also consider soliciting wider contributions of openQA tests; I think the openQA experiment has been fairly successful, and I could certainly envisage some cases where maintainers might want to have tests related to their packages in openQA.
I don't really know what Adam's vision was in his last sentence. But in general, yes, automating more installation/fedup/system basics tests seems to be in the plans and worthwhile.
Something I've been kicking around with release engineering a bit, and which will probably come up at Flock, is the possibility of revising the whole TC/RC approach to release composing. It's getting fairly old, and is built on some assumptions that are becoming somewhat outdated: that building and consuming composes are inherently expensive operations and hence need to be 'rationed' and carefully, manually handled. In an ideal world, I think it would be nice to take a more modern, CI-ish approach: we should be doing the whole compose process daily (not just boot.isos and lives) and running automated tests on the daily composes (openQA is already capable of this), and improving the efficiency of the process whereby updates get into composes. The 'special package requests' in TC/RC compose requests have become kinda a permanent feature, but they were never meant to be. I'd prefer if we could make the process whereby updates move to stable vastly more efficient, and perhaps have a slightly different process for handling builds we want to put in TCs/TCs but which we don't yet want to push stable (the most common case being the packages that are actually involved in *doing composes*, so in order to test them, we need to build composes with them) - something more visible and less clunky than the current 'list them in TC/RC requests and they go into a manually maintained bleed repo' approach.
Things we need
Kinda tied in to the above: Bodhi 2.0, and more efficient releng processes, are the big two I can think of - faster update pushes and composes. I think tflink and kparal are in a better position than I to know about any major resources we need.
I don't say we "need" Bodhi2, but it would definitely help us to get rid of some very nasty code (pushing comments to Bodhi), probably improving speed and reliability of our checks considerably (we often crash on Bodhi server errors).
Another thing that would help a lot, I think, are some improvements in the TC/RC engineering process, for example sending fedmsgs after a TC is complete, or having a better structure metadata describing this compose (for example, a json structure allowing us to see what different ISO images are available for Server product, their filepaths and types - DVD, netinst). We have a lot of black magic for this already, and it mostly works, but such improvements would simplify the code a lot of make it more reliable, especially when some changes are introduced.
For Taskotron itself, I don't have a feeling we're blocked on something or urgently need something done, but from the infrastructure point of view, Tim will definitely be able to provide much better details.
On Tue, 2 Jun 2015 07:02:06 -0400 (EDT) Kamil Paral kparal@redhat.com wrote:
Hello,
I have added some notes below.
The other thing to settle is what we actually want to report. Matt provided the following guidelines:
- the current state of the subproject
- future plans
- things the team needs from the rest of the project
- any blockers we can help unblock
- big resource requests?
Current state
I'd say we have a fairly solid process in place now for release validation, and everyone's probably more or less familiar with it. We also have the update testing process nailed down, though I still think we can improve it in some ways, and we *still* want Bodhi 2.0 :)
We have taskotron running for some package tests and openQA for some installation validation tests. openQA is I think still running on openSUSE boxes, but we have some work done on containerization to potentially move them to Fedora hosts (and I've also been working on packaging the openQA bits for Fedora).
I think we might still be lacking a bit in terms of Change testing; there's a personpower issue there, though. We usually wind up only cherrypicking the most obviously potentially disruptive Changes for testing, and usually from the perspective of 'does this break the release'; we rarely manage to find the time to test Changes *in themselves* and help make sure they work well.
I think that is a good summary.
To provide more background for those not familiar with what has been happening lately, we used openQA (an openSUSE project) in Fedora 22 cycle to test about 20-25 different installation test cases, using the tester name 'coconut'. We don't unfortunately have this system running in public (even though Adam's development machine was), but the pass results were sent into our wiki, and the failures were examined manually. I think this was quite a success and it saved us a lot of manual work. It also detected some issues, even though most of those issues were detected manually as well. But the main benefit here is, I believe, that we can skip a lot of repetitive manual testing (because we know it's covered by openQA) and focus on handling with real bugs, do more exploratory testing, etc.
On the Taskotron front, led mainly by Tim and Martin in the recent days, there has been some improvements and a lot of bug fixes. They are mostly hidden under the hood, but one thing I would mention was enabling tasks to store "task artifacts" - basically any files useful for later review. This is currently not yet exposed in the ResultsDB UI, but the rest of the code is there. Another invisible area which has been improved is that we've been working our way towards disposable test clients. Those will allow us to run destructive checks, and potentially any third-party checks/tasks in the future. This is by no means finished yet.
This sounds right to me. Several bugfixes, some improvements but most of them aren't immediately obvious if you're not looking for them.
At the moment, working on laying the groundwork for our future plans.
Future plans
We have a fairly well-defined roadmap for Taskotron development, I believe; we could broaden its actual test coverage, but I think we still have some infrastructure work that probably needs nailing down before we can focus on that, particularly in terms of the disposable test client work, and also I believe in terms of results management.
Disposable test clients are now our current priority in Taskotron. We also work on emitting fedmsg notifications, we need that for integration with Bodhi2. Another middle-term feature could be, I think, finishing up task artifacts to show up in ResultsDB and then using that functionality to displaying only relevant parts of test logs to maintainers (e.g. pidgin maintainer should only see test output relevant for pidgin, and not also everything else) - the last part is actually already implemented, but the dots are not connected yet.
I don't disagree with what kamil wrote here but I'd like to re-frame and reword it a little.
Taskotron Future Plans ======================
The biggest long term goal for Taskotron right now is what I'm calling dist-git style tasks. The big idea here is to enable packagers to have tasks run for their packages by just pushing changes to git, similar to how dist-git works for packages (we may end up using the same repos but that's TBD).
In order to enable this long term goal, we've broken it up into several smaller chunks: - Disposable Clients: One of the best ways to provide a good, repeatable platform for tasks is to use VMs spawned from a common image which are created before every task and destroyed after every task. This method is what we're calling "Disposable Clients" and is the current focus of our development activity
- Finding a Home for the Tasks: Do we use dist-git? Do we use a separate git repository like dist-git? Find a home for the tasks and figure out the basic methods by which everything will work.
- Locating some "guinea pig" volunteer maintainers Start small to make sure that everything is working like we plan - finding volunteers who are willing to help us iron out the kinks should make the eventual roll-out to all Fedora packages smoother
The current plan is to have all this at least in stg by the end of this calendar year. I hope that things will be done faster than that but I choose to be conservative in my estimates to cover for when things go wrong. The best tools are the ones that people don't overtly notice and having the smoothest possible rollout helps there.
Another somewhat related longer term goal for Taskotron is to enable Fedora contributors to write and submit tasks without much intervention from us. That will require most of the things for dist-git style tasks in addition to:
- Rework our triggering mechanism That code was meant to live just long enough for us to figure out what we really needed and it's not really capable of fulfilling this role of allowing non-maintainers to schedule or add tasks. We knew this was going to be needed eventually.
After that is beyond what my crystal ball can see clearly. There are plenty of things that we could focus on including (in no particular order) beaker integration, getting more strict build/update gating in place, supporting EPEL, supporting rawhide, supporting more testing types ... the list is long enough to keep us busy for years :)
That being said, we'll cross that bridge once we get there - who knows what'll change before then and I don't see a point in spending energy to plan that far out given how quickly things change. I do expect the rate of change to increase once we get the currently planned things into place.
Other Automation Plans ======================
We're still making slow progress on getting a beaker deployment in place. That should allow us to do more types of testing - especially bare metal testing, ks install testing and a few other things that don't fit well into Taskotron's current execution model.
This is being done on a best-effort basis right now, so I have no specific ETA for a full production system but the staging setup should be done before too long.
<snip>
Things we need
Kinda tied in to the above: Bodhi 2.0, and more efficient releng processes, are the big two I can think of - faster update pushes and composes. I think tflink and kparal are in a better position than I to know about any major resources we need.
I don't say we "need" Bodhi2, but it would definitely help us to get rid of some very nasty code (pushing comments to Bodhi), probably improving speed and reliability of our checks considerably (we often crash on Bodhi server errors).
Another thing that would help a lot, I think, are some improvements in the TC/RC engineering process, for example sending fedmsgs after a TC is complete, or having a better structure metadata describing this compose (for example, a json structure allowing us to see what different ISO images are available for Server product, their filepaths and types - DVD, netinst). We have a lot of black magic for this already, and it mostly works, but such improvements would simplify the code a lot of make it more reliable, especially when some changes are introduced.
For Taskotron itself, I don't have a feeling we're blocked on something or urgently need something done, but from the infrastructure point of view, Tim will definitely be able to provide much better details.
Yeah, I don't have anything specific and immediate for Taskotron or other automation. More experienced devs or devops/admin folks would probably help us get done a bit faster but that's not a requirement in any way, shape or form.
Eventually, I'd like to find some volunteers amongst Fedora maintainers for the "guinea pig" role as we get dist-git style tasks rolled out but that's not an immediate need and I hope that finding a few folks who are willing to help us iron out the kinks won't be too hard.
Tim
On Tue, 2015-06-02 at 07:02 -0400, Kamil Paral wrote:
and I could certainly envisage some cases where maintainers might want
to have tests related to their packages in openQA.
I don't really know what Adam's vision was in his last sentence. But in general, yes, automating more installation/fedup/system basics tests seems to be in the plans and worthwhile.
Well, openQA can be used for more than just installation tests. SUSE use it for app testing as well. Since we have the openQA system sitting there, it might be interesting to see about using it in this way too, and see if any packagers might be interested in using it to test their packages. It was kind of an idle thought, though.
-
Adam Williamson -
Kamil Paral -
Mike Ruckman -
Sudhir D -
Tim Flink