Hi folks!
So Colin and I had a bit of a chat about openQA Atomic testing 'integration', and one of the things that came up was the state of the 'compose check report' emails for Atomic tests.
If you're signed up to test@ or devel@ you may have noticed that a 'compose check report' mail is sent out for every mainline Fedora compose openQA tests, with info on the numbers of passed and failed tests and some other stuff.
When we set up openQA to test the Atomic Host OStree installer image and to test the nightly 'two-week Atomic' composes, I asked whether we should generate those mails, and who they should go to.
The answer I got - and the way it's configured right now - was that people only wanted the mails produced when a test *failed*, and they should be mailed to this list plus to one person directly (I think at first it was Adam Miller, it now seems to be Mike McGrath).
I've just verified that this is actually working: when openQA tests the 'two-week Atomic' composes and a failure occurs, the mail does get sent. However, there hasn't been a failure of the test since - AFAICS - 2016-10-01. So that's why no such mails have been sent out recently. I got Mike to check, and he actually did get a mail on 20167-10-01, when the test last failed.
I think the mails that have been sent to cloud@ were never approved through moderation, so they never actually appeared here. It'd be good if a list moderator could check if they see a few 'compose check' mails hung up in moderation, or something.
So, I just wanted to give a quick heads-up on that, and say that if anyone would like that configuration changed, I can do it easily enough. We could have a report generated for every compose as for the 'main' composes, but it'd be quite dull I think, because there's only one test and it almost always passes. We can also change the list of addresses that receive the mails when they're sent, if it should be changed.
Also do let me know if running more tests on the Atomic installer image would be desirable. For now we just run a single test - a straight- through install test on x86_64 BIOS - since that's all I was asked for initially. We could run the same test on UEFI, if desired, and we could run some of the post-install tests that are run on other images, and we could run some of the install variant tests; I just don't know which ones are relevant / useful for the Atomic installer image.
On 03/17/2017 06:23 PM, Adam Williamson wrote:
Hi folks!
So Colin and I had a bit of a chat about openQA Atomic testing 'integration', and one of the things that came up was the state of the 'compose check report' emails for Atomic tests.
If you're signed up to test@ or devel@ you may have noticed that a 'compose check report' mail is sent out for every mainline Fedora compose openQA tests, with info on the numbers of passed and failed tests and some other stuff.
When we set up openQA to test the Atomic Host OStree installer image and to test the nightly 'two-week Atomic' composes, I asked whether we should generate those mails, and who they should go to.
The answer I got - and the way it's configured right now - was that people only wanted the mails produced when a test *failed*, and they should be mailed to this list plus to one person directly (I think at first it was Adam Miller, it now seems to be Mike McGrath).
I've just verified that this is actually working: when openQA tests the 'two-week Atomic' composes and a failure occurs, the mail does get sent. However, there hasn't been a failure of the test since - AFAICS - 2016-10-01. So that's why no such mails have been sent out recently. I got Mike to check, and he actually did get a mail on 20167-10-01, when the test last failed.
woot for passing tests! Adam, do you know if there is any way to send a weekly report that basically states how many tests ran and how many passed? That would at least let us know that they were there and still running and passing.
I imagine now that the tests results are in resultsdb the answer is going to be something related to that.
I think the mails that have been sent to cloud@ were never approved through moderation, so they never actually appeared here. It'd be good if a list moderator could check if they see a few 'compose check' mails hung up in moderation, or something.
I found this thread from a long time ago on this topic: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/...
Looks like someone tried to add that mail address to the list but doesn't seem to be working. These emails are not in the held messages from what I can tell. I think I am a moderator and not an administrator.
https://wiki.list.org/DOC/Difference%20Between%20a%20Moderator%20and%20Admin...
I need to get jzb to make me an admin so I can investigate.
So, I just wanted to give a quick heads-up on that, and say that if anyone would like that configuration changed, I can do it easily enough. We could have a report generated for every compose as for the 'main' composes, but it'd be quite dull I think, because there's only one test and it almost always passes. We can also change the list of addresses that receive the mails when they're sent, if it should be changed.
Indeed. Thanks.
I think the real answer in the long run is to have dashboards that display test results in a nice way (i'm talking flashy gui web page stuff). We could have one dashboard that highlighted failures in atomic host across different composes and then a dashboard that highlighted failures/passes in a particular or for a particular ostree. I'm not asking *YOU* for this, but I think that is the answer long term. The emails would then have a link to the test results as well as the dashboard where you can see all test results.
Also do let me know if running more tests on the Atomic installer image would be desirable. For now we just run a single test - a straight- through install test on x86_64 BIOS - since that's all I was asked for initially. We could run the same test on UEFI, if desired, and we could run some of the post-install tests that are run on other images, and we could run some of the install variant tests; I just don't know which ones are relevant / useful for the Atomic installer image.
Yes! we would love a UEFI test and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job. I thought openQA was more for interactive install testing, so please let me know where I'm wrong. roshi knows openQA and atomic-host-tests so he might be able to comment here.
Thanks Adam, Dusty
On Fri, Mar 17, 2017, at 11:00 PM, Dusty Mabe wrote:
Yes! we would love a UEFI test
Yes; conceptually most of the important scenarios we run through for Workstation/Server (e.g. dm-crypt versus not) we should also do for Atomic Host.
Adam, where is the git containing the input test suite for OpenQA?
and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job.
This part I don't quite agree with - remember the cloud images are built with the installer ISO, and *those* are tested already with a-h-t. There's going to be very little that's specific to the ISO path versus the cloud image - which is part of the whole idea of sharing the same exact ostree commit across them.
I thought openQA was more for interactive install testing,
It does do more than that - however I see it as mostly valuable for testing interactive Anaconda runs with the OpenCV approach.
IMO OpenQA's use of OpenCV is less interesting for testing "headless/non-GUI" bits, as is the case for Atomic Host. (We do of course have Cockpit and the OpenShift web console, but both of those use Selenium[1][2] already)
[1] https://github.com/cockpit-project/cockpit/blob/master/test/avocado/selenium... [2] https://github.com/openshift/origin-web-console/blob/master/package.json#L64
On 03/18/2017 10:42 AM, Colin Walters wrote:
On Fri, Mar 17, 2017, at 11:00 PM, Dusty Mabe wrote:
Yes! we would love a UEFI test
Yes; conceptually most of the important scenarios we run through for Workstation/Server (e.g. dm-crypt versus not) we should also do for Atomic Host.
Adam, where is the git containing the input test suite for OpenQA?
and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job.
This part I don't quite agree with - remember the cloud images are built with the installer ISO, and *those* are tested already with a-h-t. There's going to be very little that's specific to the ISO path versus the cloud image - which is part of the whole idea of sharing the same exact ostree commit across them.
small nit here.. I think you are right that the cloud images are built with the "install tree" that is embedded in the ISO, but that isn't quite the same thing as being installed with the ISO. Either way, I think your point is valid and that the value is in testing different install time configurations rather than fully testing the installed tree. Verifying no systemd unit tests fail and running at least one "docker test" couldn't hurt though.
Dusty
On Sat, 2017-03-18 at 10:42 -0400, Colin Walters wrote:
On Fri, Mar 17, 2017, at 11:00 PM, Dusty Mabe wrote:
Yes! we would love a UEFI test
Yes; conceptually most of the important scenarios we run through for Workstation/Server (e.g. dm-crypt versus not) we should also do for Atomic Host.
Adam, where is the git containing the input test suite for OpenQA?
https://pagure.io/fedora-qa/os-autoinst-distri-fedora
be aware, though, that the way tests work in openQA is somewhat idiosyncratic and very...specific. I can give you lots of details if you like, but the short version is that what we ultimately wind up actually *doing* so far as Atomic's concerned is: any time a fedmsg 'compose complete!' message appears for a compose that contains an Atomic Host installer image, openQA will spin up a VM with that ISO attached as an optical drive and a single empty hard disk, then do nearly the simplest possible install you can do. It just goes to INSTALLATION DESTINATION, selects the hard disk as the target and agrees to let anaconda partition it automatically, starts the install, sets a root password, and creates a user called 'test'. Once the install completes, it boots the installed system, logs in as root, checks whether there are any crash notifications or AVCs, uploads some resource usage information, and that's it.
and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job.
This part I don't quite agree with - remember the cloud images are built with the installer ISO, and *those* are tested already with a-h-t. There's going to be very little that's specific to the ISO path versus the cloud image - which is part of the whole idea of sharing the same exact ostree commit across them.
I thought openQA was more for interactive install testing,
It does do more than that - however I see it as mostly valuable for testing interactive Anaconda runs with the OpenCV approach.
IMO OpenQA's use of OpenCV is less interesting for testing "headless/non-GUI" bits, as is the case for Atomic Host. (We do of course have Cockpit and the OpenShift web console, but both of those use Selenium[1][2] already)
openQA is perfectly *capable* of doing non-graphical testing; we do actually do quite a lot of this just because it's there and relatively convenient to use. We don't use screen matching to do this, though. Quite a long time ago openQA grew the ability to do this kind of stuff through the serial console; you can use a convenience function to run any command at a console and either assert that it succeeds (which is implemented by echoing its return code plus a hash of the command - purely for identification purposes - to the serial console, then identifying when that text hits the serial console and extracting the return code), or do some kind of match on whatever the command sends to stdout.
openQA doesn't have any huge benefits as a way of doing this compared to any of the other zillion implementations of 'spin up a VM and run some commands in it', though, beyond the fact that it can be useful to have a video recording of what was actually displayed on the console during the run (which is something very few other test systems provide). It has a couple of drawbacks, principally the idiosyncracies of how tests actually work in openQA; it's possible to hide this from people by writing some kinda middle layer code which would allow you to just dump a test script in a git repo and the middle layer code would automatically generate an openQA job that runs the script, I started writing something along those lines as a side project a while back just as a proof-of-concept that we could run tests from dist-git in openQA, but I've never finished it.
If 'the atomic-host-tests' are just scripts that can be run from a console we could very trivially run them post-install as part of the openQA testing of the installer image, if desired. I agree with Colin that this is quite unlikely to produce different results from running the same tests on the equivalent Atomic host disk images in Taskotron or Autocloud, but it wouldn't really *hurt* anything (besides eating up a bit more of our limited openQA worker resources).
We do also do some basic testing of Cockpit in openQA, though nowhere near as much as its own test suite covers.
On 03/17/2017 11:00 PM, Dusty Mabe wrote:
On 03/17/2017 06:23 PM, Adam Williamson wrote:
Hi folks!
So Colin and I had a bit of a chat about openQA Atomic testing 'integration', and one of the things that came up was the state of the 'compose check report' emails for Atomic tests.
If you're signed up to test@ or devel@ you may have noticed that a 'compose check report' mail is sent out for every mainline Fedora compose openQA tests, with info on the numbers of passed and failed tests and some other stuff.
When we set up openQA to test the Atomic Host OStree installer image and to test the nightly 'two-week Atomic' composes, I asked whether we should generate those mails, and who they should go to.
The answer I got - and the way it's configured right now - was that people only wanted the mails produced when a test *failed*, and they should be mailed to this list plus to one person directly (I think at first it was Adam Miller, it now seems to be Mike McGrath).
I've just verified that this is actually working: when openQA tests the 'two-week Atomic' composes and a failure occurs, the mail does get sent. However, there hasn't been a failure of the test since - AFAICS - 2016-10-01. So that's why no such mails have been sent out recently. I got Mike to check, and he actually did get a mail on 20167-10-01, when the test last failed.
woot for passing tests! Adam, do you know if there is any way to send a weekly report that basically states how many tests ran and how many passed? That would at least let us know that they were there and still running and passing.
I imagine now that the tests results are in resultsdb the answer is going to be something related to that.
I think the mails that have been sent to cloud@ were never approved through moderation, so they never actually appeared here. It'd be good if a list moderator could check if they see a few 'compose check' mails hung up in moderation, or something.
I found this thread from a long time ago on this topic: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/...
Looks like someone tried to add that mail address to the list but doesn't seem to be working. These emails are not in the held messages from what I can tell. I think I am a moderator and not an administrator.
https://wiki.list.org/DOC/Difference%20Between%20a%20Moderator%20and%20Admin...
I need to get jzb to make me an admin so I can investigate.
I am now an admin and I have subscribed rawhide@fedoraproject.org
On Fri, 2017-03-17 at 23:00 -0400, Dusty Mabe wrote:
I've just verified that this is actually working: when openQA tests the 'two-week Atomic' composes and a failure occurs, the mail does get sent. However, there hasn't been a failure of the test since - AFAICS - 2016-10-01. So that's why no such mails have been sent out recently. I got Mike to check, and he actually did get a mail on 20167-10-01, when the test last failed.
woot for passing tests! Adam, do you know if there is any way to send a weekly report that basically states how many tests ran and how many passed? That would at least let us know that they were there and still running and passing.
I imagine now that the tests results are in resultsdb the answer is going to be something related to that.
The thing that sends the per-compose reports is just a script that I wrote, called check-compose:
https://pagure.io/fedora-qa/check-compose
there's a fedmsg consumer which notices when the last test for a compose is run, and runs check-compose for that compose with a configuration that results in the mails being sent.
check-compose currently has no ability to do a weekly summary report or anything like it, it can only generate reports for exactly one compose at a time. It wouldn't be too difficult to write such a thing (either as an extension to check-compose or as a separate project), but someone would have to do it.
check-compose queries openQA directly, rather than resultsdb; I wrote it before we were forwarding results to resultsdb. It can't be ported to only use resultsdb as it would still have to query openQA directly for some stuff that isn't forwarded to resultsdb and really can't be (like the logs it uses to report on system resource usage and stuff). At some point I'd like to adjust it to use rdb as well as direct openQA queries so it can include info on results from other test systems, but it's not at the top of my priority list.
You could write a simple 'weekly summary' kinda script purely from the info in resultsdb, and that way you could consider the results from openQA and autocloud (currently) / taskotron (future?) together. I do believe that's the general idea for this kind of status reporting indeed - if you want that kind of info, build something that uses resultsdb, whether it's some kinda web dashboard or script or whatever you like.
Yes! we would love a UEFI test and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job. I thought openQA was more for interactive install testing, so please let me know where I'm wrong. roshi knows openQA and atomic-host-tests so he might be able to comment here.
You can do just about anything with openQA, really; we *could* do all the testing we're currently doing in other systems in it. But it wouldn't necessarily be the *best* way to do every kind of testing. :) Interactive install testing is the classic example of a thing openQA can do quite well that few other things can do at all.
On Fri, Mar 17, 2017 at 03:23:45PM -0700, Adam Williamson wrote: <snip>
Also do let me know if running more tests on the Atomic installer image would be desirable. For now we just run a single test - a straight- through install test on x86_64 BIOS - since that's all I was asked for initially. We could run the same test on UEFI, if desired, and we could run some of the post-install tests that are run on other images, and we could run some of the install variant tests; I just don't know which ones are relevant / useful for the Atomic installer image. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
Adam,
Thanks for the heads up - I'll try to check if any of the messages are in moderation. As for more tests, I think adding UEFI testing would be beneficial. Does that entail a lot of extra work?
Thanks.
// Mike -- Fedora QA
On Sun, 2017-03-19 at 01:27 -0400, Mike Ruckman wrote:
On Fri, Mar 17, 2017 at 03:23:45PM -0700, Adam Williamson wrote:
<snip> > Also do let me know if running more tests on the Atomic installer image > would be desirable. For now we just run a single test - a straight- > through install test on x86_64 BIOS - since that's all I was asked for > initially. We could run the same test on UEFI, if desired, and we could > run some of the post-install tests that are run on other images, and we > could run some of the install variant tests; I just don't know which > ones are relevant / useful for the Atomic installer image. > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net
Adam,
Thanks for the heads up - I'll try to check if any of the messages are in moderation. As for more tests, I think adding UEFI testing would be beneficial. Does that entail a lot of extra work?
No, it's about thirty seconds of work. Since you and Dusty both asked for it, I'll do that.
On Mon, 2017-03-20 at 12:29 -0700, Adam Williamson wrote:
On Sun, 2017-03-19 at 01:27 -0400, Mike Ruckman wrote:
On Fri, Mar 17, 2017 at 03:23:45PM -0700, Adam Williamson wrote:
<snip> > Also do let me know if running more tests on the Atomic installer image > would be desirable. For now we just run a single test - a straight- > through install test on x86_64 BIOS - since that's all I was asked for > initially. We could run the same test on UEFI, if desired, and we could > run some of the post-install tests that are run on other images, and we > could run some of the install variant tests; I just don't know which > ones are relevant / useful for the Atomic installer image. > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net
Adam,
Thanks for the heads up - I'll try to check if any of the messages are in moderation. As for more tests, I think adding UEFI testing would be beneficial. Does that entail a lot of extra work?
No, it's about thirty seconds of work. Since you and Dusty both asked for it, I'll do that.
Sorry, I kept somehow not quite getting around to this, but it's done now. We now run the install_default test on both BIOS and UEFI for the Atomic installer images in all composes that include them.
Another request that's been sitting around for a while is for a test that does an install to a hard disk big enough that creation of a separate /home partition will be triggered:
https://phab.qa.fedoraproject.org/T689
I'm assuming that's still useful (and in fact we may as well run the same test on the 'regular' installer images too). I'll try and get around to it at some point soon.
On 04/19/2017 06:38 PM, Adam Williamson wrote:
Sorry, I kept somehow not quite getting around to this, but it's done now. We now run the install_default test on both BIOS and UEFI for the Atomic installer images in all composes that include them.
Thanks Adam
Another request that's been sitting around for a while is for a test that does an install to a hard disk big enough that creation of a separate /home partition will be triggered:
https://phab.qa.fedoraproject.org/T689
I'm assuming that's still useful (and in fact we may as well run the same test on the 'regular' installer images too). I'll try and get around to it at some point soon.
Seems like it. Thanks
-
Adam Williamson -
Colin Walters -
Dusty Mabe -
Mike Ruckman