On 03/17/2017 06:23 PM, Adam Williamson wrote:
Hi folks!
So Colin and I had a bit of a chat about openQA Atomic testing 'integration', and one of the things that came up was the state of the 'compose check report' emails for Atomic tests.
If you're signed up to test@ or devel@ you may have noticed that a 'compose check report' mail is sent out for every mainline Fedora compose openQA tests, with info on the numbers of passed and failed tests and some other stuff.
When we set up openQA to test the Atomic Host OStree installer image and to test the nightly 'two-week Atomic' composes, I asked whether we should generate those mails, and who they should go to.
The answer I got - and the way it's configured right now - was that people only wanted the mails produced when a test *failed*, and they should be mailed to this list plus to one person directly (I think at first it was Adam Miller, it now seems to be Mike McGrath).
I've just verified that this is actually working: when openQA tests the 'two-week Atomic' composes and a failure occurs, the mail does get sent. However, there hasn't been a failure of the test since - AFAICS - 2016-10-01. So that's why no such mails have been sent out recently. I got Mike to check, and he actually did get a mail on 20167-10-01, when the test last failed.
woot for passing tests! Adam, do you know if there is any way to send a weekly report that basically states how many tests ran and how many passed? That would at least let us know that they were there and still running and passing.
I imagine now that the tests results are in resultsdb the answer is going to be something related to that.
I think the mails that have been sent to cloud@ were never approved through moderation, so they never actually appeared here. It'd be good if a list moderator could check if they see a few 'compose check' mails hung up in moderation, or something.
I found this thread from a long time ago on this topic: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/...
Looks like someone tried to add that mail address to the list but doesn't seem to be working. These emails are not in the held messages from what I can tell. I think I am a moderator and not an administrator.
https://wiki.list.org/DOC/Difference%20Between%20a%20Moderator%20and%20Admin...
I need to get jzb to make me an admin so I can investigate.
So, I just wanted to give a quick heads-up on that, and say that if anyone would like that configuration changed, I can do it easily enough. We could have a report generated for every compose as for the 'main' composes, but it'd be quite dull I think, because there's only one test and it almost always passes. We can also change the list of addresses that receive the mails when they're sent, if it should be changed.
Indeed. Thanks.
I think the real answer in the long run is to have dashboards that display test results in a nice way (i'm talking flashy gui web page stuff). We could have one dashboard that highlighted failures in atomic host across different composes and then a dashboard that highlighted failures/passes in a particular or for a particular ostree. I'm not asking *YOU* for this, but I think that is the answer long term. The emails would then have a link to the test results as well as the dashboard where you can see all test results.
Also do let me know if running more tests on the Atomic installer image would be desirable. For now we just run a single test - a straight- through install test on x86_64 BIOS - since that's all I was asked for initially. We could run the same test on UEFI, if desired, and we could run some of the post-install tests that are run on other images, and we could run some of the install variant tests; I just don't know which ones are relevant / useful for the Atomic installer image.
Yes! we would love a UEFI test and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job. I thought openQA was more for interactive install testing, so please let me know where I'm wrong. roshi knows openQA and atomic-host-tests so he might be able to comment here.
Thanks Adam, Dusty