Hi folks! I want to apologize to packagers who have Rawhide updates stuck on failed openQA tests, and provide an update.
There were two big problems today. First, the Server base disk image used for some of the tests got regenerated (this happens weekly). Unfortunately, this meant it got affected by https://bugzilla.redhat.com/show_bug.cgi?id=2259266 , and no longer booted. I should have got that grub2 build untagged or the change reverted sooner - sorry. I was expecting the package to be fixed sooner, and it didn't occur to me that this image being regenerated was a risk. I've now got the package untagged, and forced a rebuild of the disk image using a grub2 build with the patch re-applied (otherwise I'd have had to wait for the next Rawhide compose to rebuild the image). I'm now working through re-running all affected tests.
Second, shell-color-prompt was changed[0] to apply the color prompt to virtual consoles. This broke a large number of openQA tests, as they often need to do things at root consoles, and to do that they need to know when they've actually managed to log in to one, and they do this by screen matching the prompt. Any test which needed to know it was at a root console was failing. I've updated the 'needle' to match the color prompt, and am re-running all affected tests.
I anticipate the mess should be cleaned up in about three or four hours (there are a lot of failures to re-run, and some of the tests take a while to run). Sorry again for the inconvenience.
I'll probably propose adding shell-color-prompt to the critical path. That would mean openQA would have gated the update (allowing me to update the needles *before* it got pushed and broke tests for every other update), but it also just seems appropriate - theoretically, a bad change to the package could cause all sorts of consequences, since it fiddles with the default prompt.
[0] https://src.fedoraproject.org/rpms/shell-color-prompt/c/1dd8c6c16d760cacfb9f...