Hi folks!
I have one of those definitional quandaries and I figured I'd throw it at the lists for some input.
Right now, we kind of take advantage of the "critical path" concept for automated update testing and gating via openQA.
openQA does not test all updates, only critical path updates plus updates containing any package on a short allowlist.
Bodhi and Greenwave do not gate all updates on the openQA tests since not all updates are tested; again we use the critpath definition. If an update is critical path, it gets gated on the openQA tests. If it isn't, it doesn't.
If you've been paying attention, that means there's a bit of a hole: the packages on the 'allowlist'. These are tested, but not gated.
What's on the allowlist? Basically, FreeIPA-related packages. We have a good set of FreeIPA tests in openQA, and both I and the FreeIPA team wanted relevant updates to be tested with them. But those packages are not on the critical path. So I set up this 'allowlist' mechanism.
However, the lack of gating kinda sucks. I would much prefer if we could gate these updates as well as just testing them.
The obvious thing to do is add the FreeIPA-related packages to the critical path, but that really is not supported by the critical path definition:
https://fedoraproject.org/wiki/Critical_path_package
which defines it as packages required to "perform the most fundamental actions on a system", with a list of specific areas, none of which is "run a domain server".
So...what to do?
I can think of I guess four options:
1. Broaden the definition of the "critical path" somehow. We could just write in that it includes FreeIPA functionality, I guess, though that seems special purpose. We could broaden it to include any functionality covered by the release criteria, which would be quite a big increase but seems like a reasonable and concise definition. Or we could write in that it includes any functionality that is exercised by the gating openQA tests, though that seems a bit arbitrary - there's no particularly great organizing principle to the openQA tests we choose to run on updates, if I'm honest, it's a sort of grab bag I came up with.
2. Keep the current "critical path" concept but define a broader group of "gated packages" somewhere (could be comps or somewhere else). This would require engineering work - we'd have to touch probably the openQA scheduler, Bodhi, and greenwave configs. It's also another maintenance burden.
3. Add gating config to each allowlisted package repo's gating.yml one by one. I don't really like this option. It'd be a chunk of work to do initially, and multiples the maintenance required when the list of tests to gate on changes for some reason.
4. Do nothing, just keep only gating things that are "critical path" on the current definition. In a utopian future, we'd manage to deploy openQA in the cloud or get a giant pile of super fast servers so we could test and gate *every* update, and this wouldn't be an issue any more.
What do folks think? Any preferences? Thanks!
On Mon, Aug 29, 2022 at 03:50:02PM -0700, Adam Williamson wrote: ...snip...
I can think of I guess four options:
- Broaden the definition of the "critical path" somehow. We could just
write in that it includes FreeIPA functionality, I guess, though that seems special purpose. We could broaden it to include any functionality covered by the release criteria, which would be quite a big increase but seems like a reasonable and concise definition. Or we could write in that it includes any functionality that is exercised by the gating openQA tests, though that seems a bit arbitrary - there's no particularly great organizing principle to the openQA tests we choose to run on updates, if I'm honest, it's a sort of grab bag I came up with.
I think this one is best... perhaps we could add something like 'and such package groups as working groups decide are critical to their edition' ?
kevin
From my perspective, anything that blocks the release is on the critical path. So any time there's a violation of the release criteria and the package is not on the critical path definition, that's a bug in the definition.
I recognize that this is a somewhat naïve view. For one, it may broaden the definition beyond the current capacity of our test infrastructure. It also may broaden the definition beyond what maintainers are willing to put up with. These are both legitimate problems. But the closer we can get to this ideal state, the better.
For anyone who is curious, I just searched for all accepted blockers in the "Fedora" product in Bugzilla. 327 components have been a blocker at least once. Some of those may no longer be blocking and others will be added over time as our criteria change. The full list with counts is at https://bcotton.fedorapeople.org/release-blocking-components.csv if you're interested.
On Tue, 2022-08-30 at 09:14 -0400, Ben Cotton wrote:
From my perspective, anything that blocks the release is on the critical path. So any time there's a violation of the release criteria and the package is not on the critical path definition, that's a bug in the definition.
I recognize that this is a somewhat naïve view. For one, it may broaden the definition beyond the current capacity of our test infrastructure. It also may broaden the definition beyond what maintainers are willing to put up with. These are both legitimate problems. But the closer we can get to this ideal state, the better.
For anyone who is curious, I just searched for all accepted blockers in the "Fedora" product in Bugzilla. 327 components have been a blocker at least once. Some of those may no longer be blocking and others will be added over time as our criteria change. The full list with counts is at https://bcotton.fedorapeople.org/release-blocking-components.csv if you're interested.
Honestly, something along these lines would be my preference too, I just don't know if others would agree/support changing the critical path definition to "all release-blocking functionality" rather than "functionality needed to boot a basically-functional system".
Thanks for the data! I will see if I can diff it against the current critpath definition; that would be interesting.