On Thu, 2022-10-20 at 07:35 +0200, Martin Pitt wrote:
Hello Adam,
Adam Williamson [2022-10-19 8:31 -0700]:
On Wed, 2022-10-19 at 07:13 +0200, Martin Pitt wrote:
Hello Adam,
Adam Williamson [2022-10-18 10:37 -0700]:
that would define a critical path for Server. That would mean updates containing any of those packages or their dependencies would be 'critical path' updates, meaning they have higher requirements to be pushed stable (+2 karma or 14 days waiting, as opposed to +1 karma or 7 days waiting for regular updates)
What is the reason for increasing the waiting period, what would it achieve?
It's essentially a byproduct. Please see the devel@ discussion I linked. We do not currently have any other good way to gate updates on the openQA tests. We can invent one but it's substantially more work than just adding things to the critpath.
The devel@ thread does not mention/justify the inceased waiting period at all. So it's just intrinsic to the "critical path" definition? Because all of its other properties totally make sense with any other explicit wait time.
Ah, yes, sorry if that wasn't clear. That is essentially the entire original practical purpose of critpath: packages in the critpath are subject to increased requirements. Exactly what the requirements for critpath and non-critpath packages *are* has changed several times over the years, but as of now, AIUI, non-critpath updates can be pushed stable if they have +1 karma or have been in updates-testing for a week. critpath updates can be pushed stable if they have +2 karma or have been in updates-testing for two weeks.
Note, if you set e.g. +3 autopush on your update, that means it will be *automatically* pushed when it reaches +3 with no -1s, but you can still *manually* push it as long as it reaches the requirements above. The minimum requirements act as floors for autopush; you can't set an autopush of 1 for a critpath update. Well, you shouldn't be able to. I think we've had bugs with that before...
I came along later and kinda piggybacked on the critical path as the set of packages to run openQA updates tests on, simply because I needed *some* way to reduce the set from "all updates" and it was handy and seemed kind appropriate.
I'm 75% done with the "do it by adding things to critpath" approach because nobody objected to that idea on devel@ , so it would've been great to have this feedback in *that* discussion :(
Sorry, I'm not on devel@ (email overflow, plus I was on PTO when that happened).
The issues you raise are all reasonable, but kind of not something new: we've never had an official way to override the critical path requirements if we think a package has good enough CI, so if you want to call that a problem, it's been a problem forever.
Yes, it is. It seems utterly hard to get Fedora to do a "tests gate by default" policy, as pretty much everything is opt-in. Opt-in gating is essentially a complete waste of time IMO and just makes everyone involved grumpy.
Note, gating acts in a kind of "required, but not sufficient" way regarding stable push. An update passing gating doesn't mean it can be pushed stable: it has to pass gating *and* meet the karma/wait time requirement.
Aside from that: so, as for having a "gate on all tests" kind of thing...until fairly recently we couldn't really do that. The tricky bit is knowing what "all tests" are for any given Thing. But since: https://pagure.io/fedora-infrastructure/issue/8989#comment-802213 happened, I guess we could - theoretically, at least - do that. We would need to make sure both Fedora CI and openQA are submitting queued "results" for every test they schedule (openQA does not do this currently, partly because I'd forgotten about this whole thing until reading this mail). Then I guess it would be possible to somehow teach greenwave about this; we'd have to invent a new kind of "all tests" requirement type for greenwave and then teach greenwave that when it sees that requirement type, it needs to query resultsdb for "queued" results to figure out what tests are actually scheduled for the thing in question. There's still a kinda small race condition there - gating status would I guess be "go" for a small window after the Thing was created but before the first test system managed to create a "queued" result. I guess this requirement type could assume there must be at least *one* test run on the Thing to avoid that issue? Anyway, yeah, it's an engineering challenge that I guess just nobody really got around to yet.
It does seem like a tricky thing in a way too, because the whole gating system is designed to be fairly open: we have two main test systems ATM (openQA and Fedora CI), but in theory anyone else can set one up and submit test results that would be considered by greenwave; all it needs is credentials to submit results to resultsdb. And we could add new tests that run on many things ("generic" tests) to Fedora CI or openQA at any time, really. So to use this you'd have to have a high degree of trust that all the tests added by anyone running a 'production' test system for Fedora would be sensible to gate your update on. Or be prepared to issue a lot of waivers, I guess.
We can leave cockpit out of the definition if you like, but that would mean it won't be gated by the distribution-wide gating policy. As I mentioned in the devel@ discussion, an alternative is to configure gating in the package repos - see https://gating-greenwave.readthedocs.io/en/latest/policies.html#remoterule-c...
- but this requires the package maintainer(s) to maintain the gating
policy.
Fully agreed, let's not do this. The current form of gating.yml is ... not useful and just a pain.
It also has the disadvantage that we don't resolve dependencies like we do for the critpath. If we put cockpit in the critpath definition, then cockpit *and all its dependencies* are added to the critpath, and thus will have the tests run and be gated on them. So if some dependency of cockpit breaks cockpit, we'll find out, and the update will be gated.
Ah, that is a good thing! The thread only loosely mentioned this:
Ideally, we should have reverse dependency updates to trigger FreeIPA tests and give their results a visibility but don't block on failures.
(FWIW, that is not my definition of "ideally"). If this is the case, I'm in. We can still do the "increase Karma" workaround, if the waiting period is not the primary reason why you want to do this, but just a byproduct of using critical-path. Then that would feel much less like an abuse.
Yeah, my point here is definitely not to apply the increased requirements, that's just a byproduct.
On the idea of having test systems submit feedback: I thought about this, and the more I think about it, the less it seems like a hack and the more it seems, well, appropriate. Why isn't it just...the right thing to do? openQA exercises cockpit pretty extensively. Your tests also do. Assuming your CI system does actually test the Fedora update in a correct Fedora context, like openQA does...it seems totally valid to me that both our test systems should submit a +1 to Bodhi if the tests pass. Why not? They're testing the software like a human would (probably better). If I was doing manual testing I might just open up Cockpit, click around a bit, and say yeah, that looks good and give it a +1; if that's what we're expecting of humans, what's wrong with having an automated test system that does things rather more thoroughly give +1s?
So, if we do that...your system is 1, my system is 1, that makes 2, which is...all the feedback the updates need to be pushed stable. ;)