Lately we've seen a surge of FTI (fails to install) bugs being proposed as freeze exceptions [1] [2]. We generally grant them, because we want the base repo to be in a consistent and buildable state. However, I wonder, isn't this approach mostly relevant for the Final release? Does it make sense to also have this approach for Beta?
The reason why I'm thinking about this is because of course there's some work connected with granting and processing these freeze exceptions (FEs). But at the same time, updates-testing is enabled by default, so users can get the fixed versions immediately, and the fixes can be pushed stable right after the Beta freeze is over. Is the extra FE-related work justified?
One reason I can think of is when the package A in question needs to be used for rebuilding/installing another package B. In that case, if package A is not pushed stable, you can't prepare an update for package B into updates-testing (or can you? Can you build several inter-connected packages together and make a Bodhi update for them? What if you have access rights to just package B but not A?). I do understand that in this case waiting until the freeze lifts might be inconvenient. What if we granted FEs for Beta just in these justified cases but not in general, in order to decrease the processing-work? Is that a good/bad idea?
Or perhaps we can grant FTI FEs automatically? Either always, or in some cases?
What are your thoughts on this?
Thanks, Kamil
[1] https://qa.fedoraproject.org/blockerbugs/milestone/39/beta/buglist [2] https://pagure.io/fedora-qa/blocker-review/issues?status=all&search_patt...
On Tue, Sep 5, 2023 at 12:31 PM Kamil Paral kparal@redhat.com wrote:
Lately we've seen a surge of FTI (fails to install) bugs being proposed as freeze exceptions [1] [2]. We generally grant them, because we want the base repo to be in a consistent and buildable state. However, I wonder, isn't this approach mostly relevant for the Final release?
Yes, you're right.
Does it make sense to also have this approach for Beta?
Since the freeze lifts after the beta release, it's not technically necessary to address these issues for Beta.
The reason why I'm thinking about this is because of course there's some work connected with granting and processing these freeze exceptions (FEs). But at the same time, updates-testing is enabled by default, so users can get the fixed versions immediately, and the fixes can be pushed stable right after the Beta freeze is over. Is the extra FE-related work justified?
The added work for a FE seems pretty minimal to me (3 people write +1, I push it to the accepted FE in about a minute), not sure if the releng perspective is different here though?
One reason I can think of is when the package A in question needs to be used for rebuilding/installing another package B. In that case, if package A is not pushed stable, you can't prepare an update for package B into updates-testing (or can you? Can you build several inter-connected packages together and make a Bodhi update for them? What if you have access rights to just package B but not A?). I do understand that in this case waiting until the freeze lifts might be inconvenient. What if we granted FEs for Beta just in these justified cases but not in general, in order to decrease the processing-work? Is that a good/bad idea?
I don't feel like further complicating our processes and guidelines. And having another special cases defined somewhere isn't going to work imo.
Or perhaps we can grant FTI FEs automatically? Either always, or in some cases?
Yeah, Instead of starting to reject the Beta FTI FEs, I'd say we probably should just automatically approve all of them? Adding something like this to the blockerbugs sounds very easy and straightforward, and would save us some time in the final cycle: *Does bug depend on the FE tracker? Is the reporter "Fedora Fails To Install"? Does it have an update marked as fixing the bug? Fine, approved, next.*
On Tue, Sep 5, 2023 at 2:05 PM Frantisek Zatloukal fzatlouk@redhat.com wrote:
The added work for a FE seems pretty minimal to me (3 people write +1, I push it to the accepted FE in about a minute), not sure if the releng perspective is different here though?
I see it a bit differently. Even when just considering myself, it amounts to frequent email distractions and non-trivial time spent on it (when summed up). I usually don't just blindly type "BetaFE +1", but try to open the Bugzilla ticket (that's slow) and at least skim through it, whether it is really what it claims to be, whether it is FTBFS or also FTI (sometimes I check myself in a VM or in a container), whether it is a non-important package or could possibly impact the release somehow. This total time is multiplied by people involved in the FE process (not just three), although I understand not everyone spends that much time with it. Then there's someone managing the agreement, it can be included in the blocker meeting (if we don't do it async in time), it lowers the readability of the blockerbugs app page because FTI bugs are high in their number, and of course somebody needs to double-check everything when creating a releng request.
It's not terrible, not at all, but especially this cycle I'm starting to think that we need some adjustments (thus this email).
I don't feel like further complicating our processes and guidelines.
Well the process is not presently super simple either. FE SOP [1] says: "FTI bugs may still be rejected in complex cases - e.g. if only one subpackage of an important package fails to install, and the fix might potentially cause problems for more important workflows using other subpackages"
This requires at least opening that Bugzilla ticket and reading through it, to figure out whether this is the case or not. If we swapped the approach - reject FTI FEs at Beta, unless they provide a direct justification during proposal - it might actually simplify things - for our team, at least. And the volume of proposed FTIs would go down. Not sure whether it's an improvement for Fedora in general, though. I'm not looking at simplifying my work at the expense of others, I'm looking for ways to optimize the process.
[1] https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process
Yeah, Instead of starting to reject the Beta FTI FEs, I'd say we probably
should just automatically approve all of them? Adding something like this to the blockerbugs sounds very easy and straightforward, and would save us some time in the final cycle: *Does bug depend on the FE tracker? Is the reporter "Fedora Fails To Install"? Does it have an update marked as fixing the bug? Fine, approved, next.*
I wouldn't need automation, at least in the first iteration. "Image oversized" is not automated either, but we know we can just tag it "accepted" and go on. A similar approach would be fine. But, as cited above from the FE SOP, this might actually bring us some troubles. So I'm not sure whether we can do this generally, without manual inspection.
On Tue, 2023-09-05 at 12:30 +0200, Kamil Paral wrote:
Lately we've seen a surge of FTI (fails to install) bugs being proposed as freeze exceptions [1] [2]. We generally grant them, because we want the base repo to be in a consistent and buildable state. However, I wonder, isn't this approach mostly relevant for the Final release? Does it make sense to also have this approach for Beta?
The reason why I'm thinking about this is because of course there's some work connected with granting and processing these freeze exceptions (FEs). But at the same time, updates-testing is enabled by default, so users can get the fixed versions immediately, and the fixes can be pushed stable right after the Beta freeze is over. Is the extra FE-related work justified?
updates-testing is not enabled by default for the upgrade.
The upgrade process uses whatever repos are enabled *in the current configuration*. So in the "typical" case, you are upgrading from a stable Fedora release with default repo configuration, in which updates-testing is not enabled. Thus updates-testing is not used for the upgrade.
This is why we have the policy of accepting clean FTI fixes during Beta freeze.
I'd like to propose an alternative change: we should make clean FTI cases "automatic freeze exceptions". By "clean" I mean cases where the package was, practically speaking, useless before the fix. Cases where it's just one subpackage of a larger package that was FTI should still be manually checked, especially if the changes are larger than just a straight targeted fix to that subpackage (e.g. a version bump).
That way I or Frantisek or whoever can just tag them accepted as they come in and we don't have to bother voting...
Or perhaps we can grant FTI FEs automatically? Either always, or in some cases?
...oh yeah, that. :D
On Tue, Sep 5, 2023 at 5:22 PM Adam Williamson adamwill@fedoraproject.org wrote:
updates-testing is not enabled by default for the upgrade.
The upgrade process uses whatever repos are enabled *in the current configuration*. So in the "typical" case, you are upgrading from a stable Fedora release with default repo configuration, in which updates-testing is not enabled. Thus updates-testing is not used for the upgrade.
This didn't occur to me, you're right. We could update our instructions to tell people to use "--enablerepo=updates-testing" when upgrading to a development release, or at least add it if they see broken dependencies and the upgrade fails to start, but only limited people would find those instructions. It's definitely better in general to fix those packages in the main repo.
I'd like to propose an alternative change: we should make clean FTI cases "automatic freeze exceptions". By "clean" I mean cases where the package was, practically speaking, useless before the fix. Cases where it's just one subpackage of a larger package that was FTI should still be manually checked, especially if the changes are larger than just a straight targeted fix to that subpackage (e.g. a version bump).
That sounds reasonable. But would we trust packagers to include this important information ("this is not a simple case...") in their FE proposal, or would we still manually check case-by-case?
On Tue, 2023-09-05 at 18:03 +0200, Kamil Paral wrote:
On Tue, Sep 5, 2023 at 5:22 PM Adam Williamson adamwill@fedoraproject.org wrote:
updates-testing is not enabled by default for the upgrade.
The upgrade process uses whatever repos are enabled *in the current configuration*. So in the "typical" case, you are upgrading from a stable Fedora release with default repo configuration, in which updates-testing is not enabled. Thus updates-testing is not used for the upgrade.
This didn't occur to me, you're right. We could update our instructions to tell people to use "--enablerepo=updates-testing" when upgrading to a development release, or at least add it if they see broken dependencies and the upgrade fails to start, but only limited people would find those instructions. It's definitely better in general to fix those packages in the main repo.
I actually prefer the current way, because it gives us a bit of a quality gate over upgrades to Beta just as we have a quality gate over new deployments of it. OK, you can still blow everything up on first update, but at least we have the opportunity to exercise control over the state on that first operation. :D
I'd like to propose an alternative change: we should make clean FTI cases "automatic freeze exceptions". By "clean" I mean cases where the package was, practically speaking, useless before the fix. Cases where it's just one subpackage of a larger package that was FTI should still be manually checked, especially if the changes are larger than just a straight targeted fix to that subpackage (e.g. a version bump).
That sounds reasonable. But would we trust packagers to include this important information ("this is not a simple case...") in their FE proposal, or would we still manually check case-by-case?
I would expect the person processing the acceptance to check it. If the submitter provided the information they should verify it.
On Tue, Sep 05, 2023 at 08:21:18AM -0700, Adam Williamson wrote:
updates-testing is not enabled by default for the upgrade.
The upgrade process uses whatever repos are enabled *in the current configuration*. So in the "typical" case, you are upgrading from a stable Fedora release with default repo configuration, in which updates-testing is not enabled. Thus updates-testing is not used for the upgrade.
This is why we have the policy of accepting clean FTI fixes during Beta freeze.
Yeah, but... when we are 'go' for Beta, we unlock the stable pushes again, so by the time Beta is actually released, most of those packages are already stable and in the base repo, no?
Of course that doesn't help testers, but they likely know how to enable updates-testing?
It seems like to me easier to just tell them 'make sure your update is stable before beta release day'.
kevin
On Tue, 2023-09-05 at 09:27 -0700, Kevin Fenzi wrote:
On Tue, Sep 05, 2023 at 08:21:18AM -0700, Adam Williamson wrote:
updates-testing is not enabled by default for the upgrade.
The upgrade process uses whatever repos are enabled *in the current configuration*. So in the "typical" case, you are upgrading from a stable Fedora release with default repo configuration, in which updates-testing is not enabled. Thus updates-testing is not used for the upgrade.
This is why we have the policy of accepting clean FTI fixes during Beta freeze.
Yeah, but... when we are 'go' for Beta, we unlock the stable pushes again, so by the time Beta is actually released, most of those packages are already stable and in the base repo, no?
Yes, the FE is intended to ease the process for folks upgrading before Beta is actually released, including folks who are upgrading to help us test the upgrade process.
-
Adam Williamson -
Frantisek Zatloukal -
Kamil Paral -
Kevin Fenzi