This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
* word editor (e.g. libreoffice-writer)
* spreadsheet editor (e.g. libreoffice-calc)
* video player (e.g. totem)
* help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral kparal@redhat.com wrote:
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
There are some alternative approaches to this, but I'm not convinced they are better:
*Alt#1: Lower the bar for everyone* In the past, we didn't have different rules for products based on their popularity. The actual testing coverage was probably different, because more popular products tend to receive more testing, but the rules were the same. If somebody feels that we should keep having exactly the same rules regardless of popularity (desktop environments, architectures, etc), the solution is to lower the bar for everyone. All desktop environments (including Workstation x86_64) would be subject to the same list of application types (as described in the proposal) that could block the release. Other apps wouldn't block.
*Alt#2: Keep the original release criterion, but make testing best effort* If we have a QA test case, we currently make sure it's fully tested before we vote GO for the release candidate. We don't need to do that. We can say that this test case is just best effort, and if it's not tested at all, it doesn't stop us from voting GO. But if anyone finds a severe issue, it can still be a blocker. We used this approach for optical media testing [1]. But in this particular case, I think it's not a good idea, and will lead to frustration. The desktop apps are too numerous, they are complex, and it's way too easy to find bugs in them. If we keep it just best effort, everything else gets the priority treatment, and the likely outcome will be that many bugs will be found (either by QA or the community) shortly before the final release date, and everyone will be angry that it wasn't found earlier. This area is very unlike the optical media example mentioned, which is very specific and quite stable in features and behavior.
[1] https://pagure.io/fesco/issue/2303
Do you consider one of these alternative approaches better than my original proposal? Or do you think there's a better way?
This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
I think is a good thought.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not.
My thought here is that, yes, we should include these in our testing and continue to block on these, as I see them as some of the quintessential programs that a basic user would install Fedora and expect to be able to use. In my experience, if a new Linux user has to turn to the terminal to get thing working or installed, they're less likely to continue on with Fedora. We do have the gnome-software app, but even still, out-of-the-box functionality for the above listed programs should be included and they should work.
That's my two-cents anyway.
Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s
I can understand this sentiment and would be okay not blocking certain things on ARM, because as you stated, it takes a bit of technical know-how to even get an ARM system running, so I think it's a safe assumption to make that an ARM user can deal with minor default-application issues. This is an over-generalization, but I think it holds true for a majority of cases.
usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_a...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
On Mon, Feb 24, 2020 at 7:19 PM Geoffrey Marr gmarr@redhat.com wrote:
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria,
because
we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not.
My thought here is that, yes, we should include these in our testing and continue to block on these, as I see them as some of the quintessential programs that a basic user would install Fedora and expect to be able to use. In my experience, if a new Linux user has to turn to the terminal to get thing working or installed, they're less likely to continue on with Fedora. We do have the gnome-software app, but even still, out-of-the-box functionality for the above listed programs should be included and they should work.
That's my two-cents anyway.
Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether
we
want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners.
It’s
I can understand this sentiment and would be okay not blocking certain things on ARM, because as you stated, it takes a bit of technical know-how to even get an ARM system running, so I think it's a safe assumption to make that an ARM user can deal with minor default-application issues. This is an over-generalization, but I think it holds true for a majority of cases.
I'm a bit confused here. You agree that some apps might be OK to be not blocking on ARM, but one paragraph above, you want those apps included in the release blocking list. And that list is relevant just to Workstation on ARM, or the KDE spin. Because on x86_64 Workstation, according to the proposal, we still block on all the apps. Can you please clarify? Thanks.
On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral kparal@redhat.com wrote:
This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw.
Thanks, Kamil Fedora QA
On Tue, Mar 3, 2020 at 2:44 PM Kamil Paral kparal@redhat.com wrote:
On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral kparal@redhat.com wrote:
This proposal intends to reduce the scope of the “*Default application functionality*” release criterion [0]:
All applications that can be launched using the standard graphical
mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test.
*= Background =*
The area which QA is responsible for has been growing and growing in recent years. We used to have just a single Fedora release with two blocking desktops (GNOME and KDE). Then Editions got introduced and we started testing Workstation, Server, Cloud and the KDE Spin. Additional architectures were introduced (fortunately i386 got obsolete) and we started testing and blocking on specific images on armhfp and aarch64. Currently there are 13 release blocking deliverables mentioned on the ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS became an official edition lately, and even though its release cycle is not tied to traditional Fedora release cycle (and that’s why it’s not mentioned on that wiki page), as an official edition QA should care about it as well. It is the same story with Fedora IoT, another recent release-blocking addition [2]. Desktop testing is one of the most time-consuming jobs with the most frequent bug occurrence. Right now, we’re supposed to be fully testing and blocking on GNOME, KDE and XFCE (although XFCE might get dropped in favor or aarch64 Workstation [3]). We cannot test all of this and honestly claim that we verified basic functionality of all the included apps on all these desktops. I believe we need to adjust the criterion and align it closer with reality. In my eyes, it’s better to have a narrower scope and perform it well than having a large scope and perform it poorly.
*= Proposal =*
Change the criterion to something along these lines:
All applications that can be launched using the standard graphical
mechanism after a default installation of Fedora Workstation on x86_64 architecture must start successfully and withstand a basic functionality test.
For other release-blocking desktops (on any architecture), the requirements only apply to the following types of applications:
web browser (e.g. firefox) [4]
file browser (e.g. nautilus)
package manager (e.g. gnome-software)
image viewer (e.g. eog)
document viewer (e.g. evince)
text editor (e.g. gedit)
archive manager (e.g. file-roller)
terminal emulator (e.g. gnome-terminal) [4]
problem reporter (e.g. abrt)
If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements.
As you can see, the original criterion was kept for Fedora’s flagship desktop edition, the one that is most prominent on https://getfedora.org and probably the one that most newcomers download. We would still verify everything on Fedora Workstation on x86_64. But any other desktop (including Workstation on aarch64) would get just reduced criteria, because we simply can’t ensure the same quality bar for the smaller desktop editions/spins. There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
I’d like to hear your thoughts on whether they should be included or not. Of course from an end-user point of view, it would be beneficial. But the question is whether we as QA can promise their testing. And also whether we want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using alternative desktops and architectures are usually far from beginners. It’s usually not difficult to install a different application. Also note that the apps included in x86_64 Workstation will be thoroughly tested so they should be very likely to work well even on other architectures (minus some arch-specific issues).
I’d like to target Fedora 32 with this proposal, which means we should decide on this proposal fast. (I wanted to propose it much sooner, but I was waiting on clarifications about new release-blocking images and also on some fesco tickets [3], some of which is still not clarified; so I’m sorry about a late proposal).
Please comment, thanks.
[0] https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_appl...
[1] https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
[2] https://pagure.io/fedora-iot/issue/23
[3] https://pagure.io/fesco/issue/2339
[4] These two are required to work even in Basic release criteria [5], but I decided to include them anyway to avoid confusion (“why is the web browser missing?”), and to make it clear that they need to satisfy the basic functionality (whereas for Basic release criteria just the barebone functionality might be deemed enough)
[5] https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw.
Hi Kamil,
I think this change makes sense. +1 from me as well.
Kalev
On Tue, Mar 3, 2020 at 2:43 PM Kamil Paral kparal@redhat.com wrote:
I haven't received much feedback on this proposal. On test list and kde list, there was a short follow-up regarding the list of release-blocking applications. On kde list, Kevin disagreed because it lowers the guaranteed quality bar for KDE, but he didn't continue discussing it with me. I haven't received any alternative proposals either. On desktop list, there was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally also provides some constructive ways how to optimize desktop testing, because we need to cut it down /somehow/), I'll put the change into effect. I'll also probably move the help viewer into the list of release-blocking applications, based on the discussion I saw.
I talked to Rex Dieter and Kevin Kofler on #fedora-kde IRC, and they raised a concern about about this sentence: "If there are multiple applications of the same type (e.g. several web browsers), only one of them needs to satisfy the requirements."
Their worry is that with this approach it is problematic to have a primary application for the majority of use cases, and then a secondary application for other use cases/different users, because if the secondary app works fine, the release will not be stopped even in case of serious issues in the primary app. Another potential problem is when another application is pulled into the distribution e.g. by a dependency, and even when it is not seen as a big issue, it can still make the original app be completely ignored in terms of release blocking. We might have found a way to avoid this by changing the rule this way:
""" If there are multiple applications of the same type (e.g. several web browsers), the primary/default one must satisfy the requirements. If the primary/default application can't be determined, only one of said applications must satisfy the requirements.
Note: Determining primary applications The usual way to determine a primary/default application is to look into system configuration where default applications are set (e.g. gnome-control-center -> Default Applications), or to launch the default application on an appropriate file type (e.g. double click on an .odt file in a file browser), or to look into a favorites menu section (e.g. KDE menu -> Favorites). """
I think the change is reasonable and I adjust my proposal to include it. If you don't like it, or like it, or think it should be changed somehow, please comment. Thanks!
On Wed, Mar 4, 2020 at 1:55 PM Kamil Paral kparal@redhat.com wrote:
""" If there are multiple applications of the same type (e.g. several web browsers), the primary/default one must satisfy the requirements. If the primary/default application can't be determined, only one of said applications must satisfy the requirements.
Note: Determining primary applications The usual way to determine a primary/default application is to look into system configuration where default applications are set (e.g. gnome-control-center -> Default Applications), or to launch the default application on an appropriate file type (e.g. double click on an .odt file in a file browser), or to look into a favorites menu section (e.g. KDE menu -> Favorites). """
I like it. This seems like a reasonable way to address their concerns. Another approach could be to have the maintaining SIG provide their preferred default and in cases where none is provided, at least one has to work. So maybe, e.g. Llama Spin doesn't care *which* web browser works, but Alpaca Spin says Firefox has to work even if another browser is functional.
This essentially lets us go with the original proposal as a default case instead of having to go through and figure it out (because it may change from release-to-release, so it has to be verified every cycle)
On Wed, Mar 4, 2020 at 9:28 PM Ben Cotton bcotton@fedoraproject.org wrote:
On Wed, Mar 4, 2020 at 1:55 PM Kamil Paral kparal@redhat.com wrote:
""" If there are multiple applications of the same type (e.g. several web
browsers), the primary/default one must satisfy the requirements. If the primary/default application can't be determined, only one of said applications must satisfy the requirements.
Note: Determining primary applications The usual way to determine a primary/default application is to look into
system configuration where default applications are set (e.g. gnome-control-center -> Default Applications), or to launch the default application on an appropriate file type (e.g. double click on an .odt file in a file browser), or to look into a favorites menu section (e.g. KDE menu -> Favorites).
"""
I like it. This seems like a reasonable way to address their concerns. Another approach could be to have the maintaining SIG provide their preferred default and in cases where none is provided, at least one has to work. So maybe, e.g. Llama Spin doesn't care *which* web browser works, but Alpaca Spin says Firefox has to work even if another browser is functional.
This essentially lets us go with the original proposal as a default case instead of having to go through and figure it out (because it may change from release-to-release, so it has to be verified every cycle)
Yes, that's also an option. However, I intentionally tried to stay away from hard-coding concrete application names in release criteria (doesn't matter who maintains them, whether QA or some SIG). There's a maintenance cost I'd like to avoid. I'm afraid that people will forget to update the criteria when there are changes in the desktop apps. Also, if the list grows and people can't remember it, it might actually be easier to figure out the default application from the desktop environment than open the criteria, find the relevant section, and learn the application names. So I'd rather go with the generic solution, and if it doesn't work well, we can always change it to what you said or something similar.
It has been 7 days since the last response to the thread and I received no further complaints, so I've put the criterion into effect: https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
I've done slight tweaks compared to the original proposal: * Help viewer has been included in the covered application types. * Primary/default application is the one that must satisfy requirements in case of multiple same-type applications present (per the proposal by KDE folks mentioned in this thread). * "only one of said applications must satisfy the requirements" has been changed to "at least one of said applications...", which has the same meaning but is clearer (I hope) * "file browser" was renamed to "file manager", which seems to be the more official term (when looking at wikipedia) * application type examples were put into a footnote, to make the criterion itself more readable (and avoid some potential confusion when people see exact application names in there and miss the "e.g.")
I'll work on adjusting associated test cases, to put them in line with the new criterion.
On Thu, Mar 12, 2020 at 10:59 AM Kamil Paral kparal@redhat.com wrote:
It has been 7 days since the last response to the thread and I received no further complaints, so I've put the criterion into effect:
https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
I've done slight tweaks compared to the original proposal:
- Help viewer has been included in the covered application types.
- Primary/default application is the one that must satisfy requirements in
case of multiple same-type applications present (per the proposal by KDE folks mentioned in this thread).
- "only one of said applications must satisfy the requirements" has been
changed to "at least one of said applications...", which has the same meaning but is clearer (I hope)
- "file browser" was renamed to "file manager", which seems to be the more
official term (when looking at wikipedia)
- application type examples were put into a footnote, to make the
criterion itself more readable (and avoid some potential confusion when people see exact application names in there and miss the "e.g.")
I'll work on adjusting associated test cases, to put them in line with the new criterion.
(Replying just to test list now, no longer across lists)
The adjusted criterion seems to affect only this test case: https://fedoraproject.org/wiki/QA:Testcase_desktop_menus
My current plan is to redo it somewhat. I want to:
1. Drop all instructions to run applications from Testcase_desktop_menus and make it Optional. If will once again reflect its title and only care about menus and icons in menus.
2. Create the following new test cases: Testcase_desktop_app_web_browser Testcase_desktop_app_file_manager Testcase_desktop_app_package_manager Testcase_desktop_app_image_viewer Testcase_desktop_app_document_viewer Testcase_desktop_app_text_editor Testcase_desktop_app_archive_manager Testcase_desktop_app_terminal_emulator Testcase_desktop_app_problem_reporter Testcase_desktop_app_help_viewer Each of these will instruct people to start and test basic functionality of an application of that type, provided by the desktop. It will also explain what to do when there are multiple such applications. These test cases will be mandatory for all desktops.
3. Create the following new test case: Testcase_desktop_app_others This will include the same instructions as in 2), but it will instruct people to test all applications not covered by specific test cases in the same matrix. This test case will be mandatory for Workstation x86_64 and Optional for others.
This will have the benefit of multiple people being able to easily collaborate on desktop testing. It will definitely be less off-putting than a single "do it all" test case. It allows people to participate in desktop testing even if they have limited time. And it allows us to write OpenQA tests for certain applications and fill out the results on the wiki.
Note: In point 2), we don't actually need 10 different test cases (templated or not). We might just create Testcase_desktop_app_single instead of those 10 and differentiate them on the matrix level the same way as we do with e.g. "Testcase_Anaconda_save_traceback_to_bugzilla" [1]. So like this: Testcase_desktop_app_single (web browser) Testcase_desktop_app_single (file manager) ... I don't know if it complicates OpenQA results submission, though.
Please give me your thoughts, thanks. Kamil
[1] https://fedoraproject.org/wiki/Test_Results:Fedora_32_Beta_1.2_Installation#...
On Thu, Mar 12, 2020 at 2:58 PM Kamil Paral kparal@redhat.com wrote:
Note: In point 2), we don't actually need 10 different test cases (templated or not). We might just create Testcase_desktop_app_single instead of those 10 and differentiate them on the matrix level the same way as we do with e.g. "Testcase_Anaconda_save_traceback_to_bugzilla" [1]. So like this: Testcase_desktop_app_single (web browser) Testcase_desktop_app_single (file manager) ... I don't know if it complicates OpenQA results submission, though.
We actually do it even in the "Default boot and install" matrix [2], all those rows use the same testcase "Testcase_Boot_default_install", just the hyperlink is named differently.
[2] https://fedoraproject.org/wiki/Test_Results:Fedora_32_Beta_1.2_Installation#...
On Thu, 2020-03-12 at 14:58 +0100, Kamil Paral wrote:
On Thu, Mar 12, 2020 at 10:59 AM Kamil Paral kparal@redhat.com wrote:
It has been 7 days since the last response to the thread and I received no further complaints, so I've put the criterion into effect:
https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
I've done slight tweaks compared to the original proposal:
- Help viewer has been included in the covered application types.
- Primary/default application is the one that must satisfy requirements in
case of multiple same-type applications present (per the proposal by KDE folks mentioned in this thread).
- "only one of said applications must satisfy the requirements" has been
changed to "at least one of said applications...", which has the same meaning but is clearer (I hope)
- "file browser" was renamed to "file manager", which seems to be the more
official term (when looking at wikipedia)
- application type examples were put into a footnote, to make the
criterion itself more readable (and avoid some potential confusion when people see exact application names in there and miss the "e.g.")
I'll work on adjusting associated test cases, to put them in line with the new criterion.
(Replying just to test list now, no longer across lists)
The adjusted criterion seems to affect only this test case: https://fedoraproject.org/wiki/QA:Testcase_desktop_menus
My current plan is to redo it somewhat. I want to:
- Drop all instructions to run applications from Testcase_desktop_menus
and make it Optional. If will once again reflect its title and only care about menus and icons in menus.
- Create the following new test cases:
Testcase_desktop_app_web_browser Testcase_desktop_app_file_manager Testcase_desktop_app_package_manager Testcase_desktop_app_image_viewer Testcase_desktop_app_document_viewer Testcase_desktop_app_text_editor Testcase_desktop_app_archive_manager Testcase_desktop_app_terminal_emulator Testcase_desktop_app_problem_reporter Testcase_desktop_app_help_viewer Each of these will instruct people to start and test basic functionality of an application of that type, provided by the desktop. It will also explain what to do when there are multiple such applications. These test cases will be mandatory for all desktops.
- Create the following new test case:
Testcase_desktop_app_others This will include the same instructions as in 2), but it will instruct people to test all applications not covered by specific test cases in the same matrix. This test case will be mandatory for Workstation x86_64 and Optional for others.
This will have the benefit of multiple people being able to easily collaborate on desktop testing. It will definitely be less off-putting than a single "do it all" test case. It allows people to participate in desktop testing even if they have limited time. And it allows us to write OpenQA tests for certain applications and fill out the results on the wiki.
Note: In point 2), we don't actually need 10 different test cases (templated or not). We might just create Testcase_desktop_app_single instead of those 10 and differentiate them on the matrix level the same way as we do with e.g. "Testcase_Anaconda_save_traceback_to_bugzilla" [1]. So like this: Testcase_desktop_app_single (web browser) Testcase_desktop_app_single (file manager) ... I don't know if it complicates OpenQA results submission, though.
Please give me your thoughts, thanks. Kamil
Plan seems broadly fine to me. python-wikitcms can handle both possible approaches here, this is why it has the concept of the 'test name' separate from the 'test case'.
On Thu, Mar 12, 2020 at 2:58 PM Kamil Paral kparal@redhat.com wrote:
The adjusted criterion seems to affect only this test case: https://fedoraproject.org/wiki/QA:Testcase_desktop_menus
My current plan is to redo it somewhat. I want to:
- Drop all instructions to run applications from Testcase_desktop_menus
and make it Optional. If will once again reflect its title and only care about menus and icons in menus.
- Create the following new test cases:
Testcase_desktop_app_web_browser Testcase_desktop_app_file_manager Testcase_desktop_app_package_manager Testcase_desktop_app_image_viewer Testcase_desktop_app_document_viewer Testcase_desktop_app_text_editor Testcase_desktop_app_archive_manager Testcase_desktop_app_terminal_emulator Testcase_desktop_app_problem_reporter Testcase_desktop_app_help_viewer Each of these will instruct people to start and test basic functionality of an application of that type, provided by the desktop. It will also explain what to do when there are multiple such applications. These test cases will be mandatory for all desktops.
- Create the following new test case:
Testcase_desktop_app_others This will include the same instructions as in 2), but it will instruct people to test all applications not covered by specific test cases in the same matrix. This test case will be mandatory for Workstation x86_64 and Optional for others.
This will have the benefit of multiple people being able to easily collaborate on desktop testing. It will definitely be less off-putting than a single "do it all" test case. It allows people to participate in desktop testing even if they have limited time. And it allows us to write OpenQA tests for certain applications and fill out the results on the wiki.
Note: In point 2), we don't actually need 10 different test cases (templated or not). We might just create Testcase_desktop_app_single instead of those 10 and differentiate them on the matrix level the same way as we do with e.g. "Testcase_Anaconda_save_traceback_to_bugzilla" [1]. So like this: Testcase_desktop_app_single (web browser) Testcase_desktop_app_single (file manager) ... I don't know if it complicates OpenQA results submission, though.
The changes are now live.
The Desktop matrix changed a lot: https://fedoraproject.org/w/index.php?title=Template%3ADesktop_test_matrix&a...
I've created these two new test cases: https://fedoraproject.org/wiki/QA:Testcase_desktop_app_basic https://fedoraproject.org/wiki/QA:Testcase_desktop_app_basic_others
I found out that we already had a specific test case for terminal and web browser. The terminal test case didn't contain anything extra, so I replaced it with the generic desktop_app_basic test case, and removed the associated criterion: https://fedoraproject.org/w/index.php?title=QA%3ATestcase_desktop_terminal&a...
The browser test case contains specific browser requirements, so I left it intact.
The desktop_menus testcase now only tests menus (again): https://fedoraproject.org/w/index.php?title=QA%3ATestcase_desktop_menus&...
And I updated criteria associations on release criteria pages: https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria... https://fedoraproject.org/w/index.php?title=Basic_Release_Criteria&type=...
I hope I didn't make any errors during the process, that was a lot of lines to change.
OpenQA admins, please note that I adjusted how Testcase_Printing_New_Printer is displayed in the matrix. I wanted to make it look the same as the other rows I added. If that breaks results submission, I'm sorry, can you adjust it please? Thanks!
On Thu, Mar 12, 2020 at 10:59 AM Kamil Paral kparal@redhat.com wrote:
It has been 7 days since the last response to the thread and I received no further complaints, so I've put the criterion into effect:
https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
It turns out I've not included "system settings" app, which was agreed to be important when discussing this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1815463
So, here's a proposal to include: * system settings in the list of applications we block on. A particular example would be gnome-control-center.
Please voice your opinion. This should be fairly non-controversial, so unless somebody finds a good reason to omit it or phrase it differently, I'll make the change live in a day or two.
On Mon, Mar 23, 2020 at 8:04 PM Kamil Paral kparal@redhat.com wrote:
On Thu, Mar 12, 2020 at 10:59 AM Kamil Paral kparal@redhat.com wrote:
It has been 7 days since the last response to the thread and I received no further complaints, so I've put the criterion into effect:
https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
It turns out I've not included "system settings" app, which was agreed to be important when discussing this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1815463
So, here's a proposal to include:
- system settings
in the list of applications we block on. A particular example would be gnome-control-center.
Please voice your opinion. This should be fairly non-controversial, so unless somebody finds a good reason to omit it or phrase it differently, I'll make the change live in a day or two.
The change is now live: https://fedoraproject.org/w/index.php?title=Fedora_32_Final_Release_Criteria...
On Mon, Feb 17, 2020 at 9:31 AM Kamil Paral kparal@redhat.com wrote:
There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
Sorry for the late reply -- it's been a busy month. I'd just like to add that I'm +1 for to the proposal, with the exception of excluding the help viewer (e.g. yelp). I think testing to make sure the help browser works is vital.
-Jared
On Thu, Mar 12, 2020 at 3:01 PM Jared K. Smith jsmith@fedoraproject.org wrote:
On Mon, Feb 17, 2020 at 9:31 AM Kamil Paral kparal@redhat.com wrote:
There are some high-profile types of applications that I considered including in the list above, but didn’t in the end:
word editor (e.g. libreoffice-writer)
spreadsheet editor (e.g. libreoffice-calc)
video player (e.g. totem)
help viewer (e.g. yelp)
Sorry for the late reply -- it's been a busy month. I'd just like to add that I'm +1 for to the proposal, with the exception of excluding the help viewer (e.g. yelp). I think testing to make sure the help browser works is vital.
So you might be glad to hear that a help viewer was added to the blocking set :-) Thanks for your feedback.
-
Adam Williamson -
Ben Cotton -
Geoffrey Marr -
Jared K. Smith -
Kalev Lember -
Kamil Paral