Hi folks! So as we noticed today, we have no test case / criterion to ensure the anaconda help actually works. (Note it was only introduced in Fedora 21 or 22). I have written a test case:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_help
and I'm proposing we add a Final criterion:
* Any element in the installer interface(s) which is clearly intended to display 'help' text must do so correctly when activated.
We could also require that the intended 'Help' elements must be present, but that's a slightly tighter requirement - I'd ask for the anaconda team's opinion on whether we should do that or not. If we don't do that, then if by some chance the Help buttons were entirely *absent*, that would not be a blocker - only if buttons are present but don't work would it be a blocker.
I'm proposing we add the criterion, and add the test case to the 'Miscellaneous' table of the Installation matrix.
On Thu, Oct 29, 2015 at 7:36 PM, Adam Williamson <adamwill@fedoraproject.org
wrote:
Hi folks! So as we noticed today, we have no test case / criterion to ensure the anaconda help actually works. (Note it was only introduced in Fedora 21 or 22). I have written a test case:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_help
and I'm proposing we add a Final criterion:
- Any element in the installer interface(s) which is clearly intended
to display 'help' text must do so correctly when activated.
We could also require that the intended 'Help' elements must be present, but that's a slightly tighter requirement - I'd ask for the anaconda team's opinion on whether we should do that or not. If we don't do that, then if by some chance the Help buttons were entirely *absent*, that would not be a blocker - only if buttons are present but don't work would it be a blocker.
I'm proposing we add the criterion, and add the test case to the 'Miscellaneous' table of the Installation matrix.
As we discussed this on Go/No-Go meeting: validity of this criteria will start with F24 release. There are no plans to use this criteria for F23. Right Adam ?
Regards, Jan
-- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
-- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
On Thu, 2015-10-29 at 20:11 +0100, Jan Kurik wrote:
On Thu, Oct 29, 2015 at 7:36 PM, Adam Williamson <adamwill@fedoraproj ect.org
wrote:
Hi folks! So as we noticed today, we have no test case / criterion to ensure the anaconda help actually works. (Note it was only introduced in Fedora 21 or 22). I have written a test case:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_help
and I'm proposing we add a Final criterion:
- Any element in the installer interface(s) which is clearly
intended to display 'help' text must do so correctly when activated.
We could also require that the intended 'Help' elements must be present, but that's a slightly tighter requirement - I'd ask for the anaconda team's opinion on whether we should do that or not. If we don't do that, then if by some chance the Help buttons were entirely *absent*, that would not be a blocker - only if buttons are present but don't work would it be a blocker.
I'm proposing we add the criterion, and add the test case to the 'Miscellaneous' table of the Installation matrix.
As we discussed this on Go/No-Go meeting: validity of this criteria will start with F24 release. There are no plans to use this criteria for F23. Right Adam ?
That was the idea, yeah, though it seems like we're currently scrambling to try and do a build with some kind of fix...
" Proceed through the installer for a while, clicking Help on each screen. "
On each screen available, or on each screen you go through? The first meaning could make for a wearisome test case. I'd rather avoid robot-like test cases (for humans, not when we automate it) and go with the second meaning, relying on fuzzy-testing (each person does that a bit differently) instead.
and I'm proposing we add a Final criterion:
- Any element in the installer interface(s) which is clearly intended
to display 'help' text must do so correctly when activated.
Sounds good to me.
PS: Originally I wanted to add that I'd also like to extend this to the installed system, not just installer. In the installed system, it would cover only the "main help entry", i.e. somewhere in main system menu. But then I realized we probably already have it covered by "default application functionality" criterion.
On Fri, 2015-10-30 at 08:04 -0400, Kamil Paral wrote:
" Proceed through the installer for a while, clicking Help on each screen. "
On each screen available, or on each screen you go through? The first meaning could make for a wearisome test case. I'd rather avoid robot- like test cases (for humans, not when we automate it) and go with the second meaning, relying on fuzzy-testing (each person does that a bit differently) instead.
I was kind of leaving it intentionally vague (hence the 'for a while'). Do it till you get bored. :) I'm fine if you want to propose a tweak, though.
and I'm proposing we add a Final criterion:
- Any element in the installer interface(s) which is clearly
intended to display 'help' text must do so correctly when activated.
Sounds good to me.
PS: Originally I wanted to add that I'd also like to extend this to the installed system, not just installer. In the installed system, it would cover only the "main help entry", i.e. somewhere in main system menu. But then I realized we probably already have it covered by "default application functionality" criterion.
Actually we intentionally took it out; we used to require that all apps installed by default have a working Help, but it was one of those polish requirements that seemed too onerous and no-one wanted to spend their time checking and we'd perennially just waive at Go/No-Go anyway, so we took it out. Some GNOME apps do not in fact have Help, e.g. Software.
On Fri, 2015-10-30 at 08:04 -0400, Kamil Paral wrote:
" Proceed through the installer for a while, clicking Help on each screen. "
On each screen available, or on each screen you go through? The first meaning could make for a wearisome test case. I'd rather avoid robot- like test cases (for humans, not when we automate it) and go with the second meaning, relying on fuzzy-testing (each person does that a bit differently) instead.
I was kind of leaving it intentionally vague (hence the 'for a while'). Do it till you get bored. :) I'm fine if you want to propose a tweak, though.
Oh, I missed/ignored the "for a while" part. OK, sounds reasonable. I'd still make it clearer ("on each screen you visit"), but I don't have problems with the vague version either.
and I'm proposing we add a Final criterion:
- Any element in the installer interface(s) which is clearly
intended to display 'help' text must do so correctly when activated.
Sounds good to me.
PS: Originally I wanted to add that I'd also like to extend this to the installed system, not just installer. In the installed system, it would cover only the "main help entry", i.e. somewhere in main system menu. But then I realized we probably already have it covered by "default application functionality" criterion.
Actually we intentionally took it out; we used to require that all apps installed by default have a working Help, but it was one of those polish requirements that seemed too onerous and no-one wanted to spend their time checking and we'd perennially just waive at Go/No-Go anyway, so we took it out. Some GNOME apps do not in fact have Help, e.g. Software.
Yeah, I really didn't want to extend this to every app's Help. Just the main system one, e.g. if you type "help" in gnome overview. I believe that one should be a blocker. But it should also be covered by "default app functionality" criterion, so no additions needed, I believe.
On Mon, 2015-11-02 at 09:48 -0500, Kamil Paral wrote:
Yeah, I really didn't want to extend this to every app's Help. Just the main system one, e.g. if you type "help" in gnome overview. I believe that one should be a blocker. But it should also be covered by "default app functionality" criterion, so no additions needed, I believe.
Yep, indeed, it shows up on the system menus so it counts for that criterion.
On Thu, 2015-10-29 at 11:36 -0700, Adam Williamson wrote:
Hi folks! So as we noticed today, we have no test case / criterion to ensure the anaconda help actually works. (Note it was only introduced in Fedora 21 or 22). I have written a test case:
https://fedoraproject.org/wiki/QA:Testcase_Anaconda_help
and I'm proposing we add a Final criterion:
- Any element in the installer interface(s) which is clearly intended
to display 'help' text must do so correctly when activated.
We could also require that the intended 'Help' elements must be present, but that's a slightly tighter requirement - I'd ask for the anaconda team's opinion on whether we should do that or not. If we don't do that, then if by some chance the Help buttons were entirely *absent*, that would not be a blocker - only if buttons are present but don't work would it be a blocker.
I'm proposing we add the criterion, and add the test case to the 'Miscellaneous' table of the Installation matrix.
As everyone seemed broadly in favour of this I'm putting it into practice now, with a couple of tweaks - kparal's 'every screen you visit' suggestion and a tweak to the test case wording which sgallagh suggested on IRC. Thanks folks!
We'll try and get this into openQA, but it's actually slightly trickier than you'd think, just because of boring details about how openQA works.
-
Adam Williamson -
Jan Kurik -
Kamil Paral