Proposal: Split up QA:Testcase_desktop_updates into three separate test cases

[Adam Williamson]

Hey folks!

So several people have commented in the past that 
https://fedoraproject.org/wiki/QA:Testcase_desktop_updates is kind of 
a mess, and I think that's correct. I think we were trying to make
things 'efficient' at the time by exercising several different things
in one test, but it really does result in a messy, hard-to-comprehend
test case.

So, I propose we split it into three separate test cases, and I've
drafted them up:

https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_base_upd
ate_cli
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desktop_
update_graphical
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desktop_
update_notification

The first would be an Alpha test case in the Base (not desktop)
results page, and just covers installing updates with yum/dnf. The
second would be a Beta test case in the Desktop page, and covers
installing updates from a graphical desktop. The third would be a
Final test case in the Desktop page, and covers notification of
available updates.

I'm also planning to draft some additional test cases covering package
installation and removal tomorrow - mainly for the test day we have
coming up, but it would probably not be a bad idea to have them as
release validation tests as well.

What do folks think? Does this look like an improvement? Thanks!

[Giulio E.]

Il giorno lun, 27/04/2015 alle 18.09 -0700, Adam Williamson ha scritto:
> Hey folks!
>
> So several people have commented in the past that 
> https://fedoraproject.org/wiki/QA:Testcase_desktop_updates is kind 
> of 
> a mess, and I think that's correct. I think we were trying to make
> things 'efficient' at the time by exercising several different things
> in one test, but it really does result in a messy, hard-to-comprehend
> test case.
>
> So, I propose we split it into three separate test cases, and I've
> drafted them up:
>
> https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_base_u
> pd
> ate_cli
> https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_deskto
> p_
> update_graphical
> https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_deskto
> p_
> update_notification
>
> The first would be an Alpha test case in the Base (not desktop)
> results page, and just covers installing updates with yum/dnf. The
> second would be a Beta test case in the Desktop page, and covers
> installing updates from a graphical desktop. The third would be a
> Final test case in the Desktop page, and covers notification of
> available updates.
>
> I'm also planning to draft some additional test cases covering 
> package
> installation and removal tomorrow - mainly for the test day we have
> coming up, but it would probably not be a bad idea to have them as
> release validation tests as well.
> What do folks think? Does this look like an improvement? Thanks!
>
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . 
> net
> http://www.happyassassin.net

Hi Adam.

When we see a single 'fail' on the current matrix, we can not be sure
about what part of the process might be broken (dnf? apper?
notifications? kde's applet?).

So, in my opinion, the tests are much clearer as they appear in your
drafts. They could improve the readability of our results. Indeed, I
prefer little, 'monolithic' test-cases which are easier to follow and
to track down.

Have a nice day!

// Giulio (juliuxpigface)

[Adam Williamson]

On Tue, 2015-04-28 at 08:07 +0200, Giulio 'juliuxpigface' wrote:
> Hi Adam.
>
> When we see a single 'fail' on the current matrix, we can not be sure
> about what part of the process might be broken (dnf? apper?
> notifications? kde's applet?).

Yep - that's one of the main problems with the CLI and graphical
results being combined.
> So, in my opinion, the tests are much clearer as they appear in your
> drafts. They could improve the readability of our results. Indeed, I
> prefer little, 'monolithic' test-cases which are easier to follow and
> to track down.

Thanks for the feedback!

[Kamil Paral]

> So, I propose we split it into three separate test cases, and I've
> drafted them up:
>
> https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_base_upd
> ate_cli
> https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desktop_
> update_graphical

I find naming all the button names to be usually too over-specified. And it changes in time. I think more generic instructions are better, unless there is something really hard to discover (like the Refresh button in GNOME Software). But that's just a side note that occurred to me while reading the test case.
> https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desktop_
> update_notification
>
> The first would be an Alpha test case in the Base (not desktop)
> results page, and just covers installing updates with yum/dnf. The
> second would be a Beta test case in the Desktop page, and covers
> installing updates from a graphical desktop. The third would be a
> Final test case in the Desktop page, and covers notification of
> available updates.
>
> I'm also planning to draft some additional test cases covering package
> installation and removal tomorrow - mainly for the test day we have
> coming up, but it would probably not be a bad idea to have them as
> release validation tests as well.
>
> What do folks think? Does this look like an improvement? Thanks!

I like it, it's better to have it split this way. Thanks.

[Adam Williamson]

On Tue, 2015-04-28 at 04:20 -0400, Kamil Paral wrote:
> > So, I propose we split it into three separate test cases, and I've
> > drafted them up:
> >
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_base
> > _upd
> > ate_cli
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desk
> > top_
> > update_graphical
>
> I find naming all the button names to be usually too over-specified. 
> And it changes in time. I think more generic instructions are 
> better, unless there is something really hard to discover (like the 
> Refresh button in GNOME Software). But that's just a side note that 
> occurred to me while reading the test case.

Heh, now you're making my arguments to me :P I remember people always
used to criticise me for my test cases being too vague, and I'd say it
was to stop them going stale...

The idea was that the meat of the instructions are the numbered
points, and those were sort of bonuses to give precise details for the
most common cases. I can try to hedge it a bit more to make that clear.

[Adam Williamson]

On Mon, 2015-04-27 at 18:09 -0700, Adam Williamson wrote:
> Hey folks!
>
> So several people have commented in the past that 
> https://fedoraproject.org/wiki/QA:Testcase_desktop_updates is kind 
> of 
> a mess, and I think that's correct. I think we were trying to make
> things 'efficient' at the time by exercising several different things
> in one test, but it really does result in a messy, hard-to-comprehend
> test case.
>
> So, I propose we split it into three separate test cases, and I've
> drafted them up:

As the feedback was positive, I've gone ahead and implemented this. I
moved the test cases to production names:

https://fedoraproject.org/wiki/QA:Testcase_base_update_cli
https://fedoraproject.org/wiki/QA:Testcase_desktop_update_graphical
https://fedoraproject.org/wiki/QA:Testcase_desktop_update_notification

adjusted the test cases listed in the criteria pages, and updated the
Base and Desktop templates to use the separated cases. This will take
effect in the result pages from Final TC2 (Final TC1 still has the old
layout).

Thanks folks!