Now that we have system-upgrade instead of fedup, I don't think we need to split every single upgrade test case between bios and uefi, as we have it right now: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
The difference is that fedup created its own upgrade environment, a new boot menu, a special initrd, contained a lot of systemd-related hacks. System-upgrade uses the standard "offline upgrade" environment, so if offline upgrades ever worked for you, there should be no difference. It doesn't create its own boot menu, it doesn't have a separate initrd. It uses the standard way of booting your system - if your system boots, the upgrade process should boot as well :) Likewise for adding a new boot entries during the upgrade - if kernel updates work well for you, there should be no problem adding a new kernel entry during upgrade, it's the same operation.
So, I'd like to save some unnecessary work and merge "x86_64 BIOS" and "x86_64 UEFI" columns into "x86_64" column in that wiki matrix.
I'd also like to add two new test cases: QA:Testcase_upgrade_dnf_previous_bios QA:Testcase_upgrade_dnf_previous_uefi
This will only serve as a way to track that we have tested both environments, but it no longer matters which product was used to perform the testing, and also we no longer need to test all combinations. So, if you tested workstation on bios, you can fill out two test cases immediately (_workstation and _bios). If you tested minimal on uefi, the same applies.
I think it's valuable to keep bios vs uefi differentiation in the table at least in the form of these two additional test cases, because it's still a system upgrade, and there are some critical components which can break (like a new major release of efibootmgr or grub). We should make sure we've tested both variants. But this way, it adds virtually no extra work (compared to testing all combinations of it) and still gives us almost the same test value.
Thoughts?
Kamil
Now that we have system-upgrade instead of fedup, I don't think we need to split every single upgrade test case between bios and uefi, as we have it right now: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
The difference is that fedup created its own upgrade environment, a new boot menu, a special initrd, contained a lot of systemd-related hacks. System-upgrade uses the standard "offline upgrade" environment, so if offline upgrades ever worked for you, there should be no difference. It doesn't create its own boot menu, it doesn't have a separate initrd. It uses the standard way of booting your system - if your system boots, the upgrade process should boot as well :) Likewise for adding a new boot entries during the upgrade - if kernel updates work well for you, there should be no problem adding a new kernel entry during upgrade, it's the same operation.
So, I'd like to save some unnecessary work and merge "x86_64 BIOS" and "x86_64 UEFI" columns into "x86_64" column in that wiki matrix.
I'd also like to add two new test cases: QA:Testcase_upgrade_dnf_previous_bios QA:Testcase_upgrade_dnf_previous_uefi
This will only serve as a way to track that we have tested both environments, but it no longer matters which product was used to perform the testing, and also we no longer need to test all combinations. So, if you tested workstation on bios, you can fill out two test cases immediately (_workstation and _bios). If you tested minimal on uefi, the same applies.
I think it's valuable to keep bios vs uefi differentiation in the table at least in the form of these two additional test cases, because it's still a system upgrade, and there are some critical components which can break (like a new major release of efibootmgr or grub). We should make sure we've tested both variants. But this way, it adds virtually no extra work (compared to testing all combinations of it) and still gives us almost the same test value.
Oh, one more thing - similarly to _bios and _uefi test cases, I'd like to change existing _workstation_encrypted test case to just _encrypted test case - it shouldn't matter which package set is installed to verify that there's no issue with unlocking your drives. This way you can easily fill 3 test cases with just a single test (e.g. minimal + encrypted + bios).
On Mon, 2015-11-23 at 10:36 -0500, Kamil Paral wrote:
Oh, one more thing - similarly to _bios and _uefi test cases, I'd like to change existing _workstation_encrypted test case to just _encrypted test case - it shouldn't matter which package set is installed to verify that there's no issue with unlocking your drives. This way you can easily fill 3 test cases with just a single test (e.g. minimal + encrypted + bios).
Just be aware that the upgrade test cases are based on templates, please don't break that setup as it saves a lot of work in duplicating content between all the various upgrade tests. I understand why you want to make the changes in the way you do, but I do think it'd be nice if possible to always treat UEFI vs. BIOS as an 'environment' difference, i.e. in the result columns, not in the test case name.
As an alternative perhaps we could have workstation_encrypted (or 'any package set encrypted', if I can tweak the template a bit) with BIOS and UEFI columns, and other tests with just x86 and ARM columns?
On Mon, 2015-11-23 at 10:36 -0500, Kamil Paral wrote:
Oh, one more thing - similarly to _bios and _uefi test cases, I'd like to change existing _workstation_encrypted test case to just _encrypted test case - it shouldn't matter which package set is installed to verify that there's no issue with unlocking your drives. This way you can easily fill 3 test cases with just a single test (e.g. minimal + encrypted + bios).
Just be aware that the upgrade test cases are based on templates, please don't break that setup as it saves a lot of work in duplicating content between all the various upgrade tests. I understand why you want to make the changes in the way you do, but I do think it'd be nice if possible to always treat UEFI vs. BIOS as an 'environment' difference, i.e. in the result columns, not in the test case name.
As an alternative perhaps we could have workstation_encrypted (or 'any package set encrypted', if I can tweak the template a bit) with BIOS and UEFI columns, and other tests with just x86 and ARM columns?
Hmm, the environments as columns work well if we want to test all combinations, but not that well if we want to test it just once, for any test case. I'd really like to say "if you have tested upgrade with uefi, whatever package set or configuration, you can tick this box", and not "tick this box only if you have tested _encrypted_ installation with uefi". That puts additional constraints on testing, and it seems to me that it's better to live with two additional lines (_uefi and _bios) rather than changing our testing to accommodate a nicer 2D structure of the matrix.
As a middle ground, what about having: .---------------------------------------------------. | BIOS UEFI| |Testcase_upgrade_dnf_previous_any | '---------------------------------------------------'
I think that's reasonable? Maybe we don't even need to modify and use the template content, just reference other test cases and say that any of them is fine.
On Tue, 2015-11-24 at 06:33 -0500, Kamil Paral wrote:
As a middle ground, what about having: .---------------------------------------------------. | BIOS UEFI| |Testcase_upgrade_dnf_previous_any | '---------------------------------------------------'
I think that's reasonable? Maybe we don't even need to modify and use the template content, just reference other test cases and say that any of them is fine.
Yeah, that would be OK, I think. Of course in a sane TCMS we'd have a better way of saying 'out of these tests, at least one needs to be done for each environment' and calculating coverage and such, but this is Wikitcms ;)
On Tue, 2015-11-24 at 06:33 -0500, Kamil Paral wrote:
As a middle ground, what about having: .---------------------------------------------------. | BIOS UEFI| |Testcase_upgrade_dnf_previous_any | '---------------------------------------------------'
I think that's reasonable? Maybe we don't even need to modify and use the template content, just reference other test cases and say that any of them is fine.
Yeah, that would be OK, I think. Of course in a sane TCMS we'd have a better way of saying 'out of these tests, at least one needs to be done for each environment' and calculating coverage and such, but this is Wikitcms ;)
I have made the changes: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
This is the new test case: https://fedoraproject.org/wiki/QA:Testcase_upgrade_dnf_previous_any
Originally I wanted to use something like this in the test case: {{Testcase upgrade|(any flavor)}}
However, that didn't work and I could wrap my mind around all the templates, so I ended up with that bare-bones description. I hope it's fine, patches welcome :)
On Thu, 2015-11-26 at 10:23 -0500, Kamil Paral wrote:
On Tue, 2015-11-24 at 06:33 -0500, Kamil Paral wrote:
As a middle ground, what about having: .---------------------------------------------------. | BIOS UEFI| |Testcase_upgrade_dnf_previous_any | '---------------------------------------------------'
I think that's reasonable? Maybe we don't even need to modify and use the template content, just reference other test cases and say that any of them is fine.
Yeah, that would be OK, I think. Of course in a sane TCMS we'd have a better way of saying 'out of these tests, at least one needs to be done for each environment' and calculating coverage and such, but this is Wikitcms ;)
I have made the changes: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
This is the new test case: https://fedoraproject.org/wiki/QA:Testcase_upgrade_dnf_previous_any
Originally I wanted to use something like this in the test case: {{Testcase upgrade|(any flavor)}}
However, that didn't work and I could wrap my mind around all the templates, so I ended up with that bare-bones description. I hope it's fine, patches welcome :)
I like it the way it is anyway, it's clearer that it's just a 'synthetic' entry in the table. Thanks.
We will need to adjust openqa_fedora_tools for this, I think - I'll check that.