[Fedora QA] #408: Fedora 20 Translation Test Day
by fedora-badges
#408: Fedora 20 Translation Test Day
----------------------+------------------------
Reporter: aalam | Owner:
Type: task | Status: new
Priority: major | Milestone: Fedora 20
Component: Test Day | Version:
Keywords: | Blocked By:
Blocking: |
----------------------+------------------------
= phenomenon =
Translation Test day for Fedora 20.
Test Date: need to decide. It is before 'Software Translation Freeze'
(2013-10-08).
proposed date: Thursday Sep 26 (2013-09-26)
= implementation recommendation =
- need to review test cases for gnome-3.10
- need to create test for Anaconda
fltg team will help for that.
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/408>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
9 years, 10 months
slow mirror workaround?
by Felix Miata
Trying to yum upgrade 19 is stuck on a mirror with no useful throughput. What
kind of workaround for this is available? Nothing jumps at me in the yum man
page. How do I specify to use a particular mirror know to work?
--
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata *** http://fm.no-ip.com/
10 years, 2 months
Very rough storage validation matrix draft
by Adam Williamson
Hi, folks. So, our current set of storage validation tests is just a
grab-bag of stuff we've held over from oldui, added and patched
piecemeal; it's not very coherent or consistent and it doesn't come
close to exercising all of the storage stuff in the installer. We wind
up sort of inventing test plans particularly for custom partitioning as
we go along, with the consequence that we're not sure what we're going
to test, what's really important to test when, etc etc.
I've made a few abortive tries at re-doing the storage tests and
basically given up because it's just a hideous thing to try and cover,
but I thought while I'm still on a momentum roll from F20 and remember
some of the issues that came up during F20 validation, I'd take another
cut at it.
Here's what I came up with:
https://fedoraproject.org/wiki/User:Adamwill/Draft_storage_matrix
The proposal is to separate out storage testing from the 'installation
testing' matrix as its own matrix, because I think we can get further
with flexible columns, and it's such a big area the installation matrix
gets pretty unwieldy with all the storage tests stuffed into it.
remember this is *extremely* rough. It may be that we don't think my
organizational approach here is right at all and we should tear it up
and start over. But I thought I'd throw it out there for comments. Most
of the tests, obviously, don't exist yet; they'd be fairly trivial to
write, I think the hard part of the problem here is deciding what we
want to test, and how to organize it.
It's an extremely difficult area; there are so many different variables
to storage configuration that it's utterly impractical to try and cover
every possible permutation even assuming the user uses the interface
perfectly. What I've tried to do is come up with something which would
exercise most of the areas we really seem to want to care about, without
being utterly unwieldy.
It is still very large, that's probably the first thing you'd notice
about it. I doubt we could run this on every build. What I think would
be a nice complement is a somewhat improved version of testcase_stats -
http://testdays.qa.fedoraproject.org/testcase_stats/ - which could track
both dimensions of each matrix, so we could try to strategically ensure
every spot on the table is covered at least once per milestone or once
per release, say.
Here's a quick key to the test names for tests I've made up which may
not be self-explanatory, yell if you need explanations of any of the
others:
-----
guided_multi_empty - the 'multi' means multiple disks, this is checking
we correctly autopart to multiple disks.
in the custom tests, 'auto' means 'using the autopart mechanism inside
custom partitioning', the blue text that lets you automatically create a
set of volumes.
existing_retain_home - this is the classic 'install over an existing
Fedora install, retain the /home partition' trick.
existing_precreated - this would be a test for running the installer
with a set of empty volumes that you just wanted to assign mount points
to.
add_to_container - add a new volume to free space within an existing
container.
assign_* tests - these are for specifying that a given partition or
container must be on a particular disk.
help_text - just checks the 'help' screen works.
small_disks - this would be a test where you check anaconda correctly
refuses to install to a disk that's too small.
cancel_encryption - this would be to check what happens if you cancel
out of the 'enter a passphrase' dialog.
shrink_maximum - try to shrink a partition to the largest possible size
(right-hand end of the slider)
shrink_minimum - shrink a partition as small as possible (left-hand end
of the slider)
shrink_no_size_change - set action for a partition to 'shrink' but don't
actually change its size
shrink_unusual_sizes - this would be for the bug we discovered in f20,
test the shrinker handles partitions with 'weird' sizes
shrink_change_action - this is just for changing the 'action' in shrink
a bunch of times and making sure it doesn't explode
multiple_trips - run through Installation Destination more than once
custom_resize_no_size - just clear the 'desired size' field for a
partition and hit 'update settings'
custom_resize_invalid_unit - try entering something like '30 ZX' in the
desired size field
custom_resize_return - change the desired size and then change it back
to the original value
custom_invalid_mount_point - try entering a mount point name that is not
allowed (invalid characters, spaces or something)
custom_invalid_filesystem - try putting a system partition on vfat or
bios boot or swap or something
----
Please, absolutely all comments, suggestions, alternative proposals,
flames etc etc welcome. I'm sure we could improve this proposal or do
better, but I think we have to try and do _something_ better than our
current tests. And I think drawing up a table like this, if nothing
else, illustrates what a hard task this is...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10 years, 2 months
Draft terminal application test case
by Adam Williamson
Proposing a new draft release validation test case:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desktop_te...
this is a fairly simple test case that just checks that a terminal
application works in a desktop environment. It would be added to the
desktop matrix along with the browser test case, to cover the lack of a
test case to enforce the terminal part of the Alpha criterion "It must
be possible to run the default web browser and a terminal application
from all release-blocking desktop environments."
Comments, improvements, suggestions etc welcome as always! As this is
pretty straightforward I'm planning to put it into production pretty
soon if there's no negative feedback.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
10 years, 3 months
Fedora Easy Karma Recommendation
by Flash
For me, step 2 in Fedora Easy Karma instructions which reads "Run
fedora-cert", wasn't immediately clear without further explanation or an
example .i.e. fedora-cert --username=flashl.
As an aside, this observation may be just be operator error, but,
although FAS account already exist with credentials, fedora-cert
rejected login credentials and wasn't happy until I used -n flag to
recreate new certificate.
Flashl
10 years, 5 months
Proposal: let's just use the FAS group already
by Adam Williamson
Hi, folks. So this one keeps popping up for individual people, and we
keep doing a quick band-aid when anyone needs group membership...but we
may as well just bite the bullet and do it properly.
Fedora QA has always been pretty informally organized, we've generally
not wanted to build a big heavy organizational structure with
hierarchies and bits of paper to sign and stuff. One facet of this is
that we don't really have an 'official' Fedora QA FAS group. There is a
'qa' group in FAS, but it has very few members, and we haven't ever
worked on the assumption that the people in it are the 'real' QA people
or anything.
For our purposes we don't really have any incredible need for a QA group
in FAS, but there are some things in Fedora which require you to be a
member of a FAS group besides cla_signed - voting in some elections, for
instance, and getting a fedorapeople.org space. It seems unfortunate
that QA contributors don't get these things unless they get themselves
added to another group.
So I'm proposing we do something simple: let's just go ahead and stick
everyone who can reasonably be considered a 'QA team member' in the FAS
'qa' group. This wouldn't be hard to do, I can make sure sufficient
people within and outside RH have moderator status in the qa group, and
then those of us who are mods can just add people appropriately. Anyone
who files karma regularly, or validation test results, or posts to
test@, or attends QA team meetings, anything like that - let's just
stick 'em in QA group, and then for the future we can just say
'moderators can give anyone who's obviously a QA person membership in
the QA group at any time, and when someone sends a self-introduction
mail and doesn't completely disappear, the QA group mods should stick
that person in the group'.
How does that sound? Seems like something we can just get done already.
Right now me, James Laska, Will Woods and Jesse Keating are the admins
of the QA group. This is obviously a bit silly. I'll drop jlaska's,
wwoods' and jesses' admin roles, and make some more appropriate people
admins and moderators instead. I'll do that right now, since it's
clearly the right thing to do...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10 years, 5 months
Fedora 20 release day FedUp bug: post-mortem
by Adam Williamson
Hi, folks. Now things have calmed down a bit in Fedora 20 and Rawhide, I
have time to write this mail!
Many of you may already know that there was a significant issue with
upgrades to Fedora 20 around release day - 2013-12-17.
Summary of the issue
--------------------
Upgrading to Fedora 20 using version 0.7 of the FedUp tool does not
work. Upgrading with version 0.8 works (in the main - of course there
are bugs, there are always bugs).
At the time Fedora 20 was released, version 0.7 of FedUp was present in
the Fedora 18 and Fedora 19 'updates' repositories. Version 0.8 of FedUp
was present in 'updates-testing' for both Fedora 18 and Fedora 19 at the
time.
Immediate response to the issue
-------------------------------
We realized quite quickly during the course of release day support that
this was the case, though at first we thought perhaps only some upgrades
were failing. Once it became clear that all 0.7-based upgrades would
fail, several folks worked hard at communicating this to as many users
in as many places as possible, including via IRC, mailing lists, the
Common Bugs page
(https://fedoraproject.org/wiki/Common_F20_bugs#fedup-07-fail ), the
forums, and social network sites like G+. We advised using fedup 0.8
from updates-testing to upgrade.
We rapidly ensured 0.8 was submitted for stable push for both F18 and
F19. It was submitted for F19 at 2013-12-17 21:12:18 (I believe Bodhi
timestamps are UTC, so that was mid-afternoon on release day in NA) and
for F18 at 2013-12-18 11:51:47 (early morning on the day after release).
However, release engineering complications (there were some problems
with stable pushes at the time) meant it wasn't finally pushed until
2013-12-19 07:23:09 UTC for F19 (late on the day after release NA time)
and 2013-12-19 14:05:50 UTC for F18 (early morning two days after
release) and wouldn't have made it to most mirrors until 2013-12-19, two
days after release, and probably 2013-12-20 in 'early' timezones in
Europe and Asia.
Proximate cause of the issue
----------------------------
We have not yet identified the direct (proximate) cause of the bug;
doing so did not seem especially important in comparison to ensuring
news of the issue was spread as widely as possible, ensuring 0.8 was
sent stable as soon as possible, and resolving some related issues (see
later). However, QA's current inference is that there is some
incompatibility between how fedup 0.7 modifies the initramfs used by the
upgrade process and/or how it configures the upgrade boot environment,
and the expectations of the upgrade environment as it exists within the
final shipped upgrade initramfs. The upgrade initramfs is generated as
part of the release compose process, and is dependent on factors
including the versions of dracut and fedup-dracut used to build it.
Broadly, we suspect that an upgrade run with fedup 0.7 which uses an
upgrade initramfs generated with fedup-dracut 0.8 will not work, for
reasons not yet identified.
Indirect causes of the issue
----------------------------
We could perhaps make a very broad characterization of the 'indirect
causes' of the issue as follows: an upgrade using fedup depends on
several moving parts, and neither our development nor testing processes
are sufficiently robust to ensure that we cover all possible
combinations of those parts.
fedup / fedup-dracut interdependencies
++++++++++++++++++++++++++++++++++++++
So far as I can discern, there is not at present any policy (whether
written or enforced by some kind of mechanism) with regard to the
inter-dependencies between the 'fedup' package side of the fedup process
and the 'fedup-dracut' side of the process which involves release
engineering generating an upgrade initramfs via fedup-dracut. As this
issue suggests that not all 'fedups' work with all 'fedup-dracuts',
perhaps this is something that might be required, but we leave that to
the superior knowledge and expertise of the FedUp maintainer.
Test procedure inadequacies
+++++++++++++++++++++++++++
Similarly, QA's upgrade testing process clearly did not sufficiently
carefully consider the same issue. This is something we have now moved
to address.
Prior to Fedora 20's release, the test cases for fedup recommended
testing the latest version of fedup from updates-testing against the
upgrade initramfs from the development/20 tree. This procedure was a
holdover from the very early days of FedUp, when it was changing daily
and testing anything older was uninteresting, and when procedures for
the generation and publishing of the upgrade initramfs had not yet been
clearly established (and TC/RC trees did not contain one). However, it
is no longer appropriate for the more mature state of FedUp development
at this point in time, and it should have been changed earlier. We in QA
apologize to the project for this oversight.
Other factors
+++++++++++++
Additionally, various parties have noted in discussion of this issue
that we would have been more likely to notice it, even with our
imperfect testing procedures, if a couple of other factors had been
different:
* The lifetime of the final release candidate
* The timing of changes to fedup
In recent Fedora releases it has become something of a habit (for which
I personally bear rather a large share of the blame) for us to reach RC
stage late, iterate RCs rapidly, and often ship an RC that was built
only days or even hours before the Go/No-Go decision. This has allowed
us to fix bugs we might not otherwise have fixed and to avoid release
delays.
However, it has the obvious danger that testing of the final release
bits may not be as comprehensive as it could be. We always ensure the
formal validation testing is sufficiently complete, but an issue like
this highlights that a few more days of testing are likely to catch
things the formal validation testing process may miss for various
reasons, including the kind of deficiency noted above. Even though our
test procedure for fedup was outdated, if the final RC had lived for two
or three days before being signed off, someone would likely have
happened across this issue in time for something to be done about it.
The fact that fedup 0.8 and fedup-dracut 0.8 landed quite late in the
cycle is also relevant. fedup 0.8 was submitted for updates-testing on
2013-12-11; fedup-dracut was submitted on 2013-12-06, but the first
compose which used it was RC1, built on 2013-12-12 (there was a delay of
several days between TC5 and RC1, as blocker bugs kept appearing and
needing to be fixed before an RC1 could be spun). RC1.1 was signed off
for release on 2013-12-12. Even the mathematically-challenged will note
that this left us extremely limited time to spot the problem. (I had
tested upgrades with the updated fedup-dracut using a 'scratch built'
upgrade initramfs rather earlier, but I must have used fedup 0.8 rather
than 0.7 for my tests).
Obviously, if these fairly significant version bumps had arrived
earlier, we may have had more time to identify issues in them. If you're
wondering how they were allowed to land so late, the answer is that they
fixed blocker bugs we had identified in earlier upgrade testing, and so
were allowed through the freeze. As we all know, it is difficult to
adhere strictly to 'best practices' with Fedora's extremely short
release cycles and ambitious pace of development, but of course it would
be best in future if we can manage to avoid landing significant changes
to fedup so late, a goal to which both QA and development groups can
contribute by identifying and fixing issues at an earlier stage.
As a 'meta' note, I think a factor that contributes to all of the above
factors may be a lack of understanding outside a very few people as to
precisely how the entire fedup process works: speaking personally, I
certainly wasn't acquainted with all the subtleties until investigating
this and other issues (not that I'd confidently claim to be an expert
even now!) I think beyond Will Woods (obviously) and possibly Tim Flink
(who did a lot of early fedup testing) and Dennis Gilmore (who tends to
be the one generating the upgrade initramfs), possibly no-one really
entirely understood the whole process.
Related issues
--------------
It is probably worth noting a somewhat-related issue at this point.
fedup 0.8's major change compared to fedup 0.7 was that it introduced
checking of GPG signatures on update packages. To facilitate this, the
signing key for the release to which one is upgrading must be available
to fedup running on the release from which one is upgrading. Again, we
did not have this fully in place at the time of Fedora 20's release.
The fedora-release-19-5 update added Fedora 20's key to Fedora 19:
https://admin.fedoraproject.org/updates/FEDORA-2013-21411/fedora-release-... . It was submitted on 2013-11-14 and pushed stable on 2013-12-03, so this was in place ahead of release.
However, for Fedora 18, the relevant update -
https://admin.fedoraproject.org/updates/FEDORA-2013-23598/fedora-release-... - was submitted on 2013-12-18 and pushed stable on 2013-12-22 (and then we had to add a signed .treeinfo file to the Fedora 18 repositories or things *still* didn't work, which I think we did late on 2013-12-22 or on 2013-12-23). The fact that the keys weren't available for F18 was known around F20 release time, but was not considered urgent by the parties involved as we were not aware that fedup 0.7 simply would not work and consequently that it would be an urgent matter to make fedup 0.8 available and functional, and release engineering considered it a delicate operation to add the keys for Fedora 19 and 20 to Fedora 18, and one which they were not inclined to rush.
Post-release reports also make it clear that fedup will abort if GPG
keys for *any* repository fedup finds available for the target release
cannot be found. i.e., if you have RPM Fusion or another popular third
party repository configured, it's quite likely your upgrade will fail,
because third party repos didn't have the signing key issue lined up
(not surprising if we couldn't even entirely manage it ourselves). We
were not sufficiently aware of this behaviour before release, and did
not communicate it very well. The underlying causes of this are much the
same as the underlying causes of the main issue - the fedup which
enabled GPG checking landing very late, inadequate/incorrect test
procedures, and limited knowledge of the details of fedup operation
outside a small group of people.
Addressing the problems
-----------------------
I've noted above that so far as specific code responses to any of these
issues go, we should probably defer to the wisdom of the maintainer.
However, I've filed a couple of intentionally vague and open-ended
tickets on fedup to provide a forum for action:
https://github.com/wgwoods/fedup/issues/42
https://github.com/wgwoods/fedup/issues/43
In terms of QA test procedures, we (QA) have already taken action that
should help guard against a repeat of this kind of issue in future. The
FedUp test cases - for instance,
https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_des... - have been adjusted to recommend testing the latest fedup from stable or updates (not from updates-testing), and to test against the current TC/RC tree (not the daily-updated development/ tree), now TC/RC trees contain the upgrade initramfs image. The FedUp and Upgrading wiki pages (https://fedoraproject.org/wiki/FedUp and https://fedoraproject.org/wiki/Upgrading ) have also been updated to be more consistent and correct for the current state of fedup, and the Installation Guide's section on upgrading has also been updated. Our test procedures and upgrade documentation should now be much more coherent and consistent than they were just prior to Fedora 20's release.
In wider terms, this issue is another indicator on top of several
previous ones that we should redouble our efforts to get 'releaseable'
RCs built days ahead of go/no-go, rather than hours. That's a whole
story in itself, but this is something the parties involved are all
aware of and working on. Of course, the whole release process may look
somewhat different in a Fedora.next world, but as long as we have our
current release schedule and freeze policies, this issue is likely to
exist at least in essence.
It's also another good indicator that we should do whatever we can to
try and land major changes much earlier in the release cycle. This is
hardly a new observation, of course, nor an issue of which many relevant
people were previously unaware, and there are always good reasons why we
wind up landing the kitchen sink a week before release, but it's always
good to have another reminder.
On the positive side, the simple fact that this issue occurred has
probably led to a wider understanding of at least some of the details of
how fedup operates, and the fact that more people in the project have
that knowledge should aid us in future fedup development and testing: we
should be careful to keep that knowledge in mind as we build and test
future releases.
Conclusion
----------
Er, thanks for reading this far? :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10 years, 5 months
Many attempts to install f20 today -
by Robert Moskowitz
Today was the day to install f20 on my Lenovo x120e. I had two drives
to work with: an old 320Gb HD and a new 240Gb SSD.
I thought I would be smart and hibernate my system with the current
drive (f17). Don't know why I did not just poweroff. Something major
went wrong and now that drive will not boot up. So building the new
drive became imperative.
Short and long of it see bugzilla reports: 1006304, 1033749, & 1046482.
I am up and running with i386 on the 320Gb HD. I was using x86_64 with
f17. At least I have an installation and I can access my emails and
figure out how to get x86_64 on the SSD.
The problems writing the bootloader MAY be due to USB problems I have
observed on this system. ~5 months ago, I was no longer able to write
USB sticks or SD cards (in the SD slot). I have been able to write to
USB HD and USB CD and DVD. Or at least it seems so.
How can I determine if I really do have USB problems and have to buy new
hardware?
With all the attempts to install, I spent a lot of time with the
installation and have a couple of small strange behaviours to report:
In setting the location from New York to Detroit, if I did not get
Detroit the 1st time or even when back to try and select something else,
I lost the down arrow to scroll beyond the 'a's. I had to type Detroit
in on the location bar. This was consistant behaviour. Had enough
times doing this step.
If I selected my local repo, Updates became not an option. Regardless if
I used the DVD install (i386 and x86_64) or the netinstal (only tried
x86_64), consistantly this became greyed out. I could not provide the
URL for where I have my local updates repo. It did not matter if I added
repo=url to the boot line, or did it in the GUI. The moment I selected
my own http URL, I lost updates.
As far as adding repo= to the boot line, i386 and x86_64 work
differently! But I suspect you know that. tab with i386 and 'e' with
x86_64.
Sometimes when I selected LVM for the partitioning, the LVM partition
name would be 'fedora_19'. I left it like that in this install that
finally worked and here is what df has to say for itself:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora_19-root 29G 4.8G 23G 18% /
devtmpfs 1.3G 0 1.3G 0% /dev
tmpfs 1.3G 164K 1.3G 1% /dev/shm
tmpfs 1.3G 1016K 1.3G 1% /run
tmpfs 1.3G 0 1.3G 0% /sys/fs/cgroup
tmpfs 1.3G 44K 1.3G 1% /tmp
/dev/sda1 477M 96M 352M 22% /boot
/dev/mapper/fedora_19-home 257G 32G 212G 14% /home
Some times it used the host name. Depended on what steps I took to get
to setting up the LVM partition.
In summary of what is important:
Why have my installs to the SSD failed (writing bootloader).
Why can I not provide my updates URL.
thank you
10 years, 5 months
