Plan / proposal: enable openQA update testing and potentially
gating on Rawhide updates
by Adam Williamson
Hi folks!
We've had openQA testing of updates for stable and branched releases,
and gating based on those tests, enabled for a while now. I believe
this is going quite well, and I think we addressed the issues reported
when we first enabled gating - Bodhi's gating status updates work more
smoothly now, and openQA respects Bodhi's "re-run tests" button so
failed tests can be re-triggered.
A few weeks ago, I enabled testing of Rawhide updates in the openQA
lab/stg instance. This was to see how smoothly the tests run, how often
we run into unexpected failures or problems, and whether the hardware
resources we have are sufficient for the extra load.
So far this has been going more smoothly than I anticipated, if
anything. The workers seem to keep up with the test load, even though
one out of three worker systems for the stg instance is currently out
of commission (we're using it to investigate a bug). We do get
occasional failures which seem to be related to Rawhide kernel slowness
(e.g. operations timing out that usually don't otherwise time out), but
on the whole, the level of false failures is (I would say) acceptably
low, enough that my current regime of checking the test results daily
and restarting failed ones that don't seem to indicate a real bug
should be sufficient.
So, I'd like to propose that we enable Rawhide update testing on the
production openQA instance also. This would cause results to appear on
the Automated Tests tab in Bodhi, but they would be only informational
(and unless the update was gated by a CI test, or somehow otherwise
configured not to be pushed automatically, updates would continue to be
pushed 'stable' almost immediately on creation, regardless of the
openQA results).
More significantly, I'd also propose that we turn on gating on openQA
results for Rawhide updates. This would mean Rawhide updates would be
held from going 'stable' (and included in the next compose) until the
gating openQA tests had run and passed. We may want to do this a bit
after turning on the tests; perhaps Fedora 37 branch point would be a
natural time to do it.
Currently this would usually mean a wait from update submission to
'stable push' (which really means that the build goes into the
buildroot, and will go into the next Rawhide compose when it happens)
of somewhere between 45 minutes and a couple of hours. It would also
mean that if Rawhide updates for inter-dependent packages are not
correctly grouped, the dependent update(s) will fail testing and be
gated until the update they depend on has passed testing and been
pushed. The tests for the dependent update(s) would then need to be re-
run, either by someone hitting the button in Bodhi or an openQA admin
noticing and restarting them, before the dependent update(s) could be
pushed.
In the worst case, if updated packages A and B both need the other to
work correctly but the updates are submitted separately, both updates
may fail tests and be blocked. This could only be resolved by waiving
the failures, or replacing the separate updates with an update
containing both packages.
All of those considerations are already true for stable and branched
releases, but people are probably more used to grouping updates for
stable and branched than doing it for Rawhide, and the typical flow of
going from a build to an update provides more opportunity to create
grouped updates for branched/stable. For Rawhide the easiest way to do
it if you need to do it is to do the builds in a side tag and use
Bodhi's ability to create updates from a side tag.
As with branched/stable, only critical path updates would have the
tests run and be gated on the results. Non-critpath updates would be
unaffected. (There's a small allowlist of non-critpath packages for
which the tests are also run, but they are not currently gated on the
results).
I think doing this could really help us keep Rawhide solid and avoid
introducing major compose-breaking bugs, at minimal cost. But it's a
significant change and I wanted to see what folks think. In particular,
if you find the existing gating of updates for stable/branched releases
to cause problems in any way, I'd love to hear about it.
Thanks folks!
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
10 months, 3 weeks
Small rant: installer environment size
by Adam Williamson
Hi folks! Today I woke up and found
https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me
down a bit of an "installer environment size" rabbit hole.
As of today, with that new dep in webkitgtk, Rawhide's network install
images are 703M in size. Here's a potted history of network install
image sizes:
Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M)
Fedora 13: 208M
Fedora 17: 162M (last "old UI")
Fedora 18: 294M (first "new UI")
Fedora 23: 415M
Fedora 28: 583M
Fedora 33: 686M
Fedora 37: 665M
Fedora Rawhide: 703M
The installer does not really do much more in Rawhide than it did in
FC8. Even after the UI rewrite in F18, we were only at 294M. Now the
image is well over 2x as big and does...basically the same.
Why does this matter? Well, the images being large is moderately
annoying in itself just in terms of transfer times and so on. But more
importantly, AIUI at least, the entire installer environment is loaded
into RAM at startup - it kinda has to be, we don't have anywhere else
to put it. The bigger it is, the more RAM you need to install Fedora.
The size of the installer environment (for which the size of the
network install image is more or less a perfect proxy) is one of the
two key factors in this, the other being how much RAM DNF uses during
package install.
So, I did a bit of poking about into *what* is taking up all that
space. There's a variety of answers, but there's two major culprits:
1. firmware
2. yelp (which pulls in webkitgtk and its deps)
I've been using du and baobab (the GNOME visual disk usage analyzer,
which is great) to examine the filesystems, but I ran a couple of test
builds to confirm these suspects, especially after the impact of
compression (it's hard to check the *compressed* size of things in the
installer environment directly).
I did a scratch build of lorax which does not pull in firmware
packages, and had openQA build a netinst using that lorax. It came out
at 489M - 214M smaller than current netinsts, a size we last managed in
Fedora 26. I did a scratch build of anaconda with its requirement of
yelp dropped (which would break help pages), and built a netinst with
that; it came out at 662M - 41M smaller than current images. I haven't
run a combined test yet, but it ought to come out around 448M, around
the size of Fedora 24.
Even then we'd still be about 50% larger than the Fedora 18 image, for
not really any added functionality.
I've moaned about the sheer amount and size of firmware blobs in other
forums before, but 214M compressed is *really* obnoxious. We must be
able to do something to clean this up (further than it's already
cleaned up - this is *after* we dropped low-hanging fruit like
enterprise switch 'firmwares' and garbage like that; most of the
remaining size seems to be huge amounts of probably-very-similar
firmware files for AMD graphics adapters and Intel wireless adapters).
I know some folks were trying to work on this (there was talk that we
could drop quite a lot of files that would only be loaded by older
kernels no longer in Fedora); any news on how far along that effort is?
Other obvious things that take up a lot of space:
1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
224M uncompressed. A quick test just compressing the file with xz on my
system shows it compresses to around 11M, though, so that's probably
all it adds up to after compression (the image is an xz-compressed
squashfs)
2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to
23M. We are, I think, basically stuck with this for mesa-dri-drivers ,
but does it have to be so *big*?
3. libicudata.so.71.1 - 30.4M, compresses to 7M. This is in the
webkitgtk dep chain but seems to still be pulled in without it, not
sure what else is requiring it.
4. /usr/share/locale - 112M in total (uncompressed, not sure how much
compressed) of translated strings from a ton of packages. No idea how
many of these are really *needed* in the installer environment. We can
maybe come up with a way to have lorax strip some, if we can come up
with a viable way to figure out which. Obviously-fairly-large ones are
from gnupg2 and libgweather4. I do recall we have some logic somewhere
to decide which languages have a certain level of translation in
anaconda; perhaps we could only include the strings for these
languages?
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 2 months
Starting Flatpak SIG
by Kalev Lember
Hi all,
I would like to kick of a Fedora Flatpak Packaging SIG that meets
regularly and has IRC meetings.
Our flatpaks have gotten some bad rep recently and I think for a good
reason. We don't have a lot of them and we only have a handful of people
working on them. We also have a fairly obvious conflict of interest with
Flathub and a bunch of Fedora people are doing flatpaks in Flathub
instead of Fedora and advocating against using Fedora flatpaks.
Here's an attempt to try to get more people involved :) The awkward
situation here is that I am not 100% convinced myself that we should
continue doing Fedora flatpaks, but maybe with a bit more organization
around this we can get some more eyes on the problem and solve some of
the issues.
Please put your name in https://fedoraproject.org/wiki/SIGs/Flatpak if
you are interested and I'll try to organize a poll for a weekly meeting
time. I'd suggest to start the meetings on the second week of January
when the holidays are over.
I've added a few people to CC that I thought might be interested in
this, but anyone that has interest in the area is more than welcome to join.
[Edit: the email got rejected from the mailing list due to too many
recipients so I'm re-sending it without the CC list. The CC'd people
hopefully got the message already.]
Some questions I have around organization: Can someone help register the
#fedora-flatpaks IRC channel under the Fedora umbrella? I am not sure
how to best do that. Does matrix need any special sauce to get a room?
What would be a good name for a pagure.io project?
pagure.io/fedora-flatpaks or pagure.io/fedora-flatpak-sig or
pagure.io/flatpak-sig or something else?
Do we need a mailing list? My initial gut feeling is no, but I'm
interested in what other people think.
If you reply to this, please make sure to reply to the
devel(a)lists.fedoraproject.org post as I am cross-posting it to desktop@
as well.
--
Thanks,
Kalev
1 year, 3 months
[Fedora 38] Call for Test Days
by Sumantro Mukherjee
Hi Fedora users, developers, and friends!
It's time to start thinking about Test Days for Fedora 38.
For anyone who isn't aware, a Test Day is an event usually focused
around IRC for interaction and a Wiki page for instructions and results,
with the aim being to get a bunch of interested users and developers
together to test a specific feature or area of the distribution. You can
run a Test Day on just about anything for which it would be useful to do
some fairly focused testing in 'real time' with a group of testers; it
doesn't have to be code, for instance, we often run Test Days for
l10n/i18n topics. For more information on Test Days, see
https://fedoraproject.org/wiki/QA/Test_Days .
Anyone who wants to can host their own Test Day, or you can request that
the QA group helps you out with organization or any combination of the
two. To propose a Test Day, just file a ticket in fedora-qa pagure - here's
an example https://pagure.io/fedora-qa/issue/624 . For
instructions on hosting a Test Day, see
https://fedoraproject.org/wiki/QA/SOP_Test_Day_management .
You can see the schedule at https://pagure.io/fedora-qa/issues?tags=test+days .
There are many slots open right now. Consider the development
schedule, though, in deciding when you want to run your Test Day - for
some topics you may want to avoid
the time before the Beta release or the time after the feature freeze
or the Final Freeze.
We normally aim to schedule Test Days on Thursdays; however, if you want
to run a series of related Test Days, it's often a good idea to do
something like Tuesday / Wednesday / Thursday of the same week (this is
how we usually run the X Test Week, for instance). If all the Thursday
slots fill up but more people want to run Test Days, we will open up
Tuesday slots as overflows. And finally, if you really want to run a
Test Day in a specific time frame due to the development schedule, but
the Thursday slot for that week is full, we can add a slot on another
day. We're flexible! Just put in your ticket the date or time frame you'd
like, and we'll figure it out from there.
If you don't want to run your own Test Day, but you are willing to
help with another, feel free to join one or more of already accepted
Test Days:
GNOME Test Day*
i18n Test Day*
Kernel Test Week(s)*
Upgrade Test Day*
IoT Test Week*
Cloud Test Day*
Fedora CoreOS Test Week*
And don't be afraid, there are a lot of more slots available for your
own Test Day!
[*] These are the test days we run generally to make sure everything
is working fine, the dates get announced as we move into the release
cycle.
If you have any questions about the Test Day process, please don't
hesitate to contact me or any member of the Fedora QA team on test at
lists.fedoraproject.org or in #fedora-qa on IRC. Thanks!
--
//sumantro
Fedora QE
TRIED AND PERSONALLY TESTED, ERGO TRUSTED
1 year, 3 months
Fedora Workstation WG minutes, 2022-12-13
by Chris Murphy
==============================================
#fedora-meeting-2: Workstation WG (2022-12-13)
==============================================
Meeting started by brainycmurf at 03:55:10 UTC. The full logs are
available at
https://meetbot.fedoraproject.org/fedora-meeting-2/2022-12-19/workstation...
.
Meeting summary
---------------
* Present members: Michael, Chris, Allan, Jens, Matthias, Tomas, Neal
(brainycmurf, 03:55:24)
* Guests: (brainycmurf, 03:55:25)
* Regrets: Kalev (brainycmurf, 03:55:25)
* Secretary: Jens (brainycmurf, 03:55:25)
* Drop flathub filtering (brainycmurf, 03:55:25)
* LINK: https://pagure.io/fedora-workstation/issue/300 (brainycmurf,
03:55:25)
* Improve the quality of the desktop wallpapers & ensure they're
maintained (brainycmurf, 03:55:31)
* LINK: https://pagure.io/fedora-workstation/issue/293 (brainycmurf,
03:55:33)
* LINK: https://fedoraproject.org/wiki/Wallpapers (brainycmurf,
03:55:37)
* LINK:
https://gitlab.com/fedora/design/team/release-artwork/default-wallpaper/-...
(brainycmurf, 03:55:41)
* ACTION: Allen to talk with Mairin (brainycmurf, 03:55:43)
* Ship preininstalled apps as Flatpaks (brainycmurf, 03:55:45)
* LINK: https://pagure.io/fedora-workstation/issue/269 (brainycmurf,
03:55:47)
* LINK: https://fedoraproject.org/wiki/SIGs/Flatpak (brainycmurf,
03:55:53)
* ACTION: Neal to note down his concerns about moving to default
flatpaks (brainycmurf, 03:55:59)
* Offline automated workstation installation not possible (brainycmurf,
03:56:01)
* LINK: https://pagure.io/fedora-workstation/issue/85 (brainycmurf,
03:56:03)
* ACTION: Michael will ping Felipe and the reporter (brainycmurf,
03:56:05)
* Announcements, Status Updates (brainycmurf, 03:56:11)
* next meeting 10th Jan (brainycmurf, 03:56:13)
* The minutes from last week have been posted. (brainycmurf,
03:56:15)
* LINK:
https://meetbot.fedoraproject.org/fedora-meeting-2/2022-12-07/workstation...
(brainycmurf, 03:56:17)
Meeting ended at 03:56:40 UTC.
Action Items
------------
* Allen to talk with Mairin
* Neal to note down his concerns about moving to default flatpaks
* Michael will ping Felipe and the reporter
Action Items, by person
-----------------------
* Michael
* Michael will ping Felipe and the reporter
* **UNASSIGNED**
* Allen to talk with Mairin
* Neal to note down his concerns about moving to default flatpaks
People Present (lines said)
---------------------------
* brainycmurf (35)
* zodbot (7)
* Michael (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
1 year, 4 months
Re: Small rant: installer environment size
by Florian Weimer
* Richard W. M. Jones:
> You only need network / wifi firmware blobs (although I'm sure they
> are in themselves large) and then you can fetch anything else needed
> for the hardware including graphics, right?
I think you need graphics to set up wifi.
Thanks,
Florian
1 year, 4 months
Re: Small rant: installer environment size
by drago01
On Thursday, December 8, 2022, Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
> On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote:
> > On Thursday, December 8, 2022, Adam Williamson <
> adamwill(a)fedoraproject.org>
> > wrote:
> >
> > > On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
> > > >
> > > > I've done a few passes, dropping a bunch of older firmware upstream
> > > > that are no longer supported in any stable kernel release, also a
> > > > bunch of de-dupe and linking of files rather than shipping of
> multiple
> > > > copies of the same firmware. It's improved things a bit,
> unfortunately
> > > > a lot of the dead firmware was tiny compared to say average modern
> > > > devices like GPUs or WiFI.
> > > >
> > > > The problem with a lot of the firmware, and with the new nvidia "open
> > > > driver" which shoves a lot of stuff into firmware in order to have an
> > > > upstreamable driver apparently the firmwares there are going to be
> > > > 30+Mb each, is that they're needed to bring up graphics/network etc
> to
> > > > even just install so I don't know how we can get around this and
> still
> > > > have a device work enough to be able to install the needed firmware
> > > > across the network.
> > > >
> > > > Ideas on how to solve that problem welcome.
> > >
> > > Sorry if this is way off, but - do we need the GPU firmwares to run a
> > > graphical install on the fallback path, just using the framebuffer set
> > > up by the firmware? How crazy would it be to just do that - ship the
> > > installer env with no GPU firmware?
> >
> >
> > That would be very crazy, as you will have a degraded user experience
> > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are
> a
> > non issue for today's hardware.
>
> Please bear in mind the difference between bare metal and virtual
> machines. The bare metal machine may have 32 GB of RAM, making a
> 800 MB install image a non-issue. For a public cloud virtual machine
> though, this could bump your VM sizing up 1 level from 2 GB quota
> to a 4 GB RAM quota, with correspondingly higher price point. Now
> most people probably don't run the installer in a public cloud,
> preferring pre-built disk images. Even in a local machine though,
> you may be using most of your 32 GB of RAM for other things (well
> firefox/chrome), so allowing extra for the VM is not without
> resource cost. If we could figure out a way to knock a few 100 MB
> off the installer RAM requirements that is valuable.
>
>
The problem I see here is not the presence of the firmware on the image,
but the fact that it seems to be loaded into memory despite not being used.
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/
> dberrange :|
> |: https://libvirt.org -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/
> dberrange :|
> _______________________________________________
> kernel mailing list -- kernel(a)lists.fedoraproject.org
> To unsubscribe send an email to kernel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://docs.fedoraproject.
> org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/kernel@
> lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-
> infrastructure/new_issue
>
1 year, 4 months
Re: Small rant: installer environment size
by Adam Williamson
On Thu, 2022-12-08 at 12:50 -0500, David Cantrell wrote:
> On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote:
> > Hi folks! Today I woke up and found
> > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me
> > down a bit of an "installer environment size" rabbit hole.
> >
> > As of today, with that new dep in webkitgtk, Rawhide's network install
> > images are 703M in size. Here's a potted history of network install
> > image sizes:
> >
> > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M)
> > Fedora 13: 208M
> > Fedora 17: 162M (last "old UI")
> > Fedora 18: 294M (first "new UI")
> > Fedora 23: 415M
> > Fedora 28: 583M
> > Fedora 33: 686M
> > Fedora 37: 665M
> > Fedora Rawhide: 703M
> >
> > The installer does not really do much more in Rawhide than it did in
> > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the
> > image is well over 2x as big and does...basically the same.
>
> I take issue with this. It is not accurate to say that the installer now does
> not do much more than it did for Fedora Core 8. There is more to the
> installer than the UI.
>
> Broadly speaking, a lot of the growth came from converging the runtime
> environment for the installer with the installed system. In Fedora Core 8 and
> previous releases, the "installer environment" was a unique and stripped down
> install. This was frustrating because it was effectively maintaining a small
> mini distro for the purposes of running the distro installer.
I meant it doesn't do much more in terms of what it achieves for the
user. But we can also just take F18 as the base point, if you like.
We're still over 2x as big as that was.
> I think some curation on firmware could happen. I think probinson@ mentions
> it later in this thread. If there were a way to identify the firmware
> necessary for the installer environment that could probably simplify things.
We already do a lot of "identifying the firmware necessary for the
installer environment", see my earlier mail about all the stuff lorax
does here. We've done the easy part, unfortunately. The stuff that's
left is stuff that is, in some sense, needed - graphics card and
wireless adapter firmwares, mainly.
>
> > Other obvious things that take up a lot of space:
> >
> > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
> > 224M uncompressed. A quick test just compressing the file with xz on my
> > system shows it compresses to around 11M, though, so that's probably
> > all it adds up to after compression (the image is an xz-compressed
> > squashfs)
>
> Can this be installed compressed? I'm not sure it can.
The "compressed" size is the effective size we're concerned about,
because the installer filesystem image is an xz-compressed squashfs. So
11M is already the "effective weight" of this file, I think - if I ran
an image build with it removed, it'd probably be ~11M smaller.
>
> > 2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to
> > 23M. We are, I think, basically stuck with this for mesa-dri-drivers ,
> > but does it have to be so *big*?
>
> If mesa-dri-drivers is not required for installation, it could be removed from
> the installer environment.
It is required - we don't get any graphics without it.
> > 4. /usr/share/locale - 112M in total (uncompressed, not sure how much
> > compressed) of translated strings from a ton of packages. No idea how
> > many of these are really *needed* in the installer environment. We can
> > maybe come up with a way to have lorax strip some, if we can come up
> > with a viable way to figure out which. Obviously-fairly-large ones are
> > from gnupg2 and libgweather4. I do recall we have some logic somewhere
> > to decide which languages have a certain level of translation in
> > anaconda; perhaps we could only include the strings for these
> > languages?
>
> On that note, /usr/share/doc, /usr/share/man, and /usr/share/info could be
> removed from the installer image if they are present. That likely won't free
> a whole lot of space, but it's not nothing.
All of those are already stripped:
https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/r...
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 4 months