Missing LLVM stack bugfix updates in stable Fedora branches

[Fabio Valentini]

Hi all,

Today I encountered another LLVM-specific bug that affects at least
one Rust package and causes non-working code to be produced, which
prompted this question:

Why are stable releases not getting bugfix releases of LLVM?

Fedora 35 is stuck at LLVM 13.0.0, while 13.0.1 has been released.
Fedora 36 is stuck at LLVM 14.0.0, while 14.0.1 through 14.0.6 have
been released.
Even Rawhide is at LLVM 14.0.5, but has not been updated to 14.0.6 yet
(released over 3 weeks ago).

However, the llvm13 compat package that exists on Fedora 36+ *has*
been updated to version 13.0.1, so I'm not sure why this update wasn't
also pushed to Fedora 35, where LLVM 13 is the default, and would
benefit much more from bugfixes provided by 13.0.1.

Given that llvm and the whole llvm ecosystem (clang, lld, rustc, ghc
on some architectures, mesa/llvmpipe) are an important part of our
stack, it seems bad that stable releases are missing out on several
bugfix updates for those critical packages.

I appreciate that updating ~a dozen packages for new LLVM point
releases is work, but I don't think having outdated LLVM components on
stable releases of Fedora is a good idea, either.

What can we do to improve this situation?

Fabio

[Peter Robinson]

Hi,

>  Today I encountered another LLVM-specific bug that affects at least
>  one Rust package and causes non-working code to be produced, which
>  prompted this question:
>
>  Why are stable releases not getting bugfix releases of LLVM?
>
>  Fedora 35 is stuck at LLVM 13.0.0, while 13.0.1 has been released.
>  Fedora 36 is stuck at LLVM 14.0.0, while 14.0.1 through 14.0.6 have
>  been released.
>  Even Rawhide is at LLVM 14.0.5, but has not been updated to 14.0.6 yet
>  (released over 3 weeks ago).
>
>  However, the llvm13 compat package that exists on Fedora 36+ *has*
>  been updated to version 13.0.1, so I'm not sure why this update wasn't
>  also pushed to Fedora 35, where LLVM 13 is the default, and would
>  benefit much more from bugfixes provided by 13.0.1.
>
>  Given that llvm and the whole llvm ecosystem (clang, lld, rustc, ghc
>  on some architectures, mesa/llvmpipe) are an important part of our
>  stack, it seems bad that stable releases are missing out on several
>  bugfix updates for those critical packages.
>
>  I appreciate that updating ~a dozen packages for new LLVM point
>  releases is work, but I don't think having outdated LLVM components on
>  stable releases of Fedora is a good idea, either.
>
>  What can we do to improve this situation? 
Have you reached out to the maintainer(s)? In the past they seem to
have been quite effective at keeping things up to date AFAIA so maybe
they've got stuff going on of late, PTO, have other priorities or
something else.

You don't mention if you've had a discussion with the maintainer(s),
whether they replied, or even cc:ed them on this mail. I personally
would tend to do that before mailing a list, but then maybe you have
and you didn't mention it here.

Peter

[Fabio Valentini]

On Mon, Jul 18, 2022 at 3:26 PM Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>  Hi,
>
>  > Today I encountered another LLVM-specific bug that affects at least
>  > one Rust package and causes non-working code to be produced, which
>  > prompted this question:
>  >
>  > Why are stable releases not getting bugfix releases of LLVM?
>  >
>  > Fedora 35 is stuck at LLVM 13.0.0, while 13.0.1 has been released.
>  > Fedora 36 is stuck at LLVM 14.0.0, while 14.0.1 through 14.0.6 have
>  > been released.
>  > Even Rawhide is at LLVM 14.0.5, but has not been updated to 14.0.6 yet
>  > (released over 3 weeks ago).
>  >
>  > However, the llvm13 compat package that exists on Fedora 36+ *has*
>  > been updated to version 13.0.1, so I'm not sure why this update wasn't
>  > also pushed to Fedora 35, where LLVM 13 is the default, and would
>  > benefit much more from bugfixes provided by 13.0.1.
>  >
>  > Given that llvm and the whole llvm ecosystem (clang, lld, rustc, ghc
>  > on some architectures, mesa/llvmpipe) are an important part of our
>  > stack, it seems bad that stable releases are missing out on several
>  > bugfix updates for those critical packages.
>  >
>  > I appreciate that updating ~a dozen packages for new LLVM point
>  > releases is work, but I don't think having outdated LLVM components on
>  > stable releases of Fedora is a good idea, either.
>  >
>  > What can we do to improve this situation?
>
>  Have you reached out to the maintainer(s)? In the past they seem to
>  have been quite effective at keeping things up to date AFAIA so maybe
>  they've got stuff going on of late, PTO, have other priorities or
>  something else.
>
>  You don't mention if you've had a discussion with the maintainer(s),
>  whether they replied, or even cc:ed them on this mail. I personally
>  would tend to do that before mailing a list, but then maybe you have
>  and you didn't mention it here. 
I have filed multiple bugs against llvm (or rust, which were then
reassigned to llvm) and llvm compat packages over the past 1-2 years,
and they've been pretty much ignored.
Also, given that rawhide *is* up-to-date with LLVM bugfix releases
until about three weeks ago, but the llvm packages on f36 and f35
weren't touched since 4 months (f36) and f35 (9 months) ago,
respectively, so I don't think PTO can be a problem here.
Or, if it is, then we'd need to have a serious talk about adding
backup maintainers to those packages ...

Fabio

[Tom Stellard]

On 7/18/22 06:33, Fabio Valentini wrote:
>  On Mon, Jul 18, 2022 at 3:26 PM Peter Robinson
> <pbrobinson(a)gmail.com&gt; wrote:
> >
> > Hi,
> >
> >> Today I encountered another LLVM-specific bug that affects at least
> >> one Rust package and causes non-working code to be produced, which
> >> prompted this question:
> >>
> >> Why are stable releases not getting bugfix releases of LLVM?
> >>
> >> Fedora 35 is stuck at LLVM 13.0.0, while 13.0.1 has been released.
> >> Fedora 36 is stuck at LLVM 14.0.0, while 14.0.1 through 14.0.6 have
> >> been released.
> >> Even Rawhide is at LLVM 14.0.5, but has not been updated to 14.0.6 yet
> >> (released over 3 weeks ago).
> >>
> >> However, the llvm13 compat package that exists on Fedora 36+ *has*
> >> been updated to version 13.0.1, so I'm not sure why this update wasn't
> >> also pushed to Fedora 35, where LLVM 13 is the default, and would
> >> benefit much more from bugfixes provided by 13.0.1.
> >>
> >> Given that llvm and the whole llvm ecosystem (clang, lld, rustc, ghc
> >> on some architectures, mesa/llvmpipe) are an important part of our
> >> stack, it seems bad that stable releases are missing out on several
> >> bugfix updates for those critical packages.
> >>
> >> I appreciate that updating ~a dozen packages for new LLVM point
> >> releases is work, but I don't think having outdated LLVM components on
> >> stable releases of Fedora is a good idea, either.
> >>
> >> What can we do to improve this situation?
> >
> > Have you reached out to the maintainer(s)? In the past they seem to
> > have been quite effective at keeping things up to date AFAIA so maybe
> > they've got stuff going on of late, PTO, have other priorities or
> > something else.
> >
> > You don't mention if you've had a discussion with the maintainer(s),
> > whether they replied, or even cc:ed them on this mail. I personally
> > would tend to do that before mailing a list, but then maybe you have
> > and you didn't mention it here.
>  
>  I have filed multiple bugs against llvm (or rust, which were then
>  reassigned to llvm) and llvm compat packages over the past 1-2 years,
>  and they've been pretty much ignored. 
Can you send me links to these bugs.

>  Also, given that rawhide *is* up-to-date with LLVM bugfix releases
>  until about three weeks ago, but the llvm packages on f36 and f35
>  weren't touched since 4 months (f36) and f35 (9 months) ago,
>  respectively, so I don't think PTO can be a problem here.
>  Or, if it is, then we'd need to have a serious talk about adding
>  backup maintainers to those packages ...
>   
We've been training up more people on LLVM packaging so we should be able
to cover the stable release of Fedora better in the future.  I'm sorry
that the stable releases have been falling behind, we'll work on getting them
back up to date.

-Tom

>  Fabio
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Fabio Valentini]

On Mon, Jul 18, 2022 at 4:43 PM Tom Stellard <tstellar(a)redhat.com&gt; wrote:
>  On 7/18/22 06:33, Fabio Valentini wrote:
>  >
>  > I have filed multiple bugs against llvm (or rust, which were then
>  > reassigned to llvm) and llvm compat packages over the past 1-2 years,
>  > and they've been pretty much ignored.
>
>  Can you send me links to these bugs. 
The current ones are (some rust bugs are already reassigned to llvm,
some weren't yet, but probably should be):

https://bugzilla.redhat.com/show_bug.cgi?id=2001328
https://bugzilla.redhat.com/show_bug.cgi?id=2020861
https://bugzilla.redhat.com/show_bug.cgi?id=2041942
https://bugzilla.redhat.com/show_bug.cgi?id=2045116
https://bugzilla.redhat.com/show_bug.cgi?id=2086106

There have been previous ones for F34 that were closed at EOL without
resolution:

https://bugzilla.redhat.com/show_bug.cgi?id=2020861
https://bugzilla.redhat.com/show_bug.cgi?id=1883457

And these are only the ones I could find quickly, I'm pretty sure
there are a few more.

>  > Also, given that rawhide *is* up-to-date with LLVM bugfix
> releases
>  > until about three weeks ago, but the llvm packages on f36 and f35
>  > weren't touched since 4 months (f36) and f35 (9 months) ago,
>  > respectively, so I don't think PTO can be a problem here.
>  > Or, if it is, then we'd need to have a serious talk about adding
>  > backup maintainers to those packages ...
>  >
>
>  We've been training up more people on LLVM packaging so we should be able
>  to cover the stable release of Fedora better in the future.  I'm sorry
>  that the stable releases have been falling behind, we'll work on getting them
>  back up to date. 
That sounds great, thank you for trying to improve the situation.

Fabio

[Fabio Valentini]

On Mon, Jul 18, 2022 at 4:43 PM Tom Stellard <tstellar(a)redhat.com&gt; wrote:
>  We've been training up more people on LLVM packaging so we should be able
>  to cover the stable release of Fedora better in the future.  I'm sorry
>  that the stable releases have been falling behind, we'll work on getting them
>  back up to date. 
I see an update for llvm 13.0.1 has been pushed to Fedora 35, thank
you for that!
Has there been progress in getting llvm updated to 14.0.5 on F36?
Also, the latest bugfix release (14.0.6) has now been out for 2
months, would it be possible to get that update for Fedora soon-ish,
as well?

Thank you for your work,
Fabio

[Tom Stellard]

On 8/18/22 08:05, Fabio Valentini wrote:
>  On Mon, Jul 18, 2022 at 4:43 PM Tom Stellard
> <tstellar(a)redhat.com&gt; wrote:
> >
> > We've been training up more people on LLVM packaging so we should be able
> > to cover the stable release of Fedora better in the future.  I'm sorry
> > that the stable releases have been falling behind, we'll work on getting them
> > back up to date.
>  
>  I see an update for llvm 13.0.1 has been pushed to Fedora 35, thank
>  you for that!
>  Has there been progress in getting llvm updated to 14.0.5 on F36?
>  Also, the latest bugfix release (14.0.6) has now been out for 2
>  months, would it be possible to get that update for Fedora soon-ish,
>  as well?
>   
I'm working on the 14.0.5 update for F36 right now.  The 14.0.6 release
is going to have to wait until after 15.0.0 lands in F37 and rawhide,
because I don't think it's worth doing the 14.0.6 update there, since
15.0.0 is so close.

14.0.6 only has a small fix that probably won't affect most Fedora users,
so I think 14.0.5 should be fine for now.

-Tom

>  Thank you for your work,
>  Fabio
>

[Fabio Valentini]

On Thu, Aug 18, 2022 at 5:23 PM Tom Stellard <tstellar(a)redhat.com&gt; wrote:
>  I'm working on the 14.0.5 update for F36 right now.  The 14.0.6 release
>  is going to have to wait until after 15.0.0 lands in F37 and rawhide,
>  because I don't think it's worth doing the 14.0.6 update there, since
>  15.0.0 is so close. 
Awesome, thanks for the update!

>  14.0.6 only has a small fix that probably won't affect most
> Fedora users,
>  so I think 14.0.5 should be fine for now. 
Sounds good to me :)

Thanks again,
Fabio

[Tom Stellard]

On 8/18/22 15:32, Fabio Valentini wrote:
>  On Thu, Aug 18, 2022 at 5:23 PM Tom Stellard
> <tstellar(a)redhat.com&gt; wrote:
> >
> > I'm working on the 14.0.5 update for F36 right now.  The 14.0.6 release
> > is going to have to wait until after 15.0.0 lands in F37 and rawhide,
> > because I don't think it's worth doing the 14.0.6 update there, since
> > 15.0.0 is so close.
>  
>  Awesome, thanks for the update!
>   
Here is the Bodhi update:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-eecc713e20


> > 14.0.6 only has a small fix that probably won't affect most
> Fedora users,
> > so I think 14.0.5 should be fine for now.
>  
>  Sounds good to me :)
>  
>  Thanks again,
>  Fabio
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue