There have been various change proposals & associated mailing list threads over the years about the possibility of moving Fedora compiler toolchain to build with a newer x86_64 baseline ABI, which have ended up rejected, with some quite strong negative feedback.
Regardless of the Fedora general policy, however, individual packages may have forced a particular x86_64 baseline ABI through their own CFLAGS defined by the upstream project.
The context is that QEMU has recently merged changes upstream that force use of the x86-64-v2 baseline for QEMU, in order get more efficient code in the TCG emulator. The changes were made in QEMU's global CFLAGS so this will affect all usage of QEMU, whether KVM, or TCG, for both VMs and user space emulation (the latter used by podman for non-native containers)
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
With regards, Daniel
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
There have been various change proposals & associated mailing list threads over the years about the possibility of moving Fedora compiler toolchain to build with a newer x86_64 baseline ABI, which have ended up rejected, with some quite strong negative feedback.
Regardless of the Fedora general policy, however, individual packages may have forced a particular x86_64 baseline ABI through their own CFLAGS defined by the upstream project.
The context is that QEMU has recently merged changes upstream that force use of the x86-64-v2 baseline for QEMU, in order get more efficient code in the TCG emulator. The changes were made in QEMU's global CFLAGS so this will affect all usage of QEMU, whether KVM, or TCG, for both VMs and user space emulation (the latter used by podman for non-native containers)
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
With regards, Daniel
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
On Wed, Jun 12, 2024 at 09:59:25AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Not trivial as a bunch of code executes from ELF constructors before main() starts.
With regards, Daniel
On Wed, Jun 12, 2024 at 10:07 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:59:25AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Not trivial as a bunch of code executes from ELF constructors before main() starts.
Wrapper application that just does feature tests and then fork/exec()'s the real application?
* Daniel P. Berrangé:
On Wed, Jun 12, 2024 at 09:59:25AM -0400, Stephen Gallagher wrote:
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Not trivial as a bunch of code executes from ELF constructors before main() starts.
Actually, you can put this into a C file that gets linked into the executable:
asm ("\ .pushsection ".note.gnu.property","a",@note\n\ .p2align 3\n\ .long 4, 16, 5\n\ .asciz "GNU"\n\ .p2align 3\n\ .long 0xc0008002, 4, 3\n\ .p2align 3\n\ .popsection");
It marks the binary as requiring x86-64-v2. It will work only on x86-64, so it has to be made conditional on architecture.
The glibc dynamic linker currently prints, “CPU ISA level is lower than required“ if the CPU requirement cannot be satisfied. I think it's still better than a crash, but maybe the wording could be improved (and some diagnostic information included).
Thanks, Florian
On Wed, Jun 12, 2024 at 03:07:02PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 12, 2024 at 09:59:25AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Not trivial as a bunch of code executes from ELF constructors before main() starts.
A little known feature of GCC constructors if you can assign a priority to them, which controls the ordering (apparently, I did not test). Smaller numbers run the constructor earlier / destructor later:
https://maskray.me/blog/2021-11-07-init-ctors-init-array
Numbers <= 100 are reserved, but unless you opt in with the -Wprio-ctor-dtor flag, you won't get a warning about this. Maybe we can add a constructor(0) which checks CPUID?
Rich.
On Mon, Jun 17, 2024 at 08:24:40AM +0100, Richard W.M. Jones wrote:
On Wed, Jun 12, 2024 at 03:07:02PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 12, 2024 at 09:59:25AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Not trivial as a bunch of code executes from ELF constructors before main() starts.
A little known feature of GCC constructors if you can assign a priority to them, which controls the ordering (apparently, I did not test). Smaller numbers run the constructor earlier / destructor later:
https://maskray.me/blog/2021-11-07-init-ctors-init-array
Numbers <= 100 are reserved, but unless you opt in with the -Wprio-ctor-dtor flag, you won't get a warning about this. Maybe we can add a constructor(0) which checks CPUID?
I think I've convinced upstream to change their approach to make their recent changes a compile-time opt-in, to allow build time choice of the non-optimized code, rather than forcing it on everyone. So hopefully we don't need todo anything in Fedora now.
With regards, Daniel
Once upon a time, Daniel P. Berrangé berrange@redhat.com said:
I think I've convinced upstream to change their approach to make their recent changes a compile-time opt-in, to allow build time choice of the non-optimized code, rather than forcing it on everyone. So hopefully we don't need todo anything in Fedora now.
Since this seems to be performance related, any chance Fedora can provide both a baseline and a x86-64-v2 version, maybe with a wrapper to automatically choose the correct verion?
On Mon, Jun 17, 2024 at 2:37 PM Chris Adams linux@cmadams.net wrote:
Once upon a time, Daniel P. Berrangé berrange@redhat.com said:
I think I've convinced upstream to change their approach to make their recent changes a compile-time opt-in, to allow build time choice of the non-optimized code, rather than forcing it on everyone. So hopefully we don't need todo anything in Fedora now.
Since this seems to be performance related, any chance Fedora can provide both a baseline and a x86-64-v2 version, maybe with a wrapper to automatically choose the correct verion?
You mean ... like this? https://fedoraproject.org/wiki/Changes/Optimized_Binaries_for_the_AMD64_Arch...
Fabio
On Mon, Jun 17, 2024 at 07:37:15AM -0500, Chris Adams wrote:
Once upon a time, Daniel P. Berrangé berrange@redhat.com said:
I think I've convinced upstream to change their approach to make their recent changes a compile-time opt-in, to allow build time choice of the non-optimized code, rather than forcing it on everyone. So hopefully we don't need todo anything in Fedora now.
Since this seems to be performance related, any chance Fedora can provide both a baseline and a x86-64-v2 version, maybe with a wrapper to automatically choose the correct verion?
I really don't want to get into the business of maintaining multiple parallel builds of the same binaries in one package. If Fedora cares about optimal performance it should just declare we're going to stop being held back by compat with ancient hardware and use -v2 baseline for everything, but obviously that's been rejected previously.
With regards, Daniel
On 19/06/2024 09:13, Daniel P. Berrangé wrote:
If Fedora cares about optimal performance it should just declare we're going to stop being held back by compat with ancient hardware and use -v2 baseline for everything, but obviously that's been rejected previously.
Maybe it's a good time for the Fedora 41 System-Wide change proposal? Switching to AVX2 will give significant performance boost for many applications.
On Wed, Jun 19, 2024 at 9:29 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 19/06/2024 09:13, Daniel P. Berrangé wrote:
If Fedora cares about optimal performance it should just declare we're going to stop being held back by compat with ancient hardware and use -v2 baseline for everything, but obviously that's been rejected previously.
Maybe it's a good time for the Fedora 41 System-Wide change proposal? Switching to AVX2 will give significant performance boost for many applications.
Last time, the approach was to do the RHEL approach (ie. lie to RPM about the architecture level we support).
Two things have changed since then:
* I suspect more of the hardware that don't support -v2 have failed out of use naturally * RPM now lets us "tell the truth" about what x86_64 sublevel we expect
If we're now starting to have software that *requires* x86_64-v2, then we should visit this with that idea in mind.
On Wednesday, 19 June 2024 at 10:39, Neal Gompa wrote:
On Wed, Jun 19, 2024 at 9:29 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 19/06/2024 09:13, Daniel P. Berrangé wrote:
If Fedora cares about optimal performance it should just declare we're going to stop being held back by compat with ancient hardware and use -v2 baseline for everything, but obviously that's been rejected previously.
Maybe it's a good time for the Fedora 41 System-Wide change proposal? Switching to AVX2 will give significant performance boost for many applications.
Last time, the approach was to do the RHEL approach (ie. lie to RPM about the architecture level we support).
Two things have changed since then:
- I suspect more of the hardware that don't support -v2 have failed
out of use naturally
- RPM now lets us "tell the truth" about what x86_64 sublevel we expect
If we're now starting to have software that *requires* x86_64-v2, then we should visit this with that idea in mind.
I still have two critical machines that are v1 (old Intel Atom CPUs). They are my local routers/NAS servers (primary and secondary). One is even pre-UEFI. They still work fine and do their job great. I don't mind if some applications I couldn't run on them anyway start requiring v2, but having the whole distro switch to v2 will prevent me from running Fedora on these two.
Regards, Dominik
On Wednesday, June 19, 2024, Dominik 'Rathann' Mierzejewski < dominik@greysector.net> wrote:
On Wednesday, 19 June 2024 at 10:39, Neal Gompa wrote:
On Wed, Jun 19, 2024 at 9:29 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 19/06/2024 09:13, Daniel P. Berrangé wrote:
If Fedora cares about optimal performance it should just declare we're going to stop being held back by compat with ancient hardware and use -v2 baseline for everything, but obviously that's been rejected previously.
Maybe it's a good time for the Fedora 41 System-Wide change proposal? Switching to AVX2 will give significant performance boost for many applications.
Last time, the approach was to do the RHEL approach (ie. lie to RPM about the architecture level we support).
Two things have changed since then:
- I suspect more of the hardware that don't support -v2 have failed
out of use naturally
- RPM now lets us "tell the truth" about what x86_64 sublevel we expect
If we're now starting to have software that *requires* x86_64-v2, then we should visit this with that idea in mind.
I still have two critical machines that are v1 (old Intel Atom CPUs). They are my local routers/NAS servers (primary and secondary). One is even pre-UEFI. They still work fine and do their job great. I don't mind if some applications I couldn't run on them anyway start requiring v2, but having the whole distro switch to v2 will prevent me from running Fedora on these two.
Yes but at some point we need to do the cut and not being held back by old / ancient hardware forever.
On Wednesday, 19 June 2024 at 17:17, drago01 wrote:
[...] at some point we need to do the cut and not being held back by old / ancient hardware forever.
What do you mean by "being held back"? What's being prevented by not requiring x86-64-v2 for all packages while allowing few select ones to have higher arch requirements? And why do "we need to"?
As Neal said, rpm allows building for target x86_64_v2, so... let's do that for those packages that require it?
Regards, Dominik
On Wed, 19 Jun 2024 at 19:13, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
On Wednesday, 19 June 2024 at 17:17, drago01 wrote:
[...] at some point we need to do the cut and not being held back by old / ancient hardware forever.
What do you mean by "being held back"? What's being prevented by not requiring x86-64-v2 for all packages while allowing few select ones to have higher arch requirements? And why do "we need to"?
As Neal said, rpm allows building for target x86_64_v2, so... let's do that for those packages that require it?
That may mean having two versions of the same package, one built against v1 and one v2, you're then doubling the workload for a whole lot of contributors for the sake of old hardware.
We had the same discussions on i686 and the first few times it was proposed it didn't get traction, but it eventually did. There was a i686 SIG which ultimately went nowhere.
To put this a different way, would you be prepared to do the work to maintain the old v1 packages if maintainers don't want to maintain 2 versions?
Peter
On Wed, Jun 19, 2024 at 8:40 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 19 Jun 2024 at 19:13, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
On Wednesday, 19 June 2024 at 17:17, drago01 wrote:
[...] at some point we need to do the cut and not being held back by old / ancient hardware forever.
What do you mean by "being held back"? What's being prevented by not requiring x86-64-v2 for all packages while allowing few select ones to have higher arch requirements? And why do "we need to"?
As Neal said, rpm allows building for target x86_64_v2, so... let's do that for those packages that require it?
That may mean having two versions of the same package, one built against v1 and one v2, you're then doubling the workload for a whole lot of contributors for the sake of old hardware.
We had the same discussions on i686 and the first few times it was proposed it didn't get traction, but it eventually did. There was a i686 SIG which ultimately went nowhere.
To put this a different way, would you be prepared to do the work to maintain the old v1 packages if maintainers don't want to maintain 2 versions?
IIUC this wouldn't require maintaining a separate package, but just adding x86_64-v2 as a separate build *target* for the same package.
Fabio
On Wed, 19 Jun 2024 at 16:23, Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Jun 19, 2024 at 8:40 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 19 Jun 2024 at 19:13, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
On Wednesday, 19 June 2024 at 17:17, drago01 wrote:
[...] at some point we need to do the cut and not being held back by old / ancient hardware forever.
What do you mean by "being held back"? What's being prevented by not requiring x86-64-v2 for all packages while allowing few select ones to have higher arch requirements? And why do "we need to"?
As Neal said, rpm allows building for target x86_64_v2, so... let's do that for those packages that require it?
That may mean having two versions of the same package, one built against v1 and one v2, you're then doubling the workload for a whole lot of contributors for the sake of old hardware.
We had the same discussions on i686 and the first few times it was proposed it didn't get traction, but it eventually did. There was a i686 SIG which ultimately went nowhere.
To put this a different way, would you be prepared to do the work to maintain the old v1 packages if maintainers don't want to maintain 2 versions?
IIUC this wouldn't require maintaining a separate package, but just adding x86_64-v2 as a separate build *target* for the same package.
I don't think Peter meant additional packages since with the i686 it didn't mean that. What it did mean was having to understand why two architectures might do things differently and why bugs might show up in one but not another. It also means that someone needs to continually help release engineering make sure that the entire Fedora build system (koji, composers, mock, createrepo, test tools, etc) make sure the packages get built, and put into the repositories correctly, that images get made correctly and such. This is continual work because as has been seen any change to one tool in an update can make the others stop working in weird ways.
I am not saying this is a 'can't be done'.. but this is not Free Work. The Fedora release engineering is pretty small, its physical and compute resources are pretty confined. You can only keep so many spinning plates going before they all come crashing down.
On Thu, Jun 20, 2024 at 7:30 AM Stephen Smoogen ssmoogen@redhat.com wrote:
On Wed, 19 Jun 2024 at 16:23, Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Jun 19, 2024 at 8:40 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, 19 Jun 2024 at 19:13, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
On Wednesday, 19 June 2024 at 17:17, drago01 wrote:
[...] at some point we need to do the cut and not being held back by old / ancient hardware forever.
What do you mean by "being held back"? What's being prevented by not requiring x86-64-v2 for all packages while allowing few select ones to have higher arch requirements? And why do "we need to"?
As Neal said, rpm allows building for target x86_64_v2, so... let's do that for those packages that require it?
That may mean having two versions of the same package, one built against v1 and one v2, you're then doubling the workload for a whole lot of contributors for the sake of old hardware.
We had the same discussions on i686 and the first few times it was proposed it didn't get traction, but it eventually did. There was a i686 SIG which ultimately went nowhere.
To put this a different way, would you be prepared to do the work to maintain the old v1 packages if maintainers don't want to maintain 2 versions?
IIUC this wouldn't require maintaining a separate package, but just adding x86_64-v2 as a separate build *target* for the same package.
I don't think Peter meant additional packages since with the i686 it didn't mean that. What it did mean was having to understand why two architectures might do things differently and why bugs might show up in one but not another. It also means that someone needs to continually help release engineering make sure that the entire Fedora build system (koji, composers, mock, createrepo, test tools, etc) make sure the packages get built, and put into the repositories correctly, that images get made correctly and such. This is continual work because as has been seen any change to one tool in an update can make the others stop working in weird ways.
I am not saying this is a 'can't be done'.. but this is not Free Work. The Fedora release engineering is pretty small, its physical and compute resources are pretty confined. You can only keep so many spinning plates going before they all come crashing down.
That covers infra and rel-eng, but there's another aspect that is being underrepresented here as well. Testing. If you take a single SRPM and build it for v1 and v2, ideally you'd test it on v1 and v2 hardware. If you only test one target, then you're basically just assuming it works[1] on the other. In that scenario, whatever target is preferred for testing is the defacto default.
I really don't think it's worth introducing more x86_64 targets across the entire package set. It doesn't scale on many levels. Pick your baseline according to whatever criteria you see best for the project and stick with it.
josh
[1] Quite frankly, I think the vast majority of the i686 builds fall into this "assume it works" category and it's a waste of resources to continue i686 on a large scale. I know "Steam games" is one of the leading use cases, and that seems like a great thing for a SIG to actually center around and continue focused package builds for. However, I've been ranting about i686 for years and have just accepted the fact that Fedora seems to want to pretend it's a worthwhile effort.
Once upon a time, Stephen Smoogen ssmoogen@redhat.com said:
I don't think Peter meant additional packages since with the i686 it didn't mean that. What it did mean was having to understand why two architectures might do things differently and why bugs might show up in one but not another.
In the i686 days, that was essentially THE second architecture for the bulk of packagers. Now we have Fedora on multiple really separate architectures, I don't feel the difference between x86_64-v1 and -v2 is such a big deal (as compared to x86_64 vs. aarch64 for example).
I'm not saying there's not a potential for issues exclusive to one x86_64 level, but it seems like a small area in comparison.
I think this needs to be decided for Fedora as soon as practical; while it'd be nice to keep the baseline at -v1 (I'd still like to see some more concrete "these CPU models are -v1" list), I also would prefer optimum performance e.g. from my VMs (and everything I have is at least -v2, most are -v3, and one is -v4). Even just bumping the baseline to -v2 won't enable some useful things like AVX2, so I think it makes sense to look more at enabling a multi-level approach (e.g. like the i386/i686 days with targeted packages built for multiple).
On Thu, Jun 20, 2024 at 8:49 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Smoogen ssmoogen@redhat.com said:
I don't think Peter meant additional packages since with the i686 it didn't mean that. What it did mean was having to understand why two architectures might do things differently and why bugs might show up in one but not another.
In the i686 days, that was essentially THE second architecture for the bulk of packagers. Now we have Fedora on multiple really separate architectures, I don't feel the difference between x86_64-v1 and -v2 is such a big deal (as compared to x86_64 vs. aarch64 for example).
I'm not saying there's not a potential for issues exclusive to one x86_64 level, but it seems like a small area in comparison.
I think this needs to be decided for Fedora as soon as practical; while it'd be nice to keep the baseline at -v1 (I'd still like to see some more concrete "these CPU models are -v1" list), I also would prefer optimum performance e.g. from my VMs (and everything I have is at least -v2, most are -v3, and one is -v4). Even just bumping the baseline to -v2 won't enable some useful things like AVX2, so I think it makes sense to look more at enabling a multi-level approach (e.g. like the i386/i686 days with targeted packages built for multiple).
Honestly, I'd like to pitch that we retarget Fedora at x86_64-v3 (yes, three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware. It could be something similar to Fedora ELN, where a subset of the main repo that might be useful on old hardware can be rebuilt (though unlike ELN, I suggest that this should be an entirely separate infrastructure not maintained by the Fedora Project). Such a project could then live or die based on willingness to maintain it and stop holding back Fedora as a whole.
On 20/06/2024 15:03, Stephen Gallagher wrote:
Honestly, I'd like to pitch that we retarget Fedora at x86_64-v3 (yes, three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware. It could be something similar to Fedora ELN, where a subset of the main repo that might be useful on old hardware can be rebuilt (though unlike ELN, I suggest that this should be an entirely separate infrastructure not maintained by the Fedora Project). Such a project could then live or die based on willingness to maintain it and stop holding back Fedora as a whole.
I definitely think going to v2 would be reasonable bit I tink forcing v3 might be a step too far.
While my desktop is v4 and my laptop is v3 I have two other machines at home which are only v2 one of which is only five years old having been built then to replace a 32 bit machine when Fedora was dropping 32 bit x86 support.
Having checked a couple of dozen machines at work that are running either F39 or F40 it's roughly a 50/50 split between ones which are v2 and ones with are v3. There are even three v1 machines there though one is redundant and the other two do really need replacing.
Tom
On Thu, Jun 20, 2024 at 03:24:18PM +0100, Tom Hughes via devel wrote:
On 20/06/2024 15:03, Stephen Gallagher wrote:
Honestly, I'd like to pitch that we retarget Fedora at x86_64-v3 (yes, three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware. It could be something similar to Fedora ELN, where a subset of the main repo that might be useful on old hardware can be rebuilt (though unlike ELN, I suggest that this should be an entirely separate infrastructure not maintained by the Fedora Project). Such a project could then live or die based on willingness to maintain it and stop holding back Fedora as a whole.
I definitely think going to v2 would be reasonable bit I tink forcing v3 might be a step too far.
If going to v3, compared v2....
For AMD, we would loose Opteron Gen4 and Opteron Gen5 models, but keep EPYC/Ryzen all generations.
For Intel, we would loose Denverton, IvyBridge, Nehalem, SandyBridge, Snowridge, Westmere, but keep Skylake, SapphireRapids, Icelake, Haswell, GraniteRapids, Cooperlake, Cascadelake, Broadwell.
Personally I'd say v2 is the better first step, as plenty of those generations we'd loose are still pretty relevant, even if not the state of the art shipping today.
With regards, Daniel
On Thu, Jun 20, 2024 at 10:49 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Jun 20, 2024 at 03:24:18PM +0100, Tom Hughes via devel wrote:
On 20/06/2024 15:03, Stephen Gallagher wrote:
Honestly, I'd like to pitch that we retarget Fedora at x86_64-v3 (yes, three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware. It could be something similar to Fedora ELN, where a subset of the main repo that might be useful on old hardware can be rebuilt (though unlike ELN, I suggest that this should be an entirely separate infrastructure not maintained by the Fedora Project). Such a project could then live or die based on willingness to maintain it and stop holding back Fedora as a whole.
I definitely think going to v2 would be reasonable bit I tink forcing v3 might be a step too far.
If going to v3, compared v2....
For AMD, we would loose Opteron Gen4 and Opteron Gen5 models, but keep EPYC/Ryzen all generations.
For Intel, we would loose Denverton, IvyBridge, Nehalem, SandyBridge, Snowridge, Westmere, but keep Skylake, SapphireRapids, Icelake, Haswell, GraniteRapids, Cooperlake, Cascadelake, Broadwell.
Personally I'd say v2 is the better first step, as plenty of those generations we'd loose are still pretty relevant, even if not the state of the art shipping today.
OK, I think I got my baselines mixed up. Sorry about that. I could have sworn what you were describing above was the effect of moving to -v4.
On Thu, Jun 20, 2024 at 10:51:56AM -0400, Stephen Gallagher wrote:
On Thu, Jun 20, 2024 at 10:49 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Jun 20, 2024 at 03:24:18PM +0100, Tom Hughes via devel wrote:
On 20/06/2024 15:03, Stephen Gallagher wrote:
Honestly, I'd like to pitch that we retarget Fedora at x86_64-v3 (yes, three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware. It could be something similar to Fedora ELN, where a subset of the main repo that might be useful on old hardware can be rebuilt (though unlike ELN, I suggest that this should be an entirely separate infrastructure not maintained by the Fedora Project). Such a project could then live or die based on willingness to maintain it and stop holding back Fedora as a whole.
I definitely think going to v2 would be reasonable bit I tink forcing v3 might be a step too far.
If going to v3, compared v2....
For AMD, we would loose Opteron Gen4 and Opteron Gen5 models, but keep EPYC/Ryzen all generations.
For Intel, we would loose Denverton, IvyBridge, Nehalem, SandyBridge, Snowridge, Westmere, but keep Skylake, SapphireRapids, Icelake, Haswell, GraniteRapids, Cooperlake, Cascadelake, Broadwell.
Personally I'd say v2 is the better first step, as plenty of those generations we'd loose are still pretty relevant, even if not the state of the art shipping today.
OK, I think I got my baselines mixed up. Sorry about that. I could have sworn what you were describing above was the effect of moving to -v4.
This table is specifically referring to named QEMU CPU models, but the names & impls match vendor generations closely enough that this is a decent crib-sheet to get an overview of the compatibility matrix:
https://www.qemu.org/docs/master/system/i386/cpu.html#abi-compatibility-leve...
With regards, Daniel
On Thursday 20 June 2024 15:48:55 BST Daniel P. Berrangé wrote:
On Thu, Jun 20, 2024 at 03:24:18PM +0100, Tom Hughes via devel wrote:
On 20/06/2024 15:03, Stephen Gallagher wrote:
Honestly, I'd like to pitch that we retarget Fedora at x86_64-v3 (yes, three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware. It could be something similar to Fedora ELN, where a subset of the main repo that might be useful on old hardware can be rebuilt (though unlike ELN, I suggest that this should be an entirely separate infrastructure not maintained by the Fedora Project). Such a project could then live or die based on willingness to maintain it and stop holding back Fedora as a whole.
I definitely think going to v2 would be reasonable bit I tink forcing v3 might be a step too far.
If going to v3, compared v2....
For AMD, we would loose Opteron Gen4 and Opteron Gen5 models, but keep EPYC/Ryzen all generations.
For Intel, we would loose Denverton, IvyBridge, Nehalem, SandyBridge, Snowridge, Westmere, but keep Skylake, SapphireRapids, Icelake, Haswell, GraniteRapids, Cooperlake, Cascadelake, Broadwell.
Note that this list is only accurate for Core and Xeon branded processors; these microarchitectures were also used for Celeron and Pentium branded processors that have stuck to x86-64v1.
Personally I'd say v2 is the better first step, as plenty of those generations we'd loose are still pretty relevant, even if not the state of the art shipping today.
For Pentium and Celeron branded processors, v2 also loses Skylake, Icelake, Haswell, Cometlake, Broadwell and others, even when their matching Core branded processors support x86-64v2 or x86-64v3.
That means that you lose all Pentium Silver processors, including the latest releases in that line, all Pentium Gold processors released before 2022, and all Celeron processors released before 2020.
Intel's use of ISA for product line segmentation has made this all a big mess :-(
With regards, Daniel --
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange |: :|
|: https://fstop138.berrange.com :| https://entangle-photo.org -o- |: https://www.instagram.com/dberrange :| -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 20/06/2024 16:34, Simon Farnsworth wrote:
For Pentium and Celeron branded processors, v2 also loses Skylake, Icelake, Haswell, Cometlake, Broadwell and others, even when their matching Core branded processors support x86-64v2 or x86-64v3.
That means that you lose all Pentium Silver processors, including the latest releases in that line, all Pentium Gold processors released before 2022, and all Celeron processors released before 2020.
I have a Celeron N3160 which is a 2016 processor bought by me in 2019 and that reports as v2.
Tom
On Thursday 20 June 2024 16:37:24 BST Tom Hughes wrote:
On 20/06/2024 16:34, Simon Farnsworth wrote:
For Pentium and Celeron branded processors, v2 also loses Skylake, Icelake, Haswell, Cometlake, Broadwell and others, even when their matching Core branded processors support x86-64v2 or x86-64v3.
That means that you lose all Pentium Silver processors, including the latest releases in that line, all Pentium Gold processors released before 2022, and all Celeron processors released before 2020.
I have a Celeron N3160 which is a 2016 processor bought by me in 2019 and that reports as v2.
Tom
Argh Intel. The Celeron 5305U from 2020 is a x86-64v1 CPU, because it's a cut- down variant of the Core i3-10110U (which is v2), but the Celeron N series is different yet again - the N3160 is v2, but the N4020 is v1 based on Intel's documentation.
So it's not even enough to go on branding + model numbers, since higher model numbers are sometimes a lower microarchitecture level than lower ones, even in the same branding.
Simon
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
On Thu, Jun 20, 2024 at 11:52 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
That's kind of my point. If people want something, but aren't willing to put any effort into it, why do they expect that the rest of the Fedora community will do it for them?
Stephen Gallagher wrote:
On Thu, Jun 20, 2024 at 11:52 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
That's kind of my point. If people want something, but aren't willing to put any effort into it, why do they expect that the rest of the Fedora community will do it for them?
I think most users would find it easiest to switch to Debian or some other distribution. There are probably rather few people with a great need to run Fedora and nothing else on their not-quite-high-end computers.
Björn Persson
On 6/20/24 2:27 PM, Björn Persson wrote:
Stephen Gallagher wrote:
On Thu, Jun 20, 2024 at 11:52 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
That's kind of my point. If people want something, but aren't willing to put any effort into it, why do they expect that the rest of the Fedora community will do it for them?
I think most users would find it easiest to switch to Debian or some other distribution. There are probably rather few people with a great need to run Fedora and nothing else on their not-quite-high-end computers.
I think most users don't have the technical know-how, or the time to dedicate to taking care of packages. If they are already stuck with a not-really-new computer, chances are they are not someone with the privilege of hours upon hours of free time to handle this stuff.
And its a reasonable assumption that even those who fit that bill might just find it easier to jump ship if they don't have a need for fedora, like you said, Björn. After all, I choose fedora on my personal machine because there are a lot of people paid to maintain my stuff working. If that stops being true, I'd probably just stop using fedora.
On Thursday, 20 June 2024 at 19:50, Guinevere Larsen wrote:
On 6/20/24 2:27 PM, Björn Persson wrote:
Stephen Gallagher wrote:
On Thu, Jun 20, 2024 at 11:52 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
That's kind of my point. If people want something, but aren't willing to put any effort into it, why do they expect that the rest of the Fedora community will do it for them?
I think most users would find it easiest to switch to Debian or some other distribution. There are probably rather few people with a great need to run Fedora and nothing else on their not-quite-high-end computers.
I think most users don't have the technical know-how, or the time to dedicate to taking care of packages. If they are already stuck with a not-really-new computer, chances are they are not someone with the privilege of hours upon hours of free time to handle this stuff.
I could pick up a few additional packages to maintain for -v1, but not the whole infrastructure. I just don't have that much time to spare. And for me, it's not that I'm "stuck with" these old machines. I just see no reason to replace them with something newer just because they're old. They still work fine for their intended purpose and the only reason to spend money to replace them so far would be if their mainboards or CPUs died (because they're soldered). I've already replaced their storage, RAM, PSUs and coolers to keep them running at least once. I think such approach is better (both economically and environmentally) compared to throwing them away and buying completely new hardware.
Regards, Dominik
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
On Thu, Jun 20, 2024 at 11:52 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
That's kind of my point. If people want something, but aren't willing to put any effort into it, why do they expect that the rest of the Fedora community will do it for them?
But changing the baseline to -v3 throws out a LOT of hardware from Fedora compatibility. And the overall level of effort required to support multiple builds for a targetted set of packages where it can help is not that high, compared to the level of effort required to create and support rebuilding a wide range of packages and producing a release.
On Thu, Jun 20, 2024 at 3:52 PM Chris Adams linux@cmadams.net wrote:
Once upon a time, Stephen Gallagher sgallagh@redhat.com said:
three) and recommend creation of a Fedora "Hardware Life Extension" Remix that can provide rebuilds of (a subset of) Fedora that they want to run on ancient hardware.
TBH I feel that approach would be doomed to the same failure as the attempts to extend Fedora life-cycles. There's enough people that would want it, but not necessarily the critical mass needed to do the work to make it happen.
Which may be OK, and therefore a success, in that the community has spoken by their lack of interest in actually doing (or making an arrangement with someone else for the doing). To give those who might be interested in creating a critical mass one should provide an extended period of time for the change (I would suggest a year, so a F43 change proposal for -v2?) and see if those interested can achieve criticality.
On Wed, Jun 19, 2024 at 8:39 AM Neal Gompa ngompa13@gmail.com wrote:
- I suspect more of the hardware that don't support -v2 have failed
out of use naturally
Due to product line feature differentiation there are more recent -v1 hardware than the aforementioned roughly 2008 date, but the one pre-nehalem -v1 system I still run as an appliance (a core 2 duo) fails to boot about 10% of the time due to power issues due to capacitor disease (and I don't feel like re-capping the motherboard and power supply for a 2008 era system), so it will finally die or be replaced soon enough. I would not be surprised if other such aged desktops and laptops are not also near hardware end of life due to other system component lifetime issues.
I will also note that since that -v1 desktop/laptop systems of the legacy architecture do not support EPT, even native architecture virtualization is extremely poor performance wise, which is why I don't care about qemu on -v1 systems (and don't have it installed on my -v1 system). If qemu is compiled as -v2+ only I would never notice the difference on that system as I don't even install it on that system (although I would, presumably, see the performance enhancements on my -v2+ desktop).
- RPM now lets us "tell the truth" about what x86_64 sublevel we expect
And if compiling qemu with -v2 has a performance benefit, that seems to be a positive path forward.
On Friday, 21 June 2024 at 01:47, Gary Buhrmaster wrote:
On Wed, Jun 19, 2024 at 8:39 AM Neal Gompa ngompa13@gmail.com wrote:
[...]
I will also note that since that -v1 desktop/laptop systems of the legacy architecture do not support EPT,
If you mean Extended Page Table here, then Wikipedia[1] says it was introduced in 2010 with the Westmere[2] micro-architecture.
[1] https://en.wikipedia.org/wiki/Second_Level_Address_Translation#EPT [2] https://en.wikipedia.org/wiki/Westmere_(microarchitecture)
I don't know any way to tell if my Cedar View Atom D2550 CPU from 2012 supports it or not. Not that I'd want to run QEMU on it.
even native architecture virtualization is extremely poor performance wise, which is why I don't care about qemu on -v1 systems (and don't have it installed on my -v1 system). If qemu is compiled as -v2+ only I would never notice the difference on that system as I don't even install it on that system (although I would, presumably, see the performance enhancements on my -v2+ desktop).
+1
- RPM now lets us "tell the truth" about what x86_64 sublevel we expect
And if compiling qemu with -v2 has a performance benefit, that seems to be a positive path forward.
+1. Just build it with --target x86_64_v2 and announce it widely. -1 to building all Fedora as v2.
Regards, Dominik
On Fri, Jun 21, 2024 at 11:51 AM Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
If you mean Extended Page Table here
Yes, I used a shorthand term, since I am apparently too steeped in the architectural details.
I don't know any way to tell if my Cedar View Atom D2550 CPU from 2012 supports it or not. Not that I'd want to run QEMU on it.
I don't think it does, but if you really want to know, you can always go to Intel's processor model site (ark), and look for line of the form: "Intel® VT-x with Extended Page Tables (EPT)" for your processor (if it is not there, the answer is no, if it is there you have it), or just look at /proc/cpuinfo and look for "ept" in the vmx flags.
+1. Just build it with --target x86_64_v2 and announce it widely
That should probably be the most reasonable path forward (allowing those that want to step up to propose and build their qemu-slower-is-better package to do so).
Please note that switching to x86_64-v2 does *not* give you access to AVX2 or even AVX. It stops at SSE4.2 with POPCNT.
AVX2 means x86_64-v3, which both excludes a lot more hardware and allows (in some cases) much more significant performance gains.
On 6/19/24 3:29 AM, Vitaly Zaitsev via devel wrote:
On 19/06/2024 09:13, Daniel P. Berrangé wrote:
If Fedora cares about optimal performance it should just declare we're going to stop being held back by compat with ancient hardware and use -v2 baseline for everything, but obviously that's been rejected previously.
Maybe it's a good time for the Fedora 41 System-Wide change proposal? Switching to AVX2 will give significant performance boost for many applications.
On Wed, Jun 12, 2024 at 4:00 PM Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Build the package as x86_64_v2 instead of x86_64.
Not lying about the architecture will ensure that we don't bypass RPM's own compatible architecture check.
Dne 12. 06. 24 v 19:14 Neal Gompa napsal(a):
On Wed, Jun 12, 2024 at 4:00 PM Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Jun 12, 2024 at 9:55 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 09:51:34AM -0400, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline?
I've not tested, but I would expect it to crash attempting to execute an illegal instruction
OK, that's a situation that will lead to annoying and unresolvable bug reports. Would it be possible to put something in place that would check processor capabilities early in execution before hitting any of the affected instructions?
Build the package as x86_64_v2 instead of x86_64.
Not lying about the architecture will ensure that we don't bypass RPM's own compatible architecture check.
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
Vít
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
1. Resource limitations in infrastructure hardware. You are going to add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren
Dne 21. 06. 24 v 18:27 Stephen Smoogen napsal(a):
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
- Resource limitations in infrastructure hardware. You are going to
add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not.
Yes, understandably. But are there technical limitations?
BTW I guess that e.g. some sort of inheritance reduce the amount of needed HW.
Vít
On Mon, 24 Jun 2024 at 11:21, Vít Ondruch vondruch@redhat.com wrote:
Dne 21. 06. 24 v 18:27 Stephen Smoogen napsal(a):
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
- Resource limitations in infrastructure hardware. You are going to
add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not.
Yes, understandably. But are there technical limitations?
No, and you could argue to get rid of i686
BTW I guess that e.g. some sort of inheritance reduce the amount of needed HW.
Well that brings other problems, see i686 as an example here, but there's a large percentage of noarch packages in the distro so it's not a full 1:1 package:arch increment.
P
Dne 24. 06. 24 v 20:03 Peter Robinson napsal(a):
On Mon, 24 Jun 2024 at 11:21, Vít Ondruch vondruch@redhat.com wrote:
Dne 21. 06. 24 v 18:27 Stephen Smoogen napsal(a):
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
- Resource limitations in infrastructure hardware. You are going to
add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not.
Yes, understandably. But are there technical limitations?
No, and you could argue to get rid of i686
BTW I guess that e.g. some sort of inheritance reduce the amount of needed HW.
Well that brings other problems, see i686 as an example here, but there's a large percentage of noarch packages in the distro so it's not a full 1:1 package:arch increment.
I was not speaking about noarch. I assume that you can intermix x86_64_v1 code with x86_64_v3 code on x86_64_v3 capable HW. Is that correct? IOW there could be build just subset of packages for x86_64_v3 and rest would be inherited.
Vít
P
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Tue, 25 Jun 2024 11:57:49 +0200 Vít Ondruch vondruch@redhat.com wrote:
Dne 24. 06. 24 v 20:03 Peter Robinson napsal(a):
On Mon, 24 Jun 2024 at 11:21, Vít Ondruch vondruch@redhat.com wrote:
Dne 21. 06. 24 v 18:27 Stephen Smoogen napsal(a):
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
- Resource limitations in infrastructure hardware. You are going to
add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not.
Yes, understandably. But are there technical limitations?
No, and you could argue to get rid of i686
BTW I guess that e.g. some sort of inheritance reduce the amount of needed HW.
Well that brings other problems, see i686 as an example here, but there's a large percentage of noarch packages in the distro so it's not a full 1:1 package:arch increment.
I was not speaking about noarch. I assume that you can intermix x86_64_v1 code with x86_64_v3 code on x86_64_v3 capable HW. Is that correct? IOW there could be build just subset of packages for x86_64_v3 and rest would be inherited.
aka similar to what was done with i386 and i686 back in the day, and then with ppc64 + ppc64p7 for a short period of time.
Dan
Dne 25. 06. 24 v 12:10 Dan Horák napsal(a):
On Tue, 25 Jun 2024 11:57:49 +0200 Vít Ondruch vondruch@redhat.com wrote:
Dne 24. 06. 24 v 20:03 Peter Robinson napsal(a):
On Mon, 24 Jun 2024 at 11:21, Vít Ondruch vondruch@redhat.com wrote:
Dne 21. 06. 24 v 18:27 Stephen Smoogen napsal(a):
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
- Resource limitations in infrastructure hardware. You are going to
add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not.
Yes, understandably. But are there technical limitations?
No, and you could argue to get rid of i686
BTW I guess that e.g. some sort of inheritance reduce the amount of needed HW.
Well that brings other problems, see i686 as an example here, but there's a large percentage of noarch packages in the distro so it's not a full 1:1 package:arch increment.
I was not speaking about noarch. I assume that you can intermix x86_64_v1 code with x86_64_v3 code on x86_64_v3 capable HW. Is that correct? IOW there could be build just subset of packages for x86_64_v3 and rest would be inherited.
aka similar to what was done with i386 and i686 back in the day, and then with ppc64 + ppc64p7 for a short period of time.
That predates my days here 🤷 But yes, that sounds as similar cases, so we might have some rusty know how.
Vít
Dan-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Jun 21, 2024 at 12:27:25PM -0400, Stephen Smoogen wrote:
On Fri, 21 Jun 2024 at 07:27, Vít Ondruch vondruch@redhat.com wrote:
So what is the reason to not treat x86_64_v2 as different arch then x86_64_v{1,3}. Why we keep having this discussion instead of fire one more build? Users would need to choose v1 / v2 / v3 ISO but what else?
I can think of three problems which would need to be dealt with
- Resource limitations in infrastructure hardware. You are going to
add to the amount of builds on 1 set of hardware which is already doing x86_64 and i686. You are going to add to the storage issues that Fedora Infrastructure has to juggle on the maximum 100TB koji partition (with 90TB causing some amount of degradation) due to extra packages and composes. 2. Resource limitations in infrastructure staff. Fedora Infra is doing more with less and each additional architecture and focus increases that load. 3. Resource limitations on packagers. Packagers will need to add yet another bug set to cover and determine "is it only on VX" or not.
Another reason: is it actually useful at all?
Benchmarking so far hasn't been mention in this thread. But it was discussed fairly extensively in previous interations on fedora-devel, and the results were … not impressive.
The first consideration is that many packages already employ multiple versions of functions and select the optimal version _at runtime_. In that case, there is no "baseline architecture", the program works on all µarchitectures, just faster or slower. This includes various BLAS libraries, but also very importantly glibc itself with IFUNCS, some compression libraries, etc. This means that for many programs the heavy number crunching is already optimized, and raising the µarchitecture level will have negligible effect on performance.
The second consideration is that many packages are not CPU-bound at all, or don't perform the kind of processing where AVX and other instructions make a difference.
So overall, there _might_ be some programs which would benefit from higher µarchitecture requirements. Before starting a huge effort to recompile the distro, it'd be prudent to do some local compilations and benchmarking.
But OK, let's jump forward and we identified a subset of Fedora that'd benefit. For those programs, a runtime approach based on IFUNCS or equivalent is the most powerful. Only the hot paths need to be targeted, delivering the same benefits as raising the baseline for the whole program, while being transparent to the user. Also, this approach can be more flexible, because it's OK to have many different variants, rather than just targeting four µarchitecture levels. It also requires much less resources, because we don't deliver multiple versions of the package, but instead a few versions of the hot functions, all in the same binary.
Only for programs where there is potential benefit, but we cannot do IFUNCS, compiling with a higher baseline is a useful approach.
Overall, I think that we _should_ have more software optimized for newer CPUs, but the solutions should be targeted at the right packages and the right parts of the code. Just compiling everything multiple times is IMO a waste of resources.
Zbyszek
This never made it to the packaging guidelines, but FESCo made a relevant decision a few years ago:
Libraries packaged in Fedora may require ISA extensions, however any packaged application must not crash on any officially supported architecture, either by providing a generic fallback implementation OR by cleanly exiting when the requisite hardware support is unavailable.
https://pagure.io/packaging-committee/issue/1044
On 6/12/24 9:51 AM, Stephen Gallagher wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangéberrange@redhat.com wrote:
There have been various change proposals & associated mailing list threads over the years about the possibility of moving Fedora compiler toolchain to build with a newer x86_64 baseline ABI, which have ended up rejected, with some quite strong negative feedback.
Regardless of the Fedora general policy, however, individual packages may have forced a particular x86_64 baseline ABI through their own CFLAGS defined by the upstream project.
The context is that QEMU has recently merged changes upstream that force use of the x86-64-v2 baseline for QEMU, in order get more efficient code in the TCG emulator. The changes were made in QEMU's global CFLAGS so this will affect all usage of QEMU, whether KVM, or TCG, for both VMs and user space emulation (the latter used by podman for non-native containers)
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Thus I'm wondering whether Fedora has any policy or guidance on handling such a situation both in general, and more specifically for "critical path" packages, if that difference is relevant ? The packaging guidelines aren't especially explicit about this situation, unless I've missed something beyond the "compiler flags" and "architecture support" sections.
Absent a project-wide decision to move to the newer baseline, I think the best approach we can take would be to find some way to communicate to the user that the software isn't usable. In the case of Qemu, does the application report an error or crash if it's run on hardware without the requisite baseline? -- _______________________________________________ devel mailing list --devel@lists.fedoraproject.org To unsubscribe send an email todevel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
On Wed, Jun 12, 2024 at 10:05:13AM -0400, Ben Beasley wrote:
This never made it to the packaging guidelines, but FESCo made a relevant decision a few years ago:
Libraries packaged in Fedora may require ISA extensions, however any packaged application must not crash on any officially supported architecture, either by providing a generic fallback implementation OR by cleanly exiting when the requisite hardware support is unavailable.
Wow, that is incredibly *loosely* written. That allows for glibc to build itself with '-march=x86-64-v2', or for systemd to build itself with '-march=x86_64-v2 -mneeded', at the discretion of the maintainer. This would effectively reverse the entire decision to reject the x86-64-v2 baseline feature proposal for.
Surely that is not what they meant to permit ? Surely this exception was only intended to be scoped more narrowly ? Perhaps to non-critical path libraries, or only packages outside the default release media ?
With regards, Daniel
On Wed, Jun 12, 2024 at 5:47 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Jun 12, 2024 at 10:05:13AM -0400, Ben Beasley wrote:
This never made it to the packaging guidelines, but FESCo made a relevant decision a few years ago:
Libraries packaged in Fedora may require ISA extensions, however any packaged application must not crash on any officially supported architecture, either by providing a generic fallback implementation OR by cleanly exiting when the requisite hardware support is unavailable.
Wow, that is incredibly *loosely* written. That allows for glibc to build itself with '-march=x86-64-v2', or for systemd to build itself with '-march=x86_64-v2 -mneeded', at the discretion of the maintainer. This would effectively reverse the entire decision to reject the x86-64-v2 baseline feature proposal for.
Surely that is not what they meant to permit ? Surely this exception was only intended to be scoped more narrowly ? Perhaps to non-critical path libraries, or only packages outside the default release media ?
It has been some time since I was part of this discussion (IIRC both in FPC and FESCo at the time), and yes, I don't think your interpretation matches our intentions, even though it *could* be read that way. IIRC this was mostly about adding *new* packages that have higher ISA requirements than x86_64-v1 (the current Fedora baseline). I don't think we considered that existing packages might want to push their requirements, too. But as I said, it's been some time, so I might mis-remember.
So in the situation with QEMU, IMO it would be best to do detection of AVX2 (or whatever x86_64-v2 features are required) at *runtime* and provide a slower fallback code path. Alternatively, patch the build system to build without the assumption that x86_64-v2 instructions are available.
Fabio
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
The context is that QEMU has recently merged changes upstream that force use of the x86-64-v2 baseline for QEMU, in order get more efficient code in the TCG emulator. The changes were made in QEMU's global CFLAGS so this will affect all usage of QEMU, whether KVM, or TCG, for both VMs and user space emulation (the latter used by podman for non-native containers)
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Could the answer be "we patch the build to not use the changed FLAGS"? (Perhaps also creating a qemu-zoomzoom package that's the same as the regular qemu but with the upstream CFLAGS that people who can take advantage can install instead)
Neither "Functional" nor "eFficient" are in the Fedora Foundations, but in general, I think we should prefer the former over the latter. It's better for the project overall to be a little less efficient than it could be than to surprise people with "a key package no longer works on your hardware". It would be better to explicitly move the entire distro to a higher baseline than to have some packages work and some not. (I'm not saying now is the time to make that decision, but it's clear that we should at least have an idea of the conditions where the switch becomes the right choice)
On Wed, Jun 12, 2024 at 3:50 PM Ben Cotton bcotton@fedoraproject.org wrote:
Neither "Functional" nor "eFficient" are in the Fedora Foundations, but in general, I think we should prefer the former over the latter. It's better for the project overall to be a little less efficient than it could be than to surprise people with "a key package no longer works on your hardware".
As I recall from a quick scan of the commits (and I am not going to claim I looked closely), some of the later commits are not build time choices/checks, but now just presume x86_64-v2. Reverting the removals (and maintaining) those commits is likely not a viable option for Fedora packaging.
While I would prefer that upstream had added in run time checks (and make their best choices) rather than build time/code checks, I do understand that upstreams are not likely to resource that for use cases that are considered uncommon (x86_64-v1 systems running qemu).
On Wed, Jun 12, 2024 at 11:50:00AM -0400, Ben Cotton wrote:
On Wed, Jun 12, 2024 at 8:41 AM Daniel P. Berrangé berrange@redhat.com wrote:
The context is that QEMU has recently merged changes upstream that force use of the x86-64-v2 baseline for QEMU, in order get more efficient code in the TCG emulator. The changes were made in QEMU's global CFLAGS so this will affect all usage of QEMU, whether KVM, or TCG, for both VMs and user space emulation (the latter used by podman for non-native containers)
IOW, if [when] we rebase Fedora to the next QEMU upstream release, users with older x86_64 hardware would likely be unable to run QEMU, from F41 onwards, unless some TBD action is taken.
Could the answer be "we patch the build to not use the changed FLAGS"? (Perhaps also creating a qemu-zoomzoom package that's the same as the regular qemu but with the upstream CFLAGS that people who can take advantage can install instead)
It isn't as simple as changing the CFLAGS. QEMU used to check for the CPU feature at startup, set a flag, and then later use that flag to choose different codepaths, but this logic was removed. Avoiding the flag check in hot-paths makes a perf difference.
So we would have to revert the whole patch series in Fedora. That's doable today, but I'm not confident carrying a local only revert over time.
With regards, Daniel
On Wed, Jun 12, 2024 at 05:35:24PM +0100, Daniel P. Berrangé wrote:
It isn't as simple as changing the CFLAGS. QEMU used to check for the CPU feature at startup, set a flag, and then later use that flag to choose different codepaths, but this logic was removed. Avoiding the flag check in hot-paths makes a perf difference.
Can't QEMU upstream be convinced to make it configurable, perhaps with default requiring SSE4 and when configured/built to support even older HW take the performance hit to support it?
Jakub
On Wed, Jun 12, 2024 at 12:35 PM Daniel P. Berrangé berrange@redhat.com wrote:
It isn't as simple as changing the CFLAGS. QEMU used to check for the CPU feature at startup, set a flag, and then later use that flag to choose different codepaths, but this logic was removed. Avoiding the flag check in hot-paths makes a perf difference.
I was hoping you wouldn't say that. :-)
So we would have to revert the whole patch series in Fedora. That's doable today, but I'm not confident carrying a local only revert over time.
Yeah, maintaining that long-term seems like a bad idea. It might not be a bad idea in the short term as a bridge to keep newer QEMU in the repos while we decide as a project what to do long term. I recognize, of course, that is easy for me to say this because I'm not the one who will have to do the work and carry the maintenance burden. I trust your judgement on if that's workable at all and for how long. But it would at least buy us some time so that we don't end up with the "surprise, you can't use this release on your hardware if you want to use QEMU!" situation.
On Wed, Jun 12, 2024 at 6:08 PM Ben Cotton bcotton@fedoraproject.org wrote:
.... But it would at least buy us some time so that we don't end up with the "surprise, you can't use this release on your hardware if you want to use QEMU!" situation.
Since we don't have complete instrumentation, we really don't know how many people would be affected, but it may not be all that many.
FD: I do have a x86_664-v1 system which runs as an appliance for a specific app (it is an old Core 2 Duo which is on it's last legs), but have never run (and in fact do not have installed) any qemu functionality. I suspect it would not be a reasonable platform for qemu usage in any case.
On 6/12/24 14:07, Ben Cotton wrote:
Yeah, maintaining that long-term seems like a bad idea. [...] But it would at least buy us some time so that we don't end up with the "surprise, you can't use this release on your hardware if you want to use QEMU!" situation.
The root cause seems to be the QEMU upstream's switch to x86_64_v2 ABI without a backwards compatiblity. If the software is not compatible with a given hardware, it just should not install; Fedora should not be stuck at an obsolete version for everyone, or expected to backport alternatives, or be blamed for the breakage on old systems.
I am an old man, but even I understand that sometimes backwards compatibility has to go if there are tangible benefits to breaking changes and no practical workarounds, whether it's 32-bit-only support, or X11, or QEMU; I accept it even if I am personally affected.
To prevent the surprise, I wonder if Fedora could detect and warn for old hardware :
"The future upstream changes to QEMU will require x86_64_v2 hardware, making it incompatible with your system"
before the upstream changes arrive. After that, I like Neal's suggestion of declaring it via RPM attributes, and making it non-installable/non-upgradeable on old architectures. I guess people will have to decide then if they uninstall or lock out updates with 'versionlock' or '--skip-broken'. I understand that locking updates would still eventually result in nonfunctional QEMU because of diverging dependencies.
On Wed, Jun 12, 2024 at 9:15 PM Przemek Klosowski via devel devel@lists.fedoraproject.org wrote:
On 6/12/24 14:07, Ben Cotton wrote:
Yeah, maintaining that long-term seems like a bad idea. [...] But it would at least buy us some time so that we don't end up with the "surprise, you can't use this release on your hardware if you want to use QEMU!" situation.
The root cause seems to be the QEMU upstream's switch to x86_64_v2 ABI without a backwards compatiblity. If the software is not compatible with a given hardware, it just should not install; Fedora should not be stuck at an obsolete version for everyone, or expected to backport alternatives, or be blamed for the breakage on old systems.
I am an old man, but even I understand that sometimes backwards compatibility has to go if there are tangible benefits to breaking changes and no practical workarounds, whether it's 32-bit-only support, or X11, or QEMU; I accept it even if I am personally affected.
To prevent the surprise, I wonder if Fedora could detect and warn for old hardware :
"The future upstream changes to QEMU will require x86_64_v2hardware, making it incompatible with your system"
before the upstream changes arrive. After that, I like Neal's suggestion of declaring it via RPM attributes, and making it non-installable/non-upgradeable on old architectures. I guess people will have to decide then if they uninstall or lock out updates with 'versionlock' or '--skip-broken'. I understand that locking updates would still eventually result in nonfunctional QEMU because of diverging dependencies.
This is supported since RPM 4.19, so we should just do it.
We may also want to start having a conversation about moving to x86_64-v2 RPM arch for x86_64 across the board if we're going to start encountering stuff like this.
-- 真実はいつも一つ！/ Always, there's only one truth!
Once upon a time, Neal Gompa ngompa13@gmail.com said:
We may also want to start having a conversation about moving to x86_64-v2 RPM arch for x86_64 across the board if we're going to start encountering stuff like this.
Is there a good decoder ring for which CPUs are which level? Looking at all my systems, I think they're all at least v2 (I think only one v4 so far). For example IIRC the last time this came up, there were still Atom variants being sold that were not v2 compatible.
Chris Adams wrote:
Once upon a time, Neal Gompa ngompa13@gmail.com said:
We may also want to start having a conversation about moving to x86_64-v2 RPM arch for x86_64 across the board if we're going to start encountering stuff like this.
Is there a good decoder ring for which CPUs are which level? Looking at all my systems, I think they're all at least v2 (I think only one v4 so far). For example IIRC the last time this came up, there were still Atom variants being sold that were not v2 compatible.
Running /lib64/ld-linux-x86-64.so.2 --help is one way to see what's supported. That's simpler than parsing things out of /proc/cpuinfo, for me anyway.
Daniel P. Berrangé wrote:
It isn't as simple as changing the CFLAGS. QEMU used to check for the CPU feature at startup, set a flag, and then later use that flag to choose different codepaths, but this logic was removed. Avoiding the flag check in hot-paths makes a perf difference.
So we would have to revert the whole patch series in Fedora. That's doable today, but I'm not confident carrying a local only revert over time.
But it is the ONLY approach that is compatible with Fedora policies, and as such should be required. ESPECIALLY for a package like QEMU that many people are using.
If forward-porting the reverts stops being doable, then we will have to keep the old version of QEMU or fork the project.
Kevin Kofler
On Thu, Jun 13, 2024 at 1:35 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
But it is the ONLY approach that is compatible with Fedora policies, and as such should be required. ESPECIALLY for a package like QEMU that many people are using.
Please provide your audited (by a 3rd party) data that shows that MANY (i.e. significant percentage of) people on x86_64-v1 users have a need for qemu.
Those without actual data are simply offering an opinion, and should be dismissed without further consideration.
We will all wait for your 3rd party validation of your data.
If forward-porting the reverts stops being doable, then we will have to keep the old version of QEMU or fork the project.
So, you (and a list of specific volunteers) are committed for those efforts and ready to do so?
On Wed, Jun 12, 2024 at 10:51 PM Gary Buhrmaster gary.buhrmaster@gmail.com wrote:
On Thu, Jun 13, 2024 at 1:35 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
But it is the ONLY approach that is compatible with Fedora policies, and as such should be required. ESPECIALLY for a package like QEMU that many people are using.
Please provide your audited (by a 3rd party) data that shows that MANY (i.e. significant percentage of) people on x86_64-v1 users have a need for qemu.
Those without actual data are simply offering an opinion, and should be dismissed without further consideration.
We will all wait for your 3rd party validation of your data.
This isn't a constructive way to have the conversation. As you pointed out in an earlier message, no one has this data, so all of us are left to take our best guess at it. Let's be respectful to each other about it.
For myself, I think it's reasonable to conclude there's a non-trivial amount of people using QEMU on that hardware in some fashion. Much of that is probably from podman as opposed to running large virtualized environments at this point, but the podman use case is important for a lot of people.
From the previous times the baseline conversation has happened, I suspect that I am more open to raising it than Kevin is, but I agree with his general direction here. Absent reliable data, I tend toward a conservative approach to dropping hardware support. But there's no easy answer here.
On Thu, Jun 13, 2024 at 11:44 AM Ben Cotton bcotton@fedoraproject.org wrote:
On Wed, Jun 12, 2024 at 10:51 PM Gary Buhrmaster gary.buhrmaster@gmail.com wrote:
On Thu, Jun 13, 2024 at 1:35 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
But it is the ONLY approach that is compatible with Fedora policies, and as such should be required. ESPECIALLY for a package like QEMU that many people are using.
Please provide your audited (by a 3rd party) data that shows that MANY (i.e. significant percentage of) people on x86_64-v1 users have a need for qemu.
Those without actual data are simply offering an opinion, and should be dismissed without further consideration.
We will all wait for your 3rd party validation of your data.
This isn't a constructive way to have the conversation. As you pointed out in an earlier message, no one has this data, so all of us are left to take our best guess at it. Let's be respectful to each other about it.
For myself, I think it's reasonable to conclude there's a non-trivial amount of people using QEMU on that hardware in some fashion. Much of that is probably from podman as opposed to running large virtualized environments at this point, but the podman use case is important for a lot of people.
From the previous times the baseline conversation has happened, I suspect that I am more open to raising it than Kevin is, but I agree with his general direction here. Absent reliable data, I tend toward a conservative approach to dropping hardware support. But there's no easy answer here.
If QEMU upstream is *really* set on unconditionally raising their required x86_64 ISA version, and cannot be convinced that this course of action would be painful for distributions like Fedora, then I think we should at least have a self-contained change for the QEMU update that introduces this change.
Fabio
Once upon a time, Ben Cotton bcotton@fedoraproject.org said:
For myself, I think it's reasonable to conclude there's a non-trivial amount of people using QEMU on that hardware in some fashion. Much of that is probably from podman as opposed to running large virtualized environments at this point, but the podman use case is important for a lot of people.
Without knowing which CPU models are actually affected, I don't think it's reasonable to draw any conclusions.
For personal anecdote: my oldest system I still use for anything is a 7-year-old Intel NUC (i5-7260U), and it's actually v3. My lowest-end system is a router (not running Fedora); it's a Pentium N6005 and appears to be v2 (it doesn't use glibc, but I found an awk script that seems to report it).
So yeah, it may be time to say "okay, we can't run QEMU on the old systems anymore". If upstream QEMU is going to make their software v2+ only, that's really the only reasonable conclusion; it is not reasonable to demand Fedora packagers maintain a patchset or a fork to revert that change.
But again, knowing what CPU models are what level would bring more clarity to the effect. I don't know how to look at say Intel Ark (does AMD have a similar site?) and determine "this CPU is v3". IIRC when the baseline came up before, the only current or recent CPUs that weren't v2+ were some Intel Atoms.
On 13 Jun 2024, at 13:58, Chris Adams linux@cmadams.net wrote:
Without knowing which CPU models are actually affected, I don't think it's reasonable to draw any conclusions.
Where is the v2, v3 etc levels specified?
I've not been able to web search and find a spec.
Is this called an x64-64 micro-architecture?
Armed with a spec it might be possible to search on ark.intel.com http://ark.intel.com/.
Barry
On Thu, Jun 13, 2024 at 7:50 AM Barry Scott barry@barrys-emacs.org wrote:
Where is the v2, v3 etc levels specified?
I've not been able to web search and find a spec.
Is this called an x64-64 micro-architecture?
Armed with a spec it might be possible to search on ark.intel.com.
Wikipedia has some information: https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels
On 13 Jun 2024, at 14:54, Jerry James loganjerry@gmail.com wrote:
Wikipedia has some information: https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels
After asking I did some more web searches and come up with
In this article: https://medium.com/@BetterIsHeather/x86-64-levels-944e92cd6d83 it says that th is the source of truth: https://gitlab.com/x86-psABIs/x86-64-ABI/-/blob/master/x86-64-ABI/low-level-... As defined my Intel, AMD, RedHat and SUSE.
Which agrees with Wikipedia article that cites that .tex file.
Barry
On Thu, 13 Jun 2024 at 13:58, Chris Adams linux@cmadams.net wrote:
Once upon a time, Ben Cotton bcotton@fedoraproject.org said:
For myself, I think it's reasonable to conclude there's a non-trivial amount of people using QEMU on that hardware in some fashion. Much of that is probably from podman as opposed to running large virtualized environments at this point, but the podman use case is important for a lot of people.
Without knowing which CPU models are actually affected, I don't think it's reasonable to draw any conclusions.
For personal anecdote: my oldest system I still use for anything is a 7-year-old Intel NUC (i5-7260U), and it's actually v3. My lowest-end system is a router (not running Fedora); it's a Pentium N6005 and appears to be v2 (it doesn't use glibc, but I found an awk script that seems to report it).
So yeah, it may be time to say "okay, we can't run QEMU on the old systems anymore". If upstream QEMU is going to make their software v2+ only, that's really the only reasonable conclusion; it is not reasonable to demand Fedora packagers maintain a patchset or a fork to revert that change.
But again, knowing what CPU models are what level would bring more clarity to the effect. I don't know how to look at say Intel Ark (does AMD have a similar site?) and determine "this CPU is v3". IIRC when the baseline came up before, the only current or recent CPUs that weren't v2+ were some Intel Atoms.
I've looked at a few Atom, Apollo Lake, and they're v2 (I have a couple of others to boot to check), most Atom are not v3. I have an AMD platform (AMD Turion(tm) II Neo N40L Dual-Core Processor) that is v1, but it's certainly not something I run virt on any longer.
Ben Cotton wrote:
For myself, I think it's reasonable to conclude there's a non-trivial amount of people using QEMU on that hardware in some fashion. Much of that is probably from podman as opposed to running large virtualized environments at this point, but the podman use case is important for a lot of people.
If this will affect containers, then I'm starting to worry that I might be affected. I thought I'd be fine as the computers I run virtual machines on are x86-64-v2 and -v3, but another computer that still works fine is -v1. As popular as containers are, who knows when some program I use will switch to being distributed only as a container, and thus become unusable on the -v1 box?
Björn Persson
Once upon a time, Björn Persson <Bjorn@rombobjörn.se> said:
If this will affect containers, then I'm starting to worry that I might be affected. I thought I'd be fine as the computers I run virtual machines on are x86-64-v2 and -v3, but another computer that still works fine is -v1. As popular as containers are, who knows when some program I use will switch to being distributed only as a container, and thus become unusable on the -v1 box?
Are you using containers for a different architecture (e.g. running aarch64 containers on an x86_64 host)? AFAIK that's when podman will use QEMU. If not, you would not be affected.
Chris Adams wrote:
Once upon a time, Björn Persson <Bjorn@rombobjörn.se> said:
If this will affect containers, then I'm starting to worry that I might be affected. I thought I'd be fine as the computers I run virtual machines on are x86-64-v2 and -v3, but another computer that still works fine is -v1. As popular as containers are, who knows when some program I use will switch to being distributed only as a container, and thus become unusable on the -v1 box?
Are you using containers for a different architecture (e.g. running aarch64 containers on an x86_64 host)? AFAIK that's when podman will use QEMU. If not, you would not be affected.
No I'm not, and if some upstream would decide that their program must run as a container, I'm sure they would at least allow an x86-64 container for the foreseeable future. If a need for emulation would arise, I'd do that on a newer computer. It seems I personally will be fine then.
Björn Persson
On 6/13/24 10:14 AM, Chris Adams wrote:
Once upon a time, Björn Persson <Bjorn@rombobjörn.se> said:
If this will affect containers, then I'm starting to worry that I might be affected. I thought I'd be fine as the computers I run virtual machines on are x86-64-v2 and -v3, but another computer that still works fine is -v1. As popular as containers are, who knows when some program I use will switch to being distributed only as a container, and thus become unusable on the -v1 box?
Are you using containers for a different architecture (e.g. running aarch64 containers on an x86_64 host)? AFAIK that's when podman will use QEMU. If not, you would not be affected.
Are the executables packaged separately so that the same-arch qemu could not change, but the different-arch executables won't be installable?
Gary Buhrmaster wrote:
Please provide your audited (by a 3rd party) data that shows that MANY (i.e. significant percentage of) people on x86_64-v1 users have a need for qemu.
"Many people" was just an estimate considering that QEMU is a popular package, so it is more likely to be used in general than some obscure niche package, so it is also statistically more likely to be used on any particular class of hardware, even old hardware.
Also, "many people" does NOT mean "significant percentage of", but a significant absolute number. If QEMU is used by, say, 1 million people, and, say, only 1% of those uses old "x86_64-v1" hardware (those are NOT actual numbers, just an example with hypothetical numbers to illustrate the computation involved, which is mathematically just a simple multiplication), that would still mean 10000 users are affected, which fits my definition of "many people".
But in the end, it does not matter. The policy is that ALL packages are supposed to comply with the distrowide baseline architecture, no matter how many or what percentage of users of the package actually use a computer old enough to support only the baseline.
Kevin Kofler
Could you by any chance link to the commit that introduces this issue? I'd like to bring this to the awareness of core devs in Ubuntu.
-
Aaron Rainbolt -
Barry Scott -
Ben Beasley -
Ben Cotton -
Björn Persson -
Chris Adams -
Dan Horák -
Daniel P. Berrangé -
Dominik 'Rathann' Mierzejewski -
drago01 -
Fabio Valentini -
Florian Weimer -
Gary Buhrmaster -
Guinevere Larsen -
Jakub Jelinek -
Jerry James -
Josh Boyer -
Kevin Kofler -
Neal Gompa -
Peter Robinson -
Przemek Klosowski -
Richard W.M. Jones -
Samuel Sieb -
Simon Farnsworth -
Stephen Gallagher -
Stephen Smoogen -
Todd Zullinger -
Tom Hughes -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek