Fedora RISC-V port needs to put shared objects into /usr/lib64/lp64d

[Florian Weimer]

There are multiple PRs and patches floating around that make RISC-V use
the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
various upstream projects follow that.

I think we should follow upstream, so that it's possible to use Fedora
to do upstream development without patching the sources, or elaborate
Fedora-specific configure invocations.  The other reasons is to
future-proof the Fedora port against the arrival of an alternative ABI
that is not fully backwards-compatible (the same reason why the official
RISC-V documentation requires use of these paths).

Thanks,
Florian

[Neal Gompa]

On Fri, Apr 19, 2024 at 8:23 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  There are multiple PRs and patches floating around that make RISC-V use
>  the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  various upstream projects follow that.
>
>  I think we should follow upstream, so that it's possible to use Fedora
>  to do upstream development without patching the sources, or elaborate
>  Fedora-specific configure invocations.  The other reasons is to
>  future-proof the Fedora port against the arrival of an alternative ABI
>  that is not fully backwards-compatible (the same reason why the official
>  RISC-V documentation requires use of these paths).
>  
Then we probably need to change the way our arch handling works to
encode the ABI too in RPM.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Daniel P. Berrangé]

On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
>  There are multiple PRs and patches floating around that make RISC-V
> use
>  the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  various upstream projects follow that.
>  
>  I think we should follow upstream, so that it's possible to use Fedora
>  to do upstream development without patching the sources, or elaborate
>  Fedora-specific configure invocations. 
I'm not convinced that using /usr/lib64/lp64d would lead to
*less* patching.

Apps targetting Fedora are long used to having to adapt from
using /usr/lib to /usr/lib64. Introducing the use of
/usr/lib64/lp64d instead, just for RiscV, feels likely to
break expectations resulting in apps which build fine on all
Fedora arches except for RiscV

>                                          The other reasons is to
>  future-proof the Fedora port against the arrival of an alternative ABI
>  that is not fully backwards-compatible (the same reason why the official
>  RISC-V documentation requires use of these paths). 
This implies we would have to have two different RiscV ABIs built for
every binary, and made parallel installable, as opposed to completely
distinct Fedora arch targets. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Florian Weimer]

* Daniel P. Berrangé:

>  On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
> > There are multiple PRs and patches floating around that make RISC-V use
> > the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
> > recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
> > various upstream projects follow that.
> > 
> > I think we should follow upstream, so that it's possible to use Fedora
> > to do upstream development without patching the sources, or elaborate
> > Fedora-specific configure invocations.
>
>  I'm not convinced that using /usr/lib64/lp64d would lead to
>  *less* patching.
>
>  Apps targetting Fedora are long used to having to adapt from
>  using /usr/lib to /usr/lib64. 
But that's largely baked into the upstream defaults by now (unlike the
Debian multi-arch paths).

>  Introducing the use of /usr/lib64/lp64d instead, just for RiscV,
> feels
>  likely to break expectations resulting in apps which build fine on all
>  Fedora arches except for RiscV 
I don't want us to have RPM spec file hacks just to get RISC-V to
install in the correct locations.  The symbolic link evidently does not
cover all cases.

Whatever we do, it should be upstream.  Maybe convince RISC-V to adopt
/usr/lib64.  Or have the RISC-V folks implement automated detection of
path layout in autotools, Meson etc., so that out of the box, both paths
work.

Thanks,
Florian

[Richard W.M. Jones]

On Mon, Apr 22, 2024 at 12:07:38PM +0200, Florian Weimer wrote:
>  * Daniel P. Berrangé:
>  
>  > On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
>  >> There are multiple PRs and patches floating around that make RISC-V use
>  >> the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  >> recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  >> various upstream projects follow that.
>  >> 
>  >> I think we should follow upstream, so that it's possible to use Fedora
>  >> to do upstream development without patching the sources, or elaborate
>  >> Fedora-specific configure invocations.
>  >
>  > I'm not convinced that using /usr/lib64/lp64d would lead to
>  > *less* patching.
>  >
>  > Apps targetting Fedora are long used to having to adapt from
>  > using /usr/lib to /usr/lib64.
>  
>  But that's largely baked into the upstream defaults by now (unlike the
>  Debian multi-arch paths).
>  
>  > Introducing the use of /usr/lib64/lp64d instead, just for RiscV, feels
>  > likely to break expectations resulting in apps which build fine on all
>  > Fedora arches except for RiscV
>  
>  I don't want us to have RPM spec file hacks just to get RISC-V to
>  install in the correct locations.  The symbolic link evidently does not
>  cover all cases. 
What cases aren't covered by the symlink?  We have a full, working
Fedora/RISC-V distro using it at the moment.

Rich.

>  Whatever we do, it should be upstream.  Maybe convince RISC-V to
> adopt
>  /usr/lib64.  Or have the RISC-V folks implement automated detection of
>  path layout in autotools, Meson etc., so that out of the box, both paths
>  work.
>  
>  Thanks,
>  Florian
>  --
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Florian Weimer]

* Richard W. M. Jones:

> > I don't want us to have RPM spec file hacks just to get
> RISC-V to
> > install in the correct locations.  The symbolic link evidently does not
> > cover all cases.
>
>  What cases aren't covered by the symlink?  We have a full, working
>  Fedora/RISC-V distro using it at the moment. 
The symbolic link isn't in the buildroot.  If shared objects are listed
explicitly in %files (as some guidelines recommend) and upstream
hard-codes the ABI directory names for installation purposes, the build
fails.

Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
upstream to honor --libdir=/usr/lib64 in ./configure might be another
option.

Thanks,
Florian

[Stephen Smoogen]

I guess we need to see what RPM owns that symlink and get it into the build
root

Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren


On Mon, Apr 29, 2024 at 08:22 Florian Weimer <fweimer(a)redhat.com&gt; wrote:

>  * Richard W. M. Jones:
>
>  >> I don't want us to have RPM spec file hacks just to get RISC-V to
>  >> install in the correct locations.  The symbolic link evidently does not
>  >> cover all cases.
>  >
>  > What cases aren't covered by the symlink?  We have a full, working
>  > Fedora/RISC-V distro using it at the moment.
>
>  The symbolic link isn't in the buildroot.  If shared objects are listed
>  explicitly in %files (as some guidelines recommend) and upstream
>  hard-codes the ABI directory names for installation purposes, the build
>  fails.
>
>  Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
>  upstream to honor --libdir=/usr/lib64 in ./configure might be another
>  option.
>
>  Thanks,
>  Florian
>  --
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[David Abdurachmanov]

On Mon, Apr 29, 2024 at 3:31 PM Stephen Smoogen <ssmoogen(a)redhat.com&gt; wrote:
>  I guess we need to see what RPM owns that symlink and get it into the build root
>
>  Stephen Smoogen, Red Hat Automotive
>  Let us be kind to one another, for most of us are fighting a hard battle. -- Ian
> MacClaren
>
>
>  On Mon, Apr 29, 2024 at 08:22 Florian Weimer <fweimer(a)redhat.com&gt; wrote:
> >
> > * Richard W. M. Jones:
> >
> > >> I don't want us to have RPM spec file hacks just to get RISC-V to
> > >> install in the correct locations.  The symbolic link evidently does not
> > >> cover all cases.
> > >
> > > What cases aren't covered by the symlink?  We have a full, working
> > > Fedora/RISC-V distro using it at the moment.
> >
> > The symbolic link isn't in the buildroot.  If shared objects are listed
> > explicitly in %files (as some guidelines recommend) and upstream
> > hard-codes the ABI directory names for installation purposes, the build
> > fails.
> >
> > Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
> > upstream to honor --libdir=/usr/lib64 in ./configure might be another
> > option.
> > 
We never patched the filesystem package to properly introduce the
symlink. It's extremely rare that it wouldn't be available in
buildroot, but it does happen.

Cheers,
david

[Florian Weimer]

* Stephen Smoogen:

>  I guess we need to see what RPM owns that symlink and get it into
> the
>  build root 
Sorry, I meant $RPM_BUILDROOT or %buildroot (the staging area used by
rpmbuild).  That's not controlled by the system package manager,
obviously.

Thanks,
Florian

>  On Mon, Apr 29, 2024 at 08:22 Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
>
>   * Richard W. M. Jones:
>
>   >> I don't want us to have RPM spec file hacks just to get RISC-V to
>   >> install in the correct locations.  The symbolic link evidently does not
>   >> cover all cases.
>   >
>   > What cases aren't covered by the symlink?  We have a full, working
>   > Fedora/RISC-V distro using it at the moment.
>
>   The symbolic link isn't in the buildroot.  If shared objects are listed
>   explicitly in %files (as some guidelines recommend) and upstream
>   hard-codes the ABI directory names for installation purposes, the build
>   fails.
>
>   Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
>   upstream to honor --libdir=/usr/lib64 in ./configure might be another
>   option.
>
>   Thanks,
>   Florian
>   --
>   _______________________________________________
>   devel mailing list -- devel(a)lists.fedoraproject.org
>   To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>   Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>   List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>   List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>   Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>
>  --
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Richard W.M. Jones]

On Mon, Apr 29, 2024 at 04:17:37PM +0200, Florian Weimer wrote:
>  * Stephen Smoogen:
>  
>  > I guess we need to see what RPM owns that symlink and get it into the
>  > build root
>  
>  Sorry, I meant $RPM_BUILDROOT or %buildroot (the staging area used by
>  rpmbuild).  That's not controlled by the system package manager,
>  obviously. 
I see - AIUI %buildroot is always empty when RPM starts the %install
phase.  Maybe we can modify RPM macros to drop in the symlink here?

Still, in practical terms hundreds of C libraries are built today and
don't have any issue.

Rich.

>  Thanks,
>  Florian
>  
>  > On Mon, Apr 29, 2024 at 08:22 Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  >
>  >  * Richard W. M. Jones:
>  >
>  >  >> I don't want us to have RPM spec file hacks just to get RISC-V to
>  >  >> install in the correct locations.  The symbolic link evidently does not
>  >  >> cover all cases.
>  >  >
>  >  > What cases aren't covered by the symlink?  We have a full, working
>  >  > Fedora/RISC-V distro using it at the moment.
>  >
>  >  The symbolic link isn't in the buildroot.  If shared objects are listed
>  >  explicitly in %files (as some guidelines recommend) and upstream
>  >  hard-codes the ABI directory names for installation purposes, the build
>  >  fails.
>  >
>  >  Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
>  >  upstream to honor --libdir=/usr/lib64 in ./configure might be another
>  >  option.
>  >
>  >  Thanks,
>  >  Florian
>  >  --
>  >  _______________________________________________
>  >  devel mailing list -- devel(a)lists.fedoraproject.org
>  >  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  >  Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  >  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  >  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  >  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>  >
>  > --
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>  --
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard W.M. Jones]

On Mon, Apr 29, 2024 at 02:21:56PM +0200, Florian Weimer wrote:
>  * Richard W. M. Jones:
>  
>  >> I don't want us to have RPM spec file hacks just to get RISC-V to
>  >> install in the correct locations.  The symbolic link evidently does not
>  >> cover all cases.
>  >
>  > What cases aren't covered by the symlink?  We have a full, working
>  > Fedora/RISC-V distro using it at the moment.
>  
>  The symbolic link isn't in the buildroot. 
This is indeed a problem.  The symlink should be moved from glibc to
filesystem.  I don't know why we put it in glibc, historical anomaly I
guess.  I'll prepare a couple of PRs today which will have to be
carefully pushed in a single side tag to fix this.

>  If shared objects are listed
>  explicitly in %files (as some guidelines recommend) and upstream
>  hard-codes the ABI directory names for installation purposes, the build
>  fails.
>
>  Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
>  upstream to honor --libdir=/usr/lib64 in ./configure might be another
>  option. 
I'm still unclear on what exactly the problem is that this solves.
We've built hundreds of C libraries in Fedora/RISC-V with the current
set up and it appears to work fine.  Will moving the symlink (alone)
fix this or is there some other error?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Florian Weimer]

* Richard W. M. Jones:

>  On Mon, Apr 29, 2024 at 02:21:56PM +0200, Florian Weimer wrote:
> > * Richard W. M. Jones:
> > 
> > >> I don't want us to have RPM spec file hacks just to get RISC-V to
> > >> install in the correct locations.  The symbolic link evidently does not
> > >> cover all cases.
> > >
> > > What cases aren't covered by the symlink?  We have a full, working
> > > Fedora/RISC-V distro using it at the moment.
> > 
> > The symbolic link isn't in the buildroot.
>
>  This is indeed a problem.  The symlink should be moved from glibc to
>  filesystem.  I don't know why we put it in glibc, historical anomaly I
>  guess.  I'll prepare a couple of PRs today which will have to be
>  carefully pushed in a single side tag to fix this.
>
> > If shared objects are listed
> > explicitly in %files (as some guidelines recommend) and upstream
> > hard-codes the ABI directory names for installation purposes, the build
> > fails.
> >
> > Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
> > upstream to honor --libdir=/usr/lib64 in ./configure might be another
> > option.
>
>  I'm still unclear on what exactly the problem is that this solves.
>  We've built hundreds of C libraries in Fedora/RISC-V with the current
>  set up and it appears to work fine.  Will moving the symlink (alone)
>  fix this or is there some other error? 
Sorry, see my other response.  This is about the
%buildroot/$RPM_BUILDROOT staging area used during the rpmbuild process,
not the system installation.

Thanks,
Florian

[Dan Horák]

On Tue, 30 Apr 2024 09:49:34 +0200
Florian Weimer <fweimer(a)redhat.com&gt; wrote:

>  * Richard W. M. Jones:
>  
>  > On Mon, Apr 29, 2024 at 02:21:56PM +0200, Florian Weimer wrote:
>  >> * Richard W. M. Jones:
>  >> 
>  >> >> I don't want us to have RPM spec file hacks just to get RISC-V to
>  >> >> install in the correct locations.  The symbolic link evidently does
> not
>  >> >> cover all cases.
>  >> >
>  >> > What cases aren't covered by the symlink?  We have a full, working
>  >> > Fedora/RISC-V distro using it at the moment.
>  >> 
>  >> The symbolic link isn't in the buildroot.
>  >
>  > This is indeed a problem.  The symlink should be moved from glibc to
>  > filesystem.  I don't know why we put it in glibc, historical anomaly I
>  > guess.  I'll prepare a couple of PRs today which will have to be
>  > carefully pushed in a single side tag to fix this.
>  >
>  >> If shared objects are listed
>  >> explicitly in %files (as some guidelines recommend) and upstream
>  >> hard-codes the ABI directory names for installation purposes, the build
>  >> fails.
>  >>
>  >> Setting %_libdir to /usr/lib64/lp64d instead might work.  Fixing
>  >> upstream to honor --libdir=/usr/lib64 in ./configure might be another
>  >> option.
>  >
>  > I'm still unclear on what exactly the problem is that this solves.
>  > We've built hundreds of C libraries in Fedora/RISC-V with the current
>  > set up and it appears to work fine.  Will moving the symlink (alone)
>  > fix this or is there some other error?
>  
>  Sorry, see my other response.  This is about the
>  %buildroot/$RPM_BUILDROOT staging area used during the rpmbuild process,
>  not the system installation. 
perhaps rpmbuild could create a minimal directory structure
(%_bindir, %_libdir) when starting the %install section. And maybe it
could be generally beneficial ...


		Dan

[Andrea Bolognani]

On Tue, Apr 30, 2024 at 08:19:41AM GMT, Richard W.M. Jones wrote:
>  On Mon, Apr 29, 2024 at 02:21:56PM +0200, Florian Weimer wrote:
>  > > What cases aren't covered by the symlink?  We have a full, working
>  > > Fedora/RISC-V distro using it at the moment.
>  >
>  > The symbolic link isn't in the buildroot.
>
>  This is indeed a problem.  The symlink should be moved from glibc to
>  filesystem.  I don't know why we put it in glibc, historical anomaly I
>  guess.  I'll prepare a couple of PRs today which will have to be
>  carefully pushed in a single side tag to fix this. 
Shouldn't the symlink point in the opposite direction anyway?
/usr/lib64/lp64d is the actual canonical path, /usr/lib64 is just for
compatibility.

Though apparently (see elsewhere in the thread) Gentoo does it this
way too, so maybe there's something I'm missing that makes the
current direction more desirable.

-- 
Andrea Bolognani / Red Hat / Virtualization

[Florian Weimer]

* Andrea Bolognani:

>  On Tue, Apr 30, 2024 at 08:19:41AM GMT, Richard W.M. Jones wrote:
> > On Mon, Apr 29, 2024 at 02:21:56PM +0200, Florian Weimer wrote:
> > > > What cases aren't covered by the symlink?  We have a full, working
> > > > Fedora/RISC-V distro using it at the moment.
> > >
> > > The symbolic link isn't in the buildroot.
> >
> > This is indeed a problem.  The symlink should be moved from glibc to
> > filesystem.  I don't know why we put it in glibc, historical anomaly I
> > guess.  I'll prepare a couple of PRs today which will have to be
> > carefully pushed in a single side tag to fix this.
>
>  Shouldn't the symlink point in the opposite direction anyway?
>  /usr/lib64/lp64d is the actual canonical path, /usr/lib64 is just for
>  compatibility.
>
>  Though apparently (see elsewhere in the thread) Gentoo does it this
>  way too, so maybe there's something I'm missing that makes the
>  current direction more desirable. 
It's simply not possible to have a symbolic link in the other direction.

Thanks,
Florian

[Andrea Bolognani]

On Tue, Apr 30, 2024 at 10:02:52AM GMT, Florian Weimer wrote:
>  > Shouldn't the symlink point in the opposite direction
> anyway?
>  > /usr/lib64/lp64d is the actual canonical path, /usr/lib64 is just for
>  > compatibility.
>  >
>  > Though apparently (see elsewhere in the thread) Gentoo does it this
>  > way too, so maybe there's something I'm missing that makes the
>  > current direction more desirable.
>
>  It's simply not possible to have a symbolic link in the other direction. 
🤦

And that's why you don't answer emails, especially those on public
lists, before the second coffee of the day has kicked in :)

-- 
Andrea Bolognani / Red Hat / Virtualization

[Andreas K. Huettel]

Am Dienstag, 30. April 2024, 19:00:33 +11 schrieb Andrea Bolognani:
>  
>  Shouldn't the symlink point in the opposite direction anyway?
>  /usr/lib64/lp64d is the actual canonical path, /usr/lib64 is just for
>  compatibility.
>  
>  Though apparently (see elsewhere in the thread) Gentoo does it this
>  way too, so maybe there's something I'm missing that makes the
>  current direction more desirable.
>   
Apart from certain difficulties in implementing that (as Florian said),
telling upstream build systems to install their libraries into 
/usr/lib64/lp64d breaks too many (wrong) assumptions (of the library
directory being one dir level under /usr).

We tried and had some surprises.

Obviously these are upstream bugs in various packages. I don't remember
the details anymore, sorry. Explaining the RISC-V directory structure to 
people back then was somewhat painful. Today it might be easier.

-- 
Andreas K. Hüttel
dilfridge(a)gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[David Abdurachmanov]

On Mon, Apr 22, 2024 at 1:08 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  * Daniel P. Berrangé:
>
>  > On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
>  >> There are multiple PRs and patches floating around that make RISC-V use
>  >> the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  >> recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  >> various upstream projects follow that.
>  >>
>  >> I think we should follow upstream, so that it's possible to use Fedora
>  >> to do upstream development without patching the sources, or elaborate
>  >> Fedora-specific configure invocations.
>  >
>  > I'm not convinced that using /usr/lib64/lp64d would lead to
>  > *less* patching.
>  >
>  > Apps targetting Fedora are long used to having to adapt from
>  > using /usr/lib to /usr/lib64.
>
>  But that's largely baked into the upstream defaults by now (unlike the
>  Debian multi-arch paths).
>
>  > Introducing the use of /usr/lib64/lp64d instead, just for RiscV, feels
>  > likely to break expectations resulting in apps which build fine on all
>  > Fedora arches except for RiscV
>
>  I don't want us to have RPM spec file hacks just to get RISC-V to
>  install in the correct locations.  The symbolic link evidently does not
>  cover all cases. 
This is quite rare considering that there are ~24K packages in Fedora.
I have been looking at packages/SPECs for years now, and I wouldn't be
surprised that the amount of modifications (%ifarch riscv64 and
similar) is less than for other supported arches in Fedora.

The symlink was approved (by Carlos IIRC) back in 2018. We only added
it to GCC and glibc. What we never did was introduce it properly into
the filesystem package. That means we sometimes get buildroots that
lack the symlink (very rare).

We don't have control on dlopen(), DT_RUNPATH, DT_RPATH, etc. We don't
know if applications will look for libraries in /usr/lib64/lp64d and
will they fallback to /usr/lib64.

Otherwise the smallest delta between upstream Fedora and Fedora/RISCV
was ~1500 packages. There are ~1000 packages that typically don't
build in general. Those ~500 contain various things like broken *.pc
files, meson producing wrong paths (fixed since yesterday),
-mach=native (surprise!), -m64, not supported packages in general
(needs ExcludeArch: update), etc.

We are building GCC with multilib enabled, but a single ABI selected
(lp64d). We are doing this because it was the only way to get sane
(i.e. expected) paths for libraries. The last time we checked it,
disabling multilib means that libraries get installed into /usr/lib
and not /usr/lib64. Again that still doesn't solve the problem as some
applications will only check /usr/lib64/lp64d without a fallback to
/usr/lib64.

Cheers,
david

>
> Whatever we do, it should be upstream.  Maybe convince RISC-V to adopt
> /usr/lib64.  Or have the RISC-V folks implement automated detection of
> path layout in autotools, Meson etc., so that out of the box, both paths
> work.
>
> Thanks,
> Florian
> --
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Richard W.M. Jones]

On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
>  There are multiple PRs and patches floating around that make RISC-V
> use
>  the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  various upstream projects follow that.
>  
>  I think we should follow upstream, so that it's possible to use Fedora
>  to do upstream development without patching the sources, or elaborate
>  Fedora-specific configure invocations.  The other reasons is to
>  future-proof the Fedora port against the arrival of an alternative ABI
>  that is not fully backwards-compatible (the same reason why the official
>  RISC-V documentation requires use of these paths). 
Currently /usr/lib64/lp64d is a symlink to /usr/lib64:

$ ls -ld /usr/lib64/lp64d
lrwxrwxrwx 1 root root 1 Jan 16 00:00 /usr/lib64/lp64d -> .

(link is created in the glibc package).

Seems to be the best of both worlds?

TBH I'm not convinced that /usr/lib64/lp64d is a good idea, or any
major difference like this from other architectures, or that there
will be other ABIs that will have to be parallel installable.  (We
might, I suppose, change the ABI in future, but then we'd have to
recompile everything, same as we'd do on any other architecture.)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Andrea Bolognani]

On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
>  There are multiple PRs and patches floating around that make RISC-V
> use
>  the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  various upstream projects follow that.
>
>  I think we should follow upstream, so that it's possible to use Fedora
>  to do upstream development without patching the sources, or elaborate
>  Fedora-specific configure invocations.  The other reasons is to
>  future-proof the Fedora port against the arrival of an alternative ABI
>  that is not fully backwards-compatible (the same reason why the official
>  RISC-V documentation requires use of these paths). 
I just checked in a Debian riscv64 chroot and they don't seem to
follow this recommendation:

  # cat /etc/ld.so.conf.d/riscv64-linux-gnu.conf
  /usr/local/lib/riscv64-linux-gnu
  /lib/riscv64-linux-gnu
  /usr/lib/riscv64-linux-gnu

This matches what Debian does on all architectures, that is, install
libraries under fully arch-qualified paths. If Debian doesn't stray
from its usual practices for RISC-V, I'm not convinced that Fedora
needs to either.

-- 
Andrea Bolognani / Red Hat / Virtualization

[David Abdurachmanov]

On Fri, Apr 19, 2024 at 8:36 PM Andrea Bolognani <abologna(a)redhat.com&gt; wrote:
>  On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
>  > There are multiple PRs and patches floating around that make RISC-V use
>  > the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  > recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  > various upstream projects follow that.
>  >
>  > I think we should follow upstream, so that it's possible to use Fedora
>  > to do upstream development without patching the sources, or elaborate
>  > Fedora-specific configure invocations.  The other reasons is to
>  > future-proof the Fedora port against the arrival of an alternative ABI
>  > that is not fully backwards-compatible (the same reason why the official
>  > RISC-V documentation requires use of these paths).
>
>  I just checked in a Debian riscv64 chroot and they don't seem to
>  follow this recommendation:
>
>    # cat /etc/ld.so.conf.d/riscv64-linux-gnu.conf
>    /usr/local/lib/riscv64-linux-gnu
>    /lib/riscv64-linux-gnu
>    /usr/lib/riscv64-linux-gnu
>
>  This matches what Debian does on all architectures, that is, install
>  libraries under fully arch-qualified paths. If Debian doesn't stray
>  from its usual practices for RISC-V, I'm not convinced that Fedora
>  needs to either. 
A long time ago /usr/lib64 wasn't even considered as a fallback search
path in the linker (for riscv64). Some projects assume /usr/lib64,
some explicitly check /usr/lib64/<ABI>, some have fallbacks, some
don't IIRC.

We currently use a symlink (as Richard) mentioned, but it's not ideal
and causes problems (e.g. meson generates wrong paths breaking some
packages [one example: libplacebo]).

We most likely will not have ABIs installed in parallel, but we might
change ABI. Currently Linux distributions target "RV64GC", but we
don't really want that for the future RISC-V. I keep telling folks
that "RV64GC" is already "a legacy" (10+ years old), but that's
the
major target for the next few years. We are scheduled to see some
RVA22 SBCs this year.

RV64GC -> RVA20 -> RVA22 -> RVA23 -> RVA24.

That's how things evolved. RVA23 most likely will be ratified soonish.
RVA24 is most likely the next major RISC-V Profile from RVI point of
view (TBD). Server specifications are based on RVA23 (and will be
updated to RVA24 [most likely]). This defines baseline ISA, optional
extensions, etc.

Palmer sent RFC to libc-alpha to expand glibc targets on RISC-V for
testing. This does mention a potential of "lp64dv" ABI maybe for GCC
15.

Note that starting RVA23 vectors are required. Google currently picked
RVA22 + vectors as their initial baseline ISA for Android. We don't
know if the SoC/IP market will follow what Google (Android) decides or
the vendor will move fast to newer RISC-V Profiles.

From what I heard at public RVI meeting recordings I would assume that
Ubuntu will change their baseline ISA for RISC-V.

Back to original topic.

The symlink approach is simple, but also a bit ugly (remember meson
breaking some packages). We should pick one way, or another. Finally
remove the symlink. This might mean a lot of work to cleanup whatever
gets broken. We don't really have enough compute to run a full
distribution test over a short period of time.

Cheers,
david

>
> --
> Andrea Bolognani / Red Hat / Virtualization
> --
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Kevin Kofler]

David Abdurachmanov wrote:
>  We currently use a symlink (as Richard) mentioned, but it's not
> ideal
>  and causes problems (e.g. meson generates wrong paths breaking some
>  packages [one example: libplacebo]). 
Which I would say is a bug in Meson and should be fixed there.

I do not think having /usr/lib64/lp64d be a symlink to /usr/lib64 is in 
violation of any standard.

        Kevin Kofler

[David Abdurachmanov]

On Sun, Apr 21, 2024 at 2:19 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  David Abdurachmanov wrote:
>  > We currently use a symlink (as Richard) mentioned, but it's not ideal
>  > and causes problems (e.g. meson generates wrong paths breaking some
>  > packages [one example: libplacebo]).
>
>  Which I would say is a bug in Meson and should be fixed there.
>
>  I do not think having /usr/lib64/lp64d be a symlink to /usr/lib64 is in
>  violation of any standard.
>  
Correct.

If you are interested in more details we have a PR/discussion here:
https://github.com/mesonbuild/meson/pull/12808

[Richard W.M. Jones]

On Sun, Apr 21, 2024 at 06:51:02AM +0300, David Abdurachmanov wrote:
>  On Sun, Apr 21, 2024 at 2:19 AM Kevin Kofler via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>  >
>  > David Abdurachmanov wrote:
>  > > We currently use a symlink (as Richard) mentioned, but it's not ideal
>  > > and causes problems (e.g. meson generates wrong paths breaking some
>  > > packages [one example: libplacebo]).
>  >
>  > Which I would say is a bug in Meson and should be fixed there.
>  >
>  > I do not think having /usr/lib64/lp64d be a symlink to /usr/lib64 is in
>  > violation of any standard.
>  >
>  
>  Correct.
>  
>  If you are interested in more details we have a PR/discussion here:
>  https://github.com/mesonbuild/meson/pull/12808 
Whether or not this particular PR is correct, the attitude of the
Meson author displayed there made me not want to port any projects to it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Florian Weimer]

* David Abdurachmanov:

>  We most likely will not have ABIs installed in parallel, but we
> might
>  change ABI. Currently Linux distributions target "RV64GC", but we
>  don't really want that for the future RISC-V. I keep telling folks
>  that "RV64GC" is already "a legacy" (10+ years old), but that's
> the
>  major target for the next few years. We are scheduled to see some
>  RVA22 SBCs this year.
>
>  RV64GC -> RVA20 -> RVA22 -> RVA23 -> RVA24.
>
>  That's how things evolved. RVA23 most likely will be ratified soonish.
>  RVA24 is most likely the next major RISC-V Profile from RVI point of
>  view (TBD). Server specifications are based on RVA23 (and will be
>  updated to RVA24 [most likely]). This defines baseline ISA, optional
>  extensions, etc. 
Is there really an ABI change, though?  That would only happen if the
set of callee-saved registers grows, to include vector registers.
Adding new registers has compatibility problems on its own.

Fedora might want to switch to an ISA baseline that is incompatible with
all currently available CPUs, but that's not necessarily an ABI break as
such.

Thanks,
Florian

[David Abdurachmanov]

On Mon, Apr 22, 2024 at 1:12 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  * David Abdurachmanov:
>
>  > We most likely will not have ABIs installed in parallel, but we might
>  > change ABI. Currently Linux distributions target "RV64GC", but we
>  > don't really want that for the future RISC-V. I keep telling folks
>  > that "RV64GC" is already "a legacy" (10+ years old), but
> that's the
>  > major target for the next few years. We are scheduled to see some
>  > RVA22 SBCs this year.
>  >
>  > RV64GC -> RVA20 -> RVA22 -> RVA23 -> RVA24.
>  >
>  > That's how things evolved. RVA23 most likely will be ratified soonish.
>  > RVA24 is most likely the next major RISC-V Profile from RVI point of
>  > view (TBD). Server specifications are based on RVA23 (and will be
>  > updated to RVA24 [most likely]). This defines baseline ISA, optional
>  > extensions, etc.
>
>  Is there really an ABI change, though?  That would only happen if the
>  set of callee-saved registers grows, to include vector registers.
>  Adding new registers has compatibility problems on its own. 
There are two conventions for vectors in RISCV psABI: "standard" and
"calling" (not the best name).

standard does not have preserved registers across calls.
calling one does (similar to what you would expect from AVX).

The default today is standard, which means that -march=rv64gv
-mabi=lp64d enables vectors and we can mix them with existing
binaries. This is a very limited use of vectors.

GCC 14 is the 1st release that has vector support for RISCV. There
might be suggestions to add lp64dv as ABI in the future (maybe GCC 15?
Unknown) that would default to the vector calling convention. There
might be more extensions that could require ABI change IIUC, but I
don't recall the details.

RVA23 requires vectors, and vector crypto. Android will also require
vectors, it's not an option.

I hope that once RVA23 and especially RVA24 lands, RVI will announce
the next major RISCV profile (RVA24, might have a fancier marketing
name). We will see what the market does, but we have already seen some
cores being marketed as RVA24 (yet I don't really trust that). Once we
start looking at the next baseline ISA we might as well pick the best
ABI for it. We could call this "Fedora/RISCV.Next". RV64GC will
probably be 10-15 years old by that time, and it lacks any modern ISA
especially if you want high performance. I consider RVA24 a point
where we close major gaps between RISCV and other popular arches.

Cheers,
david

>
> Fedora might want to switch to an ISA baseline that is incompatible with
> all currently available CPUs, but that's not necessarily an ABI break as
> such.
>
> Thanks,
> Florian
>

[Andrea Bolognani]

On Tue, Apr 23, 2024 at 09:43:34AM +0300, David Abdurachmanov wrote:
>  On Mon, Apr 22, 2024 at 1:12 PM Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
>  > > We most likely will not have ABIs installed in parallel, but we might
>  > > change ABI. Currently Linux distributions target "RV64GC", but we
>  > > don't really want that for the future RISC-V. I keep telling folks
>  > > that "RV64GC" is already "a legacy" (10+ years old), but
> that's the
>  > > major target for the next few years. We are scheduled to see some
>  > > RVA22 SBCs this year.
>  > >
>  > > RV64GC -> RVA20 -> RVA22 -> RVA23 -> RVA24.
>  > >
>  > > That's how things evolved. RVA23 most likely will be ratified soonish.
>  > > RVA24 is most likely the next major RISC-V Profile from RVI point of
>  > > view (TBD). Server specifications are based on RVA23 (and will be
>  > > updated to RVA24 [most likely]). This defines baseline ISA, optional
>  > > extensions, etc.
>  >
>  > Is there really an ABI change, though?  That would only happen if the
>  > set of callee-saved registers grows, to include vector registers.
>  > Adding new registers has compatibility problems on its own.
>
>  There are two conventions for vectors in RISCV psABI: "standard" and
>  "calling" (not the best name).
>
>  standard does not have preserved registers across calls.
>  calling one does (similar to what you would expect from AVX).
>
>  The default today is standard, which means that -march=rv64gv
>  -mabi=lp64d enables vectors and we can mix them with existing
>  binaries. This is a very limited use of vectors.
>
>  GCC 14 is the 1st release that has vector support for RISCV. There
>  might be suggestions to add lp64dv as ABI in the future (maybe GCC 15?
>  Unknown) that would default to the vector calling convention. There
>  might be more extensions that could require ABI change IIUC, but I
>  don't recall the details.
>
>  RVA23 requires vectors, and vector crypto. Android will also require
>  vectors, it's not an option.
>
>  I hope that once RVA23 and especially RVA24 lands, RVI will announce
>  the next major RISCV profile (RVA24, might have a fancier marketing
>  name). We will see what the market does, but we have already seen some
>  cores being marketed as RVA24 (yet I don't really trust that). Once we
>  start looking at the next baseline ISA we might as well pick the best
>  ABI for it. We could call this "Fedora/RISCV.Next". RV64GC will
>  probably be 10-15 years old by that time, and it lacks any modern ISA
>  especially if you want high performance. I consider RVA24 a point
>  where we close major gaps between RISCV and other popular arches. 
Wouldn't changing -mabi effectively make the result a new Fedora
architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
to load libraries built with -mabi=lp64dv and vice versa.

If that's correct, then we can't simply have a single "riscv64"
architecture: instead, we need to call what we have today
riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
comes next.

It would be somewhat similar to existing architectures that can be
used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.

This is quite different from just bumping the ISA baseline, where
existing binaries and libraries are still usable in the new
environment.

-- 
Andrea Bolognani / Red Hat / Virtualization

[Florian Weimer]

* Andrea Bolognani:

>  Wouldn't changing -mabi effectively make the result a new Fedora
>  architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
>  to load libraries built with -mabi=lp64dv and vice versa.
>
>  If that's correct, then we can't simply have a single "riscv64"
>  architecture: instead, we need to call what we have today
>  riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  comes next. 
Right.

>  It would be somewhat similar to existing architectures that can be
>  used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes. 
With different paths, it would be more like i686 and x86_64.  That is,
you can build and run software for both variants from the same operating
system image.  That's not the case for ppc64 and ppc64le.

>  This is quite different from just bumping the ISA baseline, where
>  existing binaries and libraries are still usable in the new
>  environment. 
Right, changing the vector calling convention may change the size of
jmp_buf, and then you have a new ABI even if use of the vector features
is optional.

Thanks,
Florian

[Neal Gompa]

On Wed, Apr 24, 2024 at 7:16 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  * Andrea Bolognani:
>
>  > Wouldn't changing -mabi effectively make the result a new Fedora
>  > architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
>  > to load libraries built with -mabi=lp64dv and vice versa.
>  >
>  > If that's correct, then we can't simply have a single "riscv64"
>  > architecture: instead, we need to call what we have today
>  > riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  > comes next.
>
>  Right.
>
>  > It would be somewhat similar to existing architectures that can be
>  > used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.
>
>  With different paths, it would be more like i686 and x86_64.  That is,
>  you can build and run software for both variants from the same operating
>  system image.  That's not the case for ppc64 and ppc64le.
>  
As I recall, outside of the x86 family and s390x, all CPU platforms we
support are actually bi-endian and can have applications operate in
either mode. So I'm not sure that's a good example other than nobody
cared to specify parallel install paths for that.

>  > This is quite different from just bumping the ISA baseline,
> where
>  > existing binaries and libraries are still usable in the new
>  > environment.
>
>  Right, changing the vector calling convention may change the size of
>  jmp_buf, and then you have a new ABI even if use of the vector features
>  is optional.
>  
Arguably bumping the baseline should *also* be a new architecture
because it's a total compatibility break.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Andrea Bolognani]

On Wed, Apr 24, 2024 at 07:43:08AM -0400, Neal Gompa wrote:
>  On Wed, Apr 24, 2024 at 7:16 AM Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
>  > > Wouldn't changing -mabi effectively make the result a new Fedora
>  > > architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
>  > > to load libraries built with -mabi=lp64dv and vice versa.
>  > >
>  > > If that's correct, then we can't simply have a single
> "riscv64"
>  > > architecture: instead, we need to call what we have today
>  > > riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  > > comes next.
>  >
>  > Right.
>  >
>  > > It would be somewhat similar to existing architectures that can be
>  > > used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.
>  >
>  > With different paths, it would be more like i686 and x86_64.  That is,
>  > you can build and run software for both variants from the same operating
>  > system image.  That's not the case for ppc64 and ppc64le.
>
>  As I recall, outside of the x86 family and s390x, all CPU platforms we
>  support are actually bi-endian and can have applications operate in
>  either mode. So I'm not sure that's a good example other than nobody
>  cared to specify parallel install paths for that. 
So would you be able to natively run ppc64 binaries under a ppc64le
kernel? I don't think that'd be possible, but I don't have conclusive
proof so I could be wrong.

Similarly, would you be able to run riscv64_lp64dv binaries under a
riscv64_lp64d kernel? That doesn't sounds like it would work either.

And even if it did, you would still be unable to mix and match
binaries and libraries built for the two ABIs, so that makes them
separate Fedora architectures.

>  > > This is quite different from just bumping the ISA baseline,
> where
>  > > existing binaries and libraries are still usable in the new
>  > > environment.
>  >
>  > Right, changing the vector calling convention may change the size of
>  > jmp_buf, and then you have a new ABI even if use of the vector features
>  > is optional.
>
>  Arguably bumping the baseline should *also* be a new architecture
>  because it's a total compatibility break. 
No, you're just cutting off support for hardware that doesn't satsify
the new baseline. Which is obviously not something that should be
done lightly, but at least compatibility with existing binaries is
retained. Changing the ABI means starting from scratch.

-- 
Andrea Bolognani / Red Hat / Virtualization

[Neal Gompa]

On Wed, Apr 24, 2024 at 8:35 AM Andrea Bolognani <abologna(a)redhat.com&gt; wrote:
>  On Wed, Apr 24, 2024 at 07:43:08AM -0400, Neal Gompa wrote:
>  > On Wed, Apr 24, 2024 at 7:16 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  > > > Wouldn't changing -mabi effectively make the result a new Fedora
>  > > > architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
>  > > > to load libraries built with -mabi=lp64dv and vice versa.
>  > > >
>  > > > If that's correct, then we can't simply have a single
> "riscv64"
>  > > > architecture: instead, we need to call what we have today
>  > > > riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  > > > comes next.
>  > >
>  > > Right.
>  > >
>  > > > It would be somewhat similar to existing architectures that can be
>  > > > used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.
>  > >
>  > > With different paths, it would be more like i686 and x86_64.  That is,
>  > > you can build and run software for both variants from the same operating
>  > > system image.  That's not the case for ppc64 and ppc64le.
>  >
>  > As I recall, outside of the x86 family and s390x, all CPU platforms we
>  > support are actually bi-endian and can have applications operate in
>  > either mode. So I'm not sure that's a good example other than nobody
>  > cared to specify parallel install paths for that.
>
>  So would you be able to natively run ppc64 binaries under a ppc64le
>  kernel? I don't think that'd be possible, but I don't have conclusive
>  proof so I could be wrong.
>
>  Similarly, would you be able to run riscv64_lp64dv binaries under a
>  riscv64_lp64d kernel? That doesn't sounds like it would work either.
>
>  And even if it did, you would still be unable to mix and match
>  binaries and libraries built for the two ABIs, so that makes them
>  separate Fedora architectures.
>  
That's only the case if we don't have binary loaders that can deal
with it. Which admittedly nobody has done on Linux, but it's not
unheard of to have mixed ABI loading.

In particular the situation of running binaries of one ABI under the
kernel of another is something that has been done before even on x86.
x32-on-x64 as well as i686-on-x86_64 are both examples of this, as is
arm7hl-on-aarch64.

That *we* don't do it in Fedora doesn't mean it's not possible.

>  > > > This is quite different from just bumping the ISA
> baseline, where
>  > > > existing binaries and libraries are still usable in the new
>  > > > environment.
>  > >
>  > > Right, changing the vector calling convention may change the size of
>  > > jmp_buf, and then you have a new ABI even if use of the vector features
>  > > is optional.
>  >
>  > Arguably bumping the baseline should *also* be a new architecture
>  > because it's a total compatibility break.
>
>  No, you're just cutting off support for hardware that doesn't satsify
>  the new baseline. Which is obviously not something that should be
>  done lightly, but at least compatibility with existing binaries is
>  retained. Changing the ABI means starting from scratch.
>  
Changing the baseline doesn't necessarily imply the ABI doesn't
change. The fact that it generally *doesn't* doesn't mean that it
*won't*. This is up to the compiler toolchain to handle, and since
that's an implementation specific detail, I generally would not make
that assumption.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Andrea Bolognani]

On Wed, Apr 24, 2024 at 09:01:51AM -0400, Neal Gompa wrote:
>  On Wed, Apr 24, 2024 at 8:35 AM Andrea Bolognani
> <abologna(a)redhat.com&gt; wrote:
>  > On Wed, Apr 24, 2024 at 07:43:08AM -0400, Neal Gompa wrote:
>  > > On Wed, Apr 24, 2024 at 7:16 AM Florian Weimer <fweimer(a)redhat.com&gt;
> wrote:
>  > > > > Wouldn't changing -mabi effectively make the result a new Fedora
>  > > > > architecture? IIUC, binaries built with -mabi=lp64d wouldn't be
> able
>  > > > > to load libraries built with -mabi=lp64dv and vice versa.
>  > > > >
>  > > > > If that's correct, then we can't simply have a single
> "riscv64"
>  > > > > architecture: instead, we need to call what we have today
>  > > > > riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  > > > > comes next.
>  > > >
>  > > > Right.
>  > > >
>  > > > > It would be somewhat similar to existing architectures that can be
>  > > > > used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.
>  > > >
>  > > > With different paths, it would be more like i686 and x86_64.  That is,
>  > > > you can build and run software for both variants from the same operating
>  > > > system image.  That's not the case for ppc64 and ppc64le.
>  > >
>  > > As I recall, outside of the x86 family and s390x, all CPU platforms we
>  > > support are actually bi-endian and can have applications operate in
>  > > either mode. So I'm not sure that's a good example other than nobody
>  > > cared to specify parallel install paths for that.
>  >
>  > So would you be able to natively run ppc64 binaries under a ppc64le
>  > kernel? I don't think that'd be possible, but I don't have conclusive
>  > proof so I could be wrong.
>  >
>  > Similarly, would you be able to run riscv64_lp64dv binaries under a
>  > riscv64_lp64d kernel? That doesn't sounds like it would work either.
>
>  That's only the case if we don't have binary loaders that can deal
>  with it. Which admittedly nobody has done on Linux, but it's not
>  unheard of to have mixed ABI loading.
>
>  In particular the situation of running binaries of one ABI under the
>  kernel of another is something that has been done before even on x86.
>  x32-on-x64 as well as i686-on-x86_64 are both examples of this, as is
>  arm7hl-on-aarch64.
>
>  That *we* don't do it in Fedora doesn't mean it's not possible. 
Fair enough, but that's not the important point, this one is:

>  > And even if it did, you would still be unable to mix and match
>  > binaries and libraries built for the two ABIs, so that makes them
>  > separate Fedora architectures. 
Even if you can figure out a way to run binaries targeting a ABI-A
under a kernel targeting ABI-B, you still can't use the symbols
coming from a library targeting ABI-A from a binary targeting ABI-B.
And *that* is what makes them separate Fedora architectures.

>  > > > > This is quite different from just bumping the ISA
> baseline, where
>  > > > > existing binaries and libraries are still usable in the new
>  > > > > environment.
>  > > >
>  > > > Right, changing the vector calling convention may change the size of
>  > > > jmp_buf, and then you have a new ABI even if use of the vector features
>  > > > is optional.
>  > >
>  > > Arguably bumping the baseline should *also* be a new architecture
>  > > because it's a total compatibility break.
>  >
>  > No, you're just cutting off support for hardware that doesn't satsify
>  > the new baseline. Which is obviously not something that should be
>  > done lightly, but at least compatibility with existing binaries is
>  > retained. Changing the ABI means starting from scratch.
>
>  Changing the baseline doesn't necessarily imply the ABI doesn't
>  change. The fact that it generally *doesn't* doesn't mean that it
>  *won't*. This is up to the compiler toolchain to handle, and since
>  that's an implementation specific detail, I generally would not make
>  that assumption. 
If changing the baseline results in a different ABI, then sure,
that'd be a different Fedora architecture. But that's a much narrower
criteria than "bumping the baseline should also be a new
architecture".

-- 
Andrea Bolognani / Red Hat / Virtualization

[David Abdurachmanov]

On Wed, Apr 24, 2024 at 2:16 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  * Andrea Bolognani:
>
>  > Wouldn't changing -mabi effectively make the result a new Fedora
>  > architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
>  > to load libraries built with -mabi=lp64dv and vice versa.
>  >
>  > If that's correct, then we can't simply have a single "riscv64"
>  > architecture: instead, we need to call what we have today
>  > riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  > comes next.
>
>  Right.
>
>  > It would be somewhat similar to existing architectures that can be
>  > used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.
>
>  With different paths, it would be more like i686 and x86_64.  That is,
>  you can build and run software for both variants from the same operating
>  system image.  That's not the case for ppc64 and ppc64le. 
When we started Fedora/RISCV years ago we didn't want to things:
- multilib
- CONFIG_COMPAT

So it's one ABI per arch string. CONFIG_COMPAT is limited to a single
company effort (T-HEAD), and I haven't seen or heard about it being
used in general.

I would assume a new ABI would mean a new arch string. There are some
arch strings floating around from FreeBSD and OpenEmbedded (OE) that
aren't common: riscv64sf (FreeBSD, soft floats, might have been
removed recently), riscv64nf (OE, -mabi=lp64), riscv64nc (OE, C
extension disabled).

lp64dv is not currently being discussed for toolchains, and what it means.

There might be more ABIs because of other extensions arriving thus it
might not be only about the vectors standard vs calling conventions (I
really don't like these names).

I just want to remind you that "rv64gc" + "lp64d" combination will be
quite a restrictive combination within a few years, but we are
probably 1-3 years away from that discussion.

>
> > This is quite different from just bumping the ISA baseline, where
> > existing binaries and libraries are still usable in the new
> > environment.
>
> Right, changing the vector calling convention may change the size of
> jmp_buf, and then you have a new ABI even if use of the vector features
> is optional.
>
> Thanks,
> Florian
> --
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Richard W.M. Jones]

On Wed, Apr 24, 2024 at 06:07:38AM -0400, Andrea Bolognani wrote:
>  On Tue, Apr 23, 2024 at 09:43:34AM +0300, David Abdurachmanov wrote:
>  > On Mon, Apr 22, 2024 at 1:12 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  > > > We most likely will not have ABIs installed in parallel, but we might
>  > > > change ABI. Currently Linux distributions target "RV64GC", but
> we
>  > > > don't really want that for the future RISC-V. I keep telling folks
>  > > > that "RV64GC" is already "a legacy" (10+ years old),
> but that's the
>  > > > major target for the next few years. We are scheduled to see some
>  > > > RVA22 SBCs this year.
>  > > >
>  > > > RV64GC -> RVA20 -> RVA22 -> RVA23 -> RVA24.
>  > > >
>  > > > That's how things evolved. RVA23 most likely will be ratified
> soonish.
>  > > > RVA24 is most likely the next major RISC-V Profile from RVI point of
>  > > > view (TBD). Server specifications are based on RVA23 (and will be
>  > > > updated to RVA24 [most likely]). This defines baseline ISA, optional
>  > > > extensions, etc.
>  > >
>  > > Is there really an ABI change, though?  That would only happen if the
>  > > set of callee-saved registers grows, to include vector registers.
>  > > Adding new registers has compatibility problems on its own.
>  >
>  > There are two conventions for vectors in RISCV psABI: "standard" and
>  > "calling" (not the best name).
>  >
>  > standard does not have preserved registers across calls.
>  > calling one does (similar to what you would expect from AVX).
>  >
>  > The default today is standard, which means that -march=rv64gv
>  > -mabi=lp64d enables vectors and we can mix them with existing
>  > binaries. This is a very limited use of vectors.
>  >
>  > GCC 14 is the 1st release that has vector support for RISCV. There
>  > might be suggestions to add lp64dv as ABI in the future (maybe GCC 15?
>  > Unknown) that would default to the vector calling convention. There
>  > might be more extensions that could require ABI change IIUC, but I
>  > don't recall the details.
>  >
>  > RVA23 requires vectors, and vector crypto. Android will also require
>  > vectors, it's not an option.
>  >
>  > I hope that once RVA23 and especially RVA24 lands, RVI will announce
>  > the next major RISCV profile (RVA24, might have a fancier marketing
>  > name). We will see what the market does, but we have already seen some
>  > cores being marketed as RVA24 (yet I don't really trust that). Once we
>  > start looking at the next baseline ISA we might as well pick the best
>  > ABI for it. We could call this "Fedora/RISCV.Next". RV64GC will
>  > probably be 10-15 years old by that time, and it lacks any modern ISA
>  > especially if you want high performance. I consider RVA24 a point
>  > where we close major gaps between RISCV and other popular arches.
>  
>  Wouldn't changing -mabi effectively make the result a new Fedora
>  architecture? IIUC, binaries built with -mabi=lp64d wouldn't be able
>  to load libraries built with -mabi=lp64dv and vice versa.
>  
>  If that's correct, then we can't simply have a single "riscv64"
>  architecture: instead, we need to call what we have today
>  riscv64_lp64d, and be ready for riscv64_lp64dv as well as whatever
>  comes next. 
There's no point having different package names like this because how
would it help?  You're not going to parallel install them.

If we ever wanted to do this we're going to have to coordinate a
rebuild across the whole distribution.  For this reason I don't see
how either having separate /usr/lib64/<subarch> or different RPM
package names helps at all.

Rich.

>  It would be somewhat similar to existing architectures that can be
>  used in both Little-Endian (ppc64le) and Big-Endian (ppc64) modes.
>  
>  This is quite different from just bumping the ISA baseline, where
>  existing binaries and libraries are still usable in the new
>  environment.
>  
>  -- 
>  Andrea Bolognani / Red Hat / Virtualization
>  --
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Florian Weimer]

* Andrea Bolognani:

>  On Fri, Apr 19, 2024 at 02:21:57PM +0200, Florian Weimer wrote:
> > There are multiple PRs and patches floating around that make RISC-V use
> > the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
> > recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
> > various upstream projects follow that.
> >
> > I think we should follow upstream, so that it's possible to use Fedora
> > to do upstream development without patching the sources, or elaborate
> > Fedora-specific configure invocations.  The other reasons is to
> > future-proof the Fedora port against the arrival of an alternative ABI
> > that is not fully backwards-compatible (the same reason why the official
> > RISC-V documentation requires use of these paths).
>
>  I just checked in a Debian riscv64 chroot and they don't seem to
>  follow this recommendation:
>
>    # cat /etc/ld.so.conf.d/riscv64-linux-gnu.conf
>    /usr/local/lib/riscv64-linux-gnu
>    /lib/riscv64-linux-gnu
>    /usr/lib/riscv64-linux-gnu
>
>  This matches what Debian does on all architectures, that is, install
>  libraries under fully arch-qualified paths. If Debian doesn't stray
>  from its usual practices for RISC-V, I'm not convinced that Fedora
>  needs to either. 
Debian is really not a good example here because they have not
upstreamed their path layout.

I really don't think it's a good idea to clutter dozens of spec files
with changes for what's arguably just a toy architecture today, with
plenty of CPU-incompatible changes still coming.

Thanks,
Florian

[Andreas K. Huettel]

Hi, 

I'm handling the RISC-V libdirs in Gentoo.

>  There are multiple PRs and patches floating around that make RISC-V
> use
>  the /usr/lib64 directory, like other 64-bit ports.  However, RISC-V
>  recommends to use /usr/lib64/lp64d for the Fedora ABI variant, and
>  various upstream projects follow that. 
From memory I think as per spec both variants are valid.
For more than one ABI, only the latter makes sense of course.

(It gets more complicated for 32bit, because there it's /usr/lib versus
/usr/lib32/ilp32d ...)

Just for your information, here's how Gentoo is doing things to stay compatible
with as many variants as possible:

1) Non-multilib setups, 64bit 
   /usr/lib64

2) Non-multilib setups, 32bit:
   /usr/lib

3) Multilib setups (example with primary ABI lp64d):
   * The *primary* ABI is using the non-multilib path, to keep binary compatibility
        /usr/lib64
   * All *other* ABI use the multilib, two-level paths:
        /usr/lib64/lp64
        /usr/lib32/ilp32d
        /usr/lib32/ilp32
   * The two-level path of the *primary* ABI is a symlink to the non-multilib path.
        /usr/lib64/lp64d => /usr/lib64
        (or equivalent, /usr/lib64/lp64d => . )

Installing the primary ABI into a two-level libdir is something we tried in the past, 
but it lead to some pain. There are too many build systems out there that assume
that $(get_libdir) has one path level only, and then use expressions like (particularly
stupid invented example)

CONFIG_FILE=/usr/$(get_libdir)/../../etc/myconfig.conf

which leads to chaos once $(get_libdir) outputs "lib64/lp64d" ...

Cheers, 
Andreas
(on vacation, expect >24h response times)

>  I think we should follow upstream, so that it's possible to use
> Fedora
>  to do upstream development without patching the sources, or elaborate
>  Fedora-specific configure invocations.  The other reasons is to
>  future-proof the Fedora port against the arrival of an alternative ABI
>  that is not fully backwards-compatible (the same reason why the official
>  RISC-V documentation requires use of these paths). 
-- 
Andreas K. Hüttel
dilfridge(a)gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[Andreas K. Huettel]

>  
>  Just for your information, here's how Gentoo is doing things to stay compatible
>  with as many variants as possible:
>   
PS. Stage files are here:
https://www.gentoo.org/downloads/#riscv

(rv32-ilp32 is still building, rv32 musl is waiting until the worst musl-1.2.5 issues
are ironed out...)

-- 
Andreas K. Hüttel
dilfridge(a)gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge