rich deps result in packages being uninstalled from buildroot

[Zbigniew Jędrzejewski-Szmek]

Hi,

I've been trying to get 'add-determinism' deployed in buildroots. This
has been unsuccessful because of the following issue.

The dependency chain is:
redhat-rpm-config has
  Requires build-reproducibility-srpm-macros
and build-reproducibility-srpm-macros has
  Requires:    (add-determinism if python3-libs else add-determinism-nopython)
  Suggests:    add-determinism-nopython

(The idea is that we install 'add-determinism-nopython' which is self-contained,
but if python3 is installed into the buildroot, we pull in the heavier version
which has a dependency on python3-libs.)

This works well enough when installing packages using dnf on a test system.
But, in koji and mock builds, this results in rpm-build being unistalled (!).

For example, see https://koji.fedoraproject.org/koji/taskinfo?taskID=117684626
and root.log there: first rpm-build is installed along with a bunch of other
packages expected in the buildroot, but then cargo-rpm-macros is requested
(presumably via %generate_buildrequires), which depends on python3 and
python3-libs.

Dnf5 realizes that to satisfy all bounds, it can either:
a) install cargo-rpm-macros, python3, python3-libs, add-determinism, and remove
add-determinism-nopython
b) install cargo-rpm-macros, python3, python3-libs, and remove
build-reproducibility-srpm-macros,
   rpm-build, fonts-srpm-macros, redhat-rpm-config, and a few other packages.
and picks option b).

Without rpm-build installed, the build immediately faceplants.

(This can be reproduced by calling 'mock -i redhat-rpm-config' and
later 'mock -i cargo-rpm-macros'. It reproduces with both dnf5 and
dnf, so it's not a dnf5-related regresion.)

How to make this not happen, i.e. how to prevent rpm-build and other
packages that were explicitly pulled in via the @buildsys-build group
from being uninstalled? I think this is a general problem, not just
limited to the case described above. We now have hundreds of rich
Requires deps declared in packages, and each one creates the
possibility that dnf might opt to uninstall the package involved in
the rich dep, rather than installing the deps, if there is no
mechanism to prevent previously-explicitly-requested packges from
being removed.

Zbyszek

[Fabio Valentini]

On Thu, May 16, 2024 at 9:25 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  Hi,
>
>  I've been trying to get 'add-determinism' deployed in buildroots. This
>  has been unsuccessful because of the following issue.
>
>  The dependency chain is:
>  redhat-rpm-config has
>    Requires build-reproducibility-srpm-macros
>  and build-reproducibility-srpm-macros has
>    Requires:    (add-determinism if python3-libs else add-determinism-nopython)
>    Suggests:    add-determinism-nopython
>
>  (The idea is that we install 'add-determinism-nopython' which is self-contained,
>  but if python3 is installed into the buildroot, we pull in the heavier version
>  which has a dependency on python3-libs.)
>
>  This works well enough when installing packages using dnf on a test system.
>  But, in koji and mock builds, this results in rpm-build being unistalled (!).
>
>  For example, see https://koji.fedoraproject.org/koji/taskinfo?taskID=117684626
>  and root.log there: first rpm-build is installed along with a bunch of other
>  packages expected in the buildroot, but then cargo-rpm-macros is requested
>  (presumably via %generate_buildrequires), which depends on python3 and
>  python3-libs.
>
>  Dnf5 realizes that to satisfy all bounds, it can either:
>  a) install cargo-rpm-macros, python3, python3-libs, add-determinism, and remove
> add-determinism-nopython
>  b) install cargo-rpm-macros, python3, python3-libs, and remove
> build-reproducibility-srpm-macros,
>     rpm-build, fonts-srpm-macros, redhat-rpm-config, and a few other packages.
>  and picks option b). 
This looks like you're putting the resolver between a rock and a hard
place. :thinking:
I don't think I've ever seen packages being *removed* when installing
BuildRequires on top of the minimal buildroot ...

Would it be possible to adapt the packages so that add-determinism and
add-determinism-nopython are parallel-installable, and have the macro
fall back to the add-determinism-nopython executable if the
add-determinism executable is not available?
That way BuildRequires are additive and wouldn't force package removal
from the buildroot, and the rich dependency could be simpler - i.e.
`Requires: (add-determinism if python3-libs)`, without Suggests or
else branch.

Fabio

[Neal Gompa]

On Thu, May 16, 2024 at 3:10 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Thu, May 16, 2024 at 9:25 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > Hi,
>  >
>  > I've been trying to get 'add-determinism' deployed in buildroots. This
>  > has been unsuccessful because of the following issue.
>  >
>  > The dependency chain is:
>  > redhat-rpm-config has
>  >   Requires build-reproducibility-srpm-macros
>  > and build-reproducibility-srpm-macros has
>  >   Requires:    (add-determinism if python3-libs else add-determinism-nopython)
>  >   Suggests:    add-determinism-nopython
>  >
>  > (The idea is that we install 'add-determinism-nopython' which is
> self-contained,
>  > but if python3 is installed into the buildroot, we pull in the heavier version
>  > which has a dependency on python3-libs.)
>  >
>  > This works well enough when installing packages using dnf on a test system.
>  > But, in koji and mock builds, this results in rpm-build being unistalled (!).
>  >
>  > For example, see https://koji.fedoraproject.org/koji/taskinfo?taskID=117684626
>  > and root.log there: first rpm-build is installed along with a bunch of other
>  > packages expected in the buildroot, but then cargo-rpm-macros is requested
>  > (presumably via %generate_buildrequires), which depends on python3 and
>  > python3-libs.
>  >
>  > Dnf5 realizes that to satisfy all bounds, it can either:
>  > a) install cargo-rpm-macros, python3, python3-libs, add-determinism, and remove
> add-determinism-nopython
>  > b) install cargo-rpm-macros, python3, python3-libs, and remove
> build-reproducibility-srpm-macros,
>  >    rpm-build, fonts-srpm-macros, redhat-rpm-config, and a few other packages.
>  > and picks option b).
>
>  This looks like you're putting the resolver between a rock and a hard
>  place. :thinking:
>  I don't think I've ever seen packages being *removed* when installing
>  BuildRequires on top of the minimal buildroot ...
>
>  Would it be possible to adapt the packages so that add-determinism and
>  add-determinism-nopython are parallel-installable, and have the macro
>  fall back to the add-determinism-nopython executable if the
>  add-determinism executable is not available?
>  That way BuildRequires are additive and wouldn't force package removal
>  from the buildroot, and the rich dependency could be simpler - i.e.
>  `Requires: (add-determinism if python3-libs)`, without Suggests or
>  else branch. 
I have the question of why is dnf5 running as if "--allow-erasing" is
always passed to it? Older versions of DNF explicitly didn't do that
because we get weird behaviors like this.




--
真実はいつも一つ！/ Always, there's only one truth!

[Petr Pisar]

V Thu, May 16, 2024 at 04:29:21AM -0600, Neal Gompa napsal(a):
>  On Thu, May 16, 2024 at 3:10 AM Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  >
>  > On Thu, May 16, 2024 at 9:25 AM Zbigniew Jędrzejewski-Szmek
>  > <zbyszek(a)in.waw.pl&gt; wrote:
>  > >
>  > > Hi,
>  > >
>  > > I've been trying to get 'add-determinism' deployed in buildroots.
> This
>  > > has been unsuccessful because of the following issue.
>  > >
>  > > The dependency chain is:
>  > > redhat-rpm-config has
>  > >   Requires build-reproducibility-srpm-macros
>  > > and build-reproducibility-srpm-macros has
>  > >   Requires:    (add-determinism if python3-libs else add-determinism-nopython)
>  > >   Suggests:    add-determinism-nopython
>  > >
>  > > (The idea is that we install 'add-determinism-nopython' which is
> self-contained,
>  > > but if python3 is installed into the buildroot, we pull in the heavier version
>  > > which has a dependency on python3-libs.)
>  > >
>  > > This works well enough when installing packages using dnf on a test system.
>  > > But, in koji and mock builds, this results in rpm-build being unistalled (!).
>  > >
>  > > For example, see https://koji.fedoraproject.org/koji/taskinfo?taskID=117684626
>  > > and root.log there: first rpm-build is installed along with a bunch of other
>  > > packages expected in the buildroot, but then cargo-rpm-macros is requested
>  > > (presumably via %generate_buildrequires), which depends on python3 and
>  > > python3-libs.
>  > >
>  > > Dnf5 realizes that to satisfy all bounds, it can either:
>  > > a) install cargo-rpm-macros, python3, python3-libs, add-determinism, and remove
> add-determinism-nopython
>  > > b) install cargo-rpm-macros, python3, python3-libs, and remove
> build-reproducibility-srpm-macros,
>  > >    rpm-build, fonts-srpm-macros, redhat-rpm-config, and a few other packages.
>  > > and picks option b).
>  >
>  > This looks like you're putting the resolver between a rock and a hard
>  > place. :thinking:
>  > I don't think I've ever seen packages being *removed* when installing
>  > BuildRequires on top of the minimal buildroot ...
>  >
>  > Would it be possible to adapt the packages so that add-determinism and
>  > add-determinism-nopython are parallel-installable, and have the macro
>  > fall back to the add-determinism-nopython executable if the
>  > add-determinism executable is not available?
>  > That way BuildRequires are additive and wouldn't force package removal
>  > from the buildroot, and the rich dependency could be simpler - i.e.
>  > `Requires: (add-determinism if python3-libs)`, without Suggests or
>  > else branch.
>  
>  I have the question of why is dnf5 running as if "--allow-erasing" is
>  always passed to it? Older versions of DNF explicitly didn't do that
>  because we get weird behaviors like this.
>   
It's running as if --allow-erasing is passed because --allow-erasing was
passed:

DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q',
'-M', 'de96c7c84e6d430584a7795d75824e1d', '-D',
'/var/lib/mock/f41-build-50967284-6084889-bootstrap/root', '-a',
'--capability=cap_ipc_lock',
'--bind=/tmp/mock-resolv.1old43uf:/etc/resolv.conf', '--console=pipe',
'--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash',
'--setenv=HOME=/var/lib/mock/f41-build-50967284-6084889/root/installation-homedir',
'--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin',
'--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"',
'--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8',
'--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off',
'/usr/bin/dnf5', 'builddep', '--installroot',
'/var/lib/mock/f41-build-50967284-6084889/root/',
'/var/lib/mock/f41-build-50967284-6084889/root//builddir/build/SRPMS/rust-proc-macro2-diagnostics-0.10.1-1.fc41.src.rpm',
'--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes',
'--allowerasing', '--setopt=tsflags=nocontexts'] with env {'TERM':
'vt100', 'SHELL': '/bin/bash', 'HOME':
'/var/lib/mock/f41-build-50967284-6084889/root/installation-homedir',
'HOSTNAME': 'mock', 'PATH':
'/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf
"\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot>
\\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES':
'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0',
'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:463:  Updating and loading repositories:
DEBUG util.py:463:   build                                  100% | 118.4 KiB/s |   3.8 KiB
|  00m00s
DEBUG util.py:463:  Repositories loaded.
DEBUG util.py:463:  Package                            Arch   Version                
Repository      Size
DEBUG util.py:463:  Removing dependent packages:                                          
               
DEBUG util.py:463:   build-reproducibility-srpm-macros noarch 0.2.0-6.fc41           
build      684.0   B
DEBUG util.py:463:   fonts-srpm-macros                 noarch 1:2.0.5-14.fc40        
build       55.3 KiB
DEBUG util.py:463:   forge-srpm-macros                 noarch 0.3.1-1.fc41           
build       39.0 KiB
DEBUG util.py:463:   go-srpm-macros                    noarch 3.6.0-1.fc41           
build       60.8 KiB
DEBUG util.py:463:   python-srpm-macros                noarch 3.12-9.fc41            
build       50.5 KiB
DEBUG util.py:463:   redhat-rpm-config                 noarch 289-1.fc41             
build      183.5 KiB
DEBUG util.py:463:   rpm-build                         x86_64 4.19.1.1-2.fc41        
build      173.7 KiB

The reason why builddep is invoked with --allow-erasing is building packages
whose dependencies are alternatives to packages of a default version and thus
conflicts with them. In these cases you need uninstall conflicting packages.

A workaround could be rpm-build or mock to register rpm-build package in
/etc/dnf/protected.d configuration files. Packages listed there are prevented
from removal no matter of --allow-erasing.

Proper solution is actually minimazing content of the minimal build root, or
if not possible, then making the packages nonconficting. In ideal world
--allow-erasing would not be needed because the minimal build root would only
contain packages essential for everybody.

-- Petr

[Zbigniew Jędrzejewski-Szmek]

On Thu, May 16, 2024 at 01:14:16PM +0200, Petr Pisar wrote:
>  Proper solution is actually minimazing content of the minimal build
> root 
Most of the packages in the buildroot are libraries, pulled in via
dependencies.

@buildsys-build group is:
Mandatory packages   : bash          # basic shell env
                     : bzip2         # source extraction
                     : coreutils     # basic shell env
                     : cpio          # source extraction
                     : diffutils     # source extraction
                     : fedora-release-common   # rpm environment
                     : findutils     # basic shell env
                     : gawk          # basic shell env
                     : glibc-minimal-langpack  # we want this to avoid other langpacks
                     : grep          # basic shell env
                     : gzip          # source extraction
                     : info
                     : patch         # source extraction
                     : redhat-rpm-config  # rpm environment
                     : rpm-build          # rpm environment
                     : sed           # basic shell env
                     : shadow-utils
                     : tar           # source extraction
                     : unzip         # source extraction
                     : util-linux    # basic shell env
                     : which         # basic shell env. (command -v is more portable, but
meh.)
                     : xz            # source extraction

Out of this list, I see only three candidates for removal:

1. info. This is presumably present for /usr/sbin/install-info.
   It's a small package (360kB) and probably not worth the hassle to
   remove. On my system, 472 rpms provide info files, and if we remove
   it from the default set, we'd need to patch a huge amount of
   packages to add it back to make the builds work.

2. util-linux. This could be replaced by util-linux-core. This
   has a bunch of deps too, so it'd save some. util-linux has 95
   binaries, and while _most_ of them have little use in a constrained
   build environment, probably a few are used somewhere. So this
   would need some investigation.

3. shadow-utils. It was added in:

   commit 79e1728c1b9e9e98d2e38b1286d90d596f233a56
   Author: Jesse Keating <jkeating(a)redhat.com&gt;
   Date:   Tue Jan 15 15:31:27 2008 +0000

    Make sure shadow-utils makes it into a buildroot, or else mock will fail

   It's been a while, and I doubt we actually need it. Maybe we could
   drop that?

Zbyszek

[Vít Ondruch]

Dne 16. 05. 24 v 14:28 Zbigniew Jędrzejewski-Szmek napsal(a):
>  On Thu, May 16, 2024 at 01:14:16PM +0200, Petr Pisar wrote:
> > Proper solution is actually minimazing content of the minimal build root
>  Most of the packages in the buildroot are libraries, pulled in via
>  dependencies.
>
>  @buildsys-build group is:
>  Mandatory packages   : bash          # basic shell env
>                        : bzip2         # source extraction
>                        : coreutils     # basic shell env
>                        : cpio          # source extraction
>                        : diffutils     # source extraction
>                        : fedora-release-common   # rpm environment
>                        : findutils     # basic shell env
>                        : gawk          # basic shell env
>                        : glibc-minimal-langpack  # we want this to avoid other langpacks
>                        : grep          # basic shell env
>                        : gzip          # source extraction
>                        : info
>                        : patch         # source extraction
>                        : redhat-rpm-config  # rpm environment
>                        : rpm-build          # rpm environment
>                        : sed           # basic shell env
>                        : shadow-utils
>                        : tar           # source extraction
>                        : unzip         # source extraction
>                        : util-linux    # basic shell env
>                        : which         # basic shell env. (command -v is more portable,
> but meh.)
>                        : xz            # source extraction
>  

And there is this proposal to reduce the amount of compression libraries:

https://github.com/rpm-software-management/rpm/issues/1396

Actually, since RPM now uses `rpmuncompress` to extract the archives, it 
could also make sense to e.g. extract this utility into separate 
sub-package and move the extraction utilities out from the 
@buildsys-build. That would not reduce the the buildroot size at first, 
but get the dependencies to the right place for the start.


Vít


>  Out of this list, I see only three candidates for removal:
>
>  1. info. This is presumably present for /usr/sbin/install-info.
>      It's a small package (360kB) and probably not worth the hassle to
>      remove. On my system, 472 rpms provide info files, and if we remove
>      it from the default set, we'd need to patch a huge amount of
>      packages to add it back to make the builds work.
>
>  2. util-linux. This could be replaced by util-linux-core. This
>      has a bunch of deps too, so it'd save some. util-linux has 95
>      binaries, and while _most_ of them have little use in a constrained
>      build environment, probably a few are used somewhere. So this
>      would need some investigation.
>
>  3. shadow-utils. It was added in:
>
>      commit 79e1728c1b9e9e98d2e38b1286d90d596f233a56
>      Author: Jesse Keating <jkeating(a)redhat.com&gt;
>      Date:   Tue Jan 15 15:31:27 2008 +0000
>
>       Make sure shadow-utils makes it into a buildroot, or else mock will fail
>
>      It's been a while, and I doubt we actually need it. Maybe we could
>      drop that?
>
>  Zbyszek
>  --
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>

[Panu Matilainen]

On 5/16/24 16:10, Vít Ondruch wrote:
>  
>  Dne 16. 05. 24 v 14:28 Zbigniew Jędrzejewski-Szmek napsal(a):
> > On Thu, May 16, 2024 at 01:14:16PM +0200, Petr Pisar wrote:
> >> Proper solution is actually minimazing content of the minimal build root
> > Most of the packages in the buildroot are libraries, pulled in via
> > dependencies.
> >
> > @buildsys-build group is:
> > Mandatory packages   : bash          # basic shell env
> >                       : bzip2         # source extraction
> >                       : coreutils     # basic shell env
> >                       : cpio          # source extraction
> >                       : diffutils     # source extraction
> >                       : fedora-release-common   # rpm environment
> >                       : findutils     # basic shell env
> >                       : gawk          # basic shell env
> >                       : glibc-minimal-langpack  # we want this to 
> > avoid other langpacks
> >                       : grep          # basic shell env
> >                       : gzip          # source extraction
> >                       : info
> >                       : patch         # source extraction
> >                       : redhat-rpm-config  # rpm environment
> >                       : rpm-build          # rpm environment
> >                       : sed           # basic shell env
> >                       : shadow-utils
> >                       : tar           # source extraction
> >                       : unzip         # source extraction
> >                       : util-linux    # basic shell env
> >                       : which         # basic shell env. (command -v 
> > is more portable, but meh.)
> >                       : xz            # source extraction
>  
>  
>  And there is this proposal to reduce the amount of compression libraries:
>  
>  https://github.com/rpm-software-management/rpm/issues/1396
>  
>  Actually, since RPM now uses `rpmuncompress` to extract the archives, it 
>  could also make sense to e.g. extract this utility into separate 
>  sub-package and move the extraction utilities out from the 
>  @buildsys-build. That would not reduce the the buildroot size at first, 
>  but get the dependencies to the right place for the start.
>   
Doesn't work.

rpmuncompress doesn't know how to uncompress ANYTHING AT ALL by itself. 
It only knows which command(s) to call to do that.

	- Panu -

[Vít Ondruch]

Dne 17. 05. 24 v 7:16 Panu Matilainen napsal(a):
>  On 5/16/24 16:10, Vít Ondruch wrote:
> >
> > Dne 16. 05. 24 v 14:28 Zbigniew Jędrzejewski-Szmek napsal(a):
> >> On Thu, May 16, 2024 at 01:14:16PM +0200, Petr Pisar wrote:
> >>> Proper solution is actually minimazing content of the minimal build 
> >>> root
> >> Most of the packages in the buildroot are libraries, pulled in via
> >> dependencies.
> >>
> >> @buildsys-build group is:
> >> Mandatory packages   : bash          # basic shell env
> >>                       : bzip2         # source extraction
> >>                       : coreutils     # basic shell env
> >>                       : cpio          # source extraction
> >>                       : diffutils     # source extraction
> >>                       : fedora-release-common   # rpm environment
> >>                       : findutils     # basic shell env
> >>                       : gawk          # basic shell env
> >>                       : glibc-minimal-langpack  # we want this to 
> >> avoid other langpacks
> >>                       : grep          # basic shell env
> >>                       : gzip          # source extraction
> >>                       : info
> >>                       : patch         # source extraction
> >>                       : redhat-rpm-config  # rpm environment
> >>                       : rpm-build          # rpm environment
> >>                       : sed           # basic shell env
> >>                       : shadow-utils
> >>                       : tar           # source extraction
> >>                       : unzip         # source extraction
> >>                       : util-linux    # basic shell env
> >>                       : which         # basic shell env. (command -v 
> >> is more portable, but meh.)
> >>                       : xz            # source extraction
> >
> >
> > And there is this proposal to reduce the amount of compression 
> > libraries:
> >
> > https://github.com/rpm-software-management/rpm/issues/1396
> >
> > Actually, since RPM now uses `rpmuncompress` to extract the archives, 
> > it could also make sense to e.g. extract this utility into separate 
> > sub-package and move the extraction utilities out from the 
> > @buildsys-build. That would not reduce the the buildroot size at 
> > first, but get the dependencies to the right place for the start.
> >
>
>  Doesn't work.
>
>  rpmuncompress doesn't know how to uncompress ANYTHING AT ALL by 
>  itself. It only knows which command(s) to call to do that.
>  

The point is that the group is not the right place to list the 
extraction utilities. The definition should be closer to RPM IMHO. Of 
course one might argue that distribution might want to have a word and 
avoid some compression algorithm, but it can't add anything what RPM 
somehow does not support.

And I didn't want to imply that rpmuncompress can extract some archive 
on its own. Although it can likely be read that way ...


Vít


>      - Panu -
>  -- 
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: 
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Sandro]

On 16-05-2024 13:14, Petr Pisar wrote:
>  A workaround could be rpm-build or mock to register rpm-build package
> in
>  /etc/dnf/protected.d configuration files. Packages listed there are prevented
>  from removal no matter of --allow-erasing. 
A bit late to the party, but I was wondering if making `add-determinism` 
and `add-determinism-nopython` require `rpm-build` would also achieve 
`rpm-build` being protected from removal as a workaround.

If either package requires it there should only be one way forward, if 
my understanding of the issue is correct.

-- Sandro

[Petr Pisar]

V Sat, May 18, 2024 at 08:20:53PM +0200, Sandro napsal(a):
>  On 16-05-2024 13:14, Petr Pisar wrote:
>  > A workaround could be rpm-build or mock to register rpm-build package in
>  > /etc/dnf/protected.d configuration files. Packages listed there are prevented
>  > from removal no matter of --allow-erasing.
>  
>  A bit late to the party, but I was wondering if making `add-determinism` and
>  `add-determinism-nopython` require `rpm-build` would also achieve
>  `rpm-build` being protected from removal as a workaround.
>  
>  If either package requires it there should only be one way forward, if my
>  understanding of the issue is correct.
>   
That's also a possible way. Many times defining the reverse dependendency can
be justified as (add-determinism) being a plugin (of rpm-build). It also helps
cleaning useless plugins (add-determinism) when the framework (rpm-build) is
uinstalled.

A drawback is creating dependency loop.

-- Petr

[Zbigniew Jędrzejewski-Szmek]

On Mon, May 20, 2024 at 10:55:58AM +0200, Petr Pisar wrote:
>  V Sat, May 18, 2024 at 08:20:53PM +0200, Sandro napsal(a):
>  > On 16-05-2024 13:14, Petr Pisar wrote:
>  > > A workaround could be rpm-build or mock to register rpm-build package in
>  > > /etc/dnf/protected.d configuration files. Packages listed there are prevented
>  > > from removal no matter of --allow-erasing.
>  > 
>  > A bit late to the party, but I was wondering if making `add-determinism` and
>  > `add-determinism-nopython` require `rpm-build` would also achieve
>  > `rpm-build` being protected from removal as a workaround.
>  > 
>  > If either package requires it there should only be one way forward, if my
>  > understanding of the issue is correct.
>  > 
>  That's also a possible way. Many times defining the reverse dependendency can
>  be justified as (add-determinism) being a plugin (of rpm-build). It also helps
>  cleaning useless plugins (add-determinism) when the framework (rpm-build) is
>  uinstalled.
>  
>  A drawback is creating dependency loop. 
Thank you for the suggestion. I think that with the current state
things are good, so I don't plan to change things, unless we discover
some breakage.

FWIW, add-determinism is fully functional without rpm-build… I expect
that most people who use it would have rpm-build installed, but
I think it's better to not create dependency loops to keep things
flexible.

Zbyszek

[Kevin Kofler]

Neal Gompa wrote:
>  I have the question of why is dnf5 running as if
> "--allow-erasing" is
>  always passed to it? Older versions of DNF explicitly didn't do that
>  because we get weird behaviors like this. 
Without --allow-erasing (which was actually passed explicitly, as Petr Pisar 
pointed out), the intended resolution:
>  a) install cargo-rpm-macros, python3, python3-libs, add-determinism,
> and
>  remove add-determinism-nopython  
could also not possibly work because:
>  remove add-determinism-nopython  
        Kevin Kofler

[Zbigniew Jędrzejewski-Szmek]

On Thu, May 16, 2024 at 11:09:01AM +0200, Fabio Valentini wrote:
>  This looks like you're putting the resolver between a rock and a
> hard
>  place. :thinking:
>  I don't think I've ever seen packages being *removed* when installing
>  BuildRequires on top of the minimal buildroot ...
>  
>  Would it be possible to adapt the packages so that add-determinism and
>  add-determinism-nopython are parallel-installable, and have the macro
>  fall back to the add-determinism-nopython executable if the
>  add-determinism executable is not available?
>  That way BuildRequires are additive and wouldn't force package removal
>  from the buildroot, and the rich dependency could be simpler - i.e.
>  `Requires: (add-determinism if python3-libs)`, without Suggests or
>  else branch. 
Yeah, I think this is the most reasonable approach.
I'll push a build that does this shortly.

(I tried to test this locally with mock, with the additional packages
in a local repo. When I install everything in one go, things work as
expected. Why I try to check the original issue, and install packages
piecemeal, mock downgrades packages to fall back to the koji version
without dependencies, instead of installing the additional deps. In
koji, it'll only have one version of the package available, so
hopefully it'll pick addition instead of removals.)

Zbyszek

[Paul Howarth]

On Thu, 16 May 2024 10:52:29 +0000
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; wrote:

>  On Thu, May 16, 2024 at 11:09:01AM +0200, Fabio Valentini wrote:
>  > This looks like you're putting the resolver between a rock and a
>  > hard place. :thinking:
>  > I don't think I've ever seen packages being *removed* when
>  > installing BuildRequires on top of the minimal buildroot ...
>  > 
>  > Would it be possible to adapt the packages so that add-determinism
>  > and add-determinism-nopython are parallel-installable, and have the
>  > macro fall back to the add-determinism-nopython executable if the
>  > add-determinism executable is not available?
>  > That way BuildRequires are additive and wouldn't force package
>  > removal from the buildroot, and the rich dependency could be
>  > simpler - i.e. `Requires: (add-determinism if python3-libs)`,
>  > without Suggests or else branch.  
>  
>  Yeah, I think this is the most reasonable approach.
>  I'll push a build that does this shortly.
>  
>  (I tried to test this locally with mock, with the additional packages
>  in a local repo. When I install everything in one go, things work as
>  expected. Why I try to check the original issue, and install packages
>  piecemeal, mock downgrades packages to fall back to the koji version
>  without dependencies, instead of installing the additional deps. In
>  koji, it'll only have one version of the package available, so
>  hopefully it'll pick addition instead of removals.) 
Not sure how this is supposed to work: add-determinism-nopython seems
to install %{_bindir}/add-determinism.nopython but the macro only seems 
to look for %{_bindir}/add-determinism, resulting in this when only
add-determinism-nopython is installed:

(https://koji.fedoraproject.org/koji/taskinfo?taskID=117791675)

+ /usr/bin/add-determinism --brp -j48
/builddir/build/BUILDROOT/perl-Finance-Quote-1.6200-1.fc41.noarch
/var/tmp/rpm-tmp.hdLEuR: line 89: /usr/bin/add-determinism: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.hdLEuR (%install)

The intent seems to be that the minimal version checks for existence of
the full version and re-execs it if found. So it seems to me that the
minimal version should be installed as %{_bindir}/add-determinism and
the full version should be called %{_bindir}/add-determinism.withpython
or something. Or maybe just tweak the rpm macro to define
%__brp_add_determinism as %{_bindir}/add-determinism.nopython.

Regards, Paul.

[Zbigniew Jędrzejewski-Szmek]

On Fri, May 17, 2024 at 09:43:22AM +0100, Paul Howarth wrote:
>  On Thu, 16 May 2024 10:52:29 +0000
>  Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; wrote:
>  
>  > On Thu, May 16, 2024 at 11:09:01AM +0200, Fabio Valentini wrote:
>  > > This looks like you're putting the resolver between a rock and a
>  > > hard place. :thinking:
>  > > I don't think I've ever seen packages being *removed* when
>  > > installing BuildRequires on top of the minimal buildroot ...
>  > > 
>  > > Would it be possible to adapt the packages so that add-determinism
>  > > and add-determinism-nopython are parallel-installable, and have the
>  > > macro fall back to the add-determinism-nopython executable if the
>  > > add-determinism executable is not available?
>  > > That way BuildRequires are additive and wouldn't force package
>  > > removal from the buildroot, and the rich dependency could be
>  > > simpler - i.e. `Requires: (add-determinism if python3-libs)`,
>  > > without Suggests or else branch.  
>  > 
>  > Yeah, I think this is the most reasonable approach.
>  > I'll push a build that does this shortly.
>  > 
>  > (I tried to test this locally with mock, with the additional packages
>  > in a local repo. When I install everything in one go, things work as
>  > expected. Why I try to check the original issue, and install packages
>  > piecemeal, mock downgrades packages to fall back to the koji version
>  > without dependencies, instead of installing the additional deps. In
>  > koji, it'll only have one version of the package available, so
>  > hopefully it'll pick addition instead of removals.)
>  
>  Not sure how this is supposed to work: add-determinism-nopython seems
>  to install %{_bindir}/add-determinism.nopython but the macro only seems 
>  to look for %{_bindir}/add-determinism, resulting in this when only
>  add-determinism-nopython is installed:
>  
>  (https://koji.fedoraproject.org/koji/taskinfo?taskID=117791675) 
Oh, this is embarrasing. I added two patches but they didn't get
applied because %setup not %autosetup. :(
I'll push a fixed build in a moment. Sorry for breaking the builds.

Zbyszek

[Zbigniew Jędrzejewski-Szmek]

On Fri, May 17, 2024 at 12:19:34PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Fri, May 17, 2024 at 09:43:22AM +0100, Paul Howarth wrote:
>  > On Thu, 16 May 2024 10:52:29 +0000
>  > Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; wrote:
>  > 
>  > > On Thu, May 16, 2024 at 11:09:01AM +0200, Fabio Valentini wrote:
>  > > > This looks like you're putting the resolver between a rock and a
>  > > > hard place. :thinking:
>  > > > I don't think I've ever seen packages being *removed* when
>  > > > installing BuildRequires on top of the minimal buildroot ...
>  > > > 
>  > > > Would it be possible to adapt the packages so that add-determinism
>  > > > and add-determinism-nopython are parallel-installable, and have the
>  > > > macro fall back to the add-determinism-nopython executable if the
>  > > > add-determinism executable is not available?
>  > > > That way BuildRequires are additive and wouldn't force package
>  > > > removal from the buildroot, and the rich dependency could be
>  > > > simpler - i.e. `Requires: (add-determinism if python3-libs)`,
>  > > > without Suggests or else branch.  
>  > > 
>  > > Yeah, I think this is the most reasonable approach.
>  > > I'll push a build that does this shortly.
>  > > 
>  > > (I tried to test this locally with mock, with the additional packages
>  > > in a local repo. When I install everything in one go, things work as
>  > > expected. Why I try to check the original issue, and install packages
>  > > piecemeal, mock downgrades packages to fall back to the koji version
>  > > without dependencies, instead of installing the additional deps. In
>  > > koji, it'll only have one version of the package available, so
>  > > hopefully it'll pick addition instead of removals.)
>  > 
>  > Not sure how this is supposed to work: add-determinism-nopython seems
>  > to install %{_bindir}/add-determinism.nopython but the macro only seems 
>  > to look for %{_bindir}/add-determinism, resulting in this when only
>  > add-determinism-nopython is installed:
>  > 
>  > (https://koji.fedoraproject.org/koji/taskinfo?taskID=117791675)
>  
>  Oh, this is embarrasing. I added two patches but they didn't get
>  applied because %setup not %autosetup. :(
>  I'll push a fixed build in a moment. Sorry for breaking the builds. 
perl-Finance-Quote-1.6200-1.fc41:
https://koji.fedoraproject.org/koji/taskinfo?taskID=117804013

Zbyszek

[Paul Howarth]

On Fri, 17 May 2024 13:32:24 +0000
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; wrote:
>  > Oh, this is embarrasing. I added two patches but they didn't
> get
>  > applied because %setup not %autosetup. :(
>  > I'll push a fixed build in a moment. Sorry for breaking the builds.
>  >  
>  
>  perl-Finance-Quote-1.6200-1.fc41:
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=117804013 
Thanks.