Hi,
what did recently change in rawhide that the %{_buildrootdir} macro isn't expanded as in
+ mkdir '%{_buildrootdir}/bin' mkdir: cannot create directory ‘%{_buildrootdir}/bin’: No such file or directory + : + cp /builddir/build/SOURCES/node-stdout-nonblocking-wrapper '%{_buildrootdir}/bin' cp: cannot create regular file '%{_buildrootdir}/bin': No such file or directory
and
echo 'export NODEJS="%{_buildrootdir}/bin/node-stdout-nonblocking-wrapper"' >> .mozconfig
later resulting in literal
0:09.50 FileNotFoundError: [Errno 2] No such file or directory: '%{_buildrootdir}/bin/node-stdout-nonblocking-wrapper'
?
Eike
On Mon, 3 Jun 2024 14:22:37 +0200 Eike Rathke erack@redhat.com wrote:
Hi,
what did recently change in rawhide that the %{_buildrootdir} macro isn't expanded as in
new rpm landed? But shouldn't %{buildroot} be used instead of %{_buildrootdir} anyway?
Dan
- mkdir '%{_buildrootdir}/bin'
mkdir: cannot create directory ‘%{_buildrootdir}/bin’: No such file or directory
- :
- cp /builddir/build/SOURCES/node-stdout-nonblocking-wrapper '%{_buildrootdir}/bin'
cp: cannot create regular file '%{_buildrootdir}/bin': No such file or directory
and
echo 'export NODEJS="%{_buildrootdir}/bin/node-stdout-nonblocking-wrapper"' >> .mozconfig
later resulting in literal
0:09.50 FileNotFoundError: [Errno 2] No such file or directory: '%{_buildrootdir}/bin/node-stdout-nonblocking-wrapper'
?
Eike
-- GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918 630B 6A6C D5B7 6563 2D3A
On 6/3/24 15:22, Eike Rathke wrote:
Hi,
what did recently change in rawhide that the %{_buildrootdir} macro isn't expanded as in
- mkdir '%{_buildrootdir}/bin'
mkdir: cannot create directory ‘%{_buildrootdir}/bin’: No such file or directory
- :
- cp /builddir/build/SOURCES/node-stdout-nonblocking-wrapper '%{_buildrootdir}/bin'
cp: cannot create regular file '%{_buildrootdir}/bin': No such file or directory
What changed is new rpm cleaned up dusty corners. I only learned of *this* dusty corner usage this morning.
%{_buildrootdir} is nothing packages should be referring to, in any circumstance really. It's a potentially shared directory among arbitrary packages (in the traditional it's ~/rpmbuild/BUILDROOT/ ) and putting anything there can conflict with other packages being built on the system. %{buildroot} is what packages should be referring to, or better yet, ${RPM_BUILD_ROOT}.
That said, these cases kinda appear to be something not intended to be packaged at all, just "I need to put this file somewhere". That's one of the many use-cases we introduced the intermediate %builddir in 4.20, under which the %{buildroot} itself is.
And with the new directory layout, there's no reason to break these usages, instead we can now support it safely: we'll just redefine %_buildrootdir to be the dirname of %{buildroot} and voila, these files end up in the per-build directory where they cannot conflict with other packages.
I hope to have a build for this tomorrow, we just need to address another related matter before it makes sense to touch this.
- Panu -
On 6/3/24 15:55, Panu Matilainen wrote:
On 6/3/24 15:22, Eike Rathke wrote:
Hi,
what did recently change in rawhide that the %{_buildrootdir} macro isn't expanded as in
- mkdir '%{_buildrootdir}/bin'
mkdir: cannot create directory ‘%{_buildrootdir}/bin’: No such file or directory
- :
- cp /builddir/build/SOURCES/node-stdout-nonblocking-wrapper
'%{_buildrootdir}/bin' cp: cannot create regular file '%{_buildrootdir}/bin': No such file or directory
What changed is new rpm cleaned up dusty corners. I only learned of *this* dusty corner usage this morning.
%{_buildrootdir} is nothing packages should be referring to, in any circumstance really. It's a potentially shared directory among arbitrary packages (in the traditional it's ~/rpmbuild/BUILDROOT/ ) and putting anything there can conflict with other packages being built on the system. %{buildroot} is what packages should be referring to, or better yet, ${RPM_BUILD_ROOT}.
That said, these cases kinda appear to be something not intended to be packaged at all, just "I need to put this file somewhere". That's one of the many use-cases we introduced the intermediate %builddir in 4.20, under which the %{buildroot} itself is.
And with the new directory layout, there's no reason to break these usages, instead we can now support it safely: we'll just redefine %_buildrootdir to be the dirname of %{buildroot} and voila, these files end up in the per-build directory where they cannot conflict with other packages.
I hope to have a build for this tomorrow, we just need to address another related matter before it makes sense to touch this.
FWIW, upstream PR is at https://github.com/rpm-software-management/rpm/pull/3139
- Panu -
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
%{_buildrootdir} is nothing packages should be referring to, in any circumstance really. It's a potentially shared directory among arbitrary packages (in the traditional it's ~/rpmbuild/BUILDROOT/ ) and putting anything there can conflict with other packages being built on the system. %{buildroot} is what packages should be referring to,
So using %{buildroot} instead of %{_buildrootdir} would be an actual replacement?
or better yet, ${RPM_BUILD_ROOT}.
Why better?
That said, these cases kinda appear to be something not intended to be packaged at all, just "I need to put this file somewhere". That's one of the many use-cases we introduced the intermediate %builddir in 4.20, under which the %{buildroot} itself is.
But that's _only_ available as of 4.20, so not in f39,f40, correct?
Eike
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
%{_buildrootdir} is nothing packages should be referring to, in any circumstance really. It's a potentially shared directory among arbitrary packages (in the traditional it's ~/rpmbuild/BUILDROOT/ ) and putting anything there can conflict with other packages being built on the system. %{buildroot} is what packages should be referring to,
So using %{buildroot} instead of %{_buildrootdir} would be an actual replacement?
Depends on the case. If, as I suspect below and you confirm in the other email this was never meant to be packaged, then %{buildroot} of course is not the right place.
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
That said, these cases kinda appear to be something not intended to be packaged at all, just "I need to put this file somewhere". That's one of the many use-cases we introduced the intermediate %builddir in 4.20, under which the %{buildroot} itself is.
But that's _only_ available as of 4.20, so not in f39,f40, correct?
Yes. The next best thing would be using %{_builddir}/%{buildsubdir}, aka $PWD of the build scriptlets, this is available in all rpm versions for packages using %setup.
- Panu -
Eike
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
And I have not checked the RPM documentation, but I think that the env variables are underdocumented in the FPG.
Vít
Dne 04. 06. 24 v 9:27 Vít Ondruch napsal(a):
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
And I have not checked the RPM documentation
There is this section:
https://rpm-software-management.github.io/rpm/manual/spec.html#build-scriptl...
also recommending RPM_ macros for scriptlets:
Note: many of these have macro counterparts which may seem more
convenient and consistent with the rest of the spec, but one should always use the environment variables inside the scripts. The reason for this is that macros are evaluated during spec parse and may not be up-to-date, whereas environment variables are evaluated at the time of their execution in the script.
Vít
, but I think that the env variables are underdocumented in the FPG.
Vít
On Tue, Jun 04, 2024 at 09:31:47AM +0200, Vít Ondruch wrote:
Dne 04. 06. 24 v 9:27 Vít Ondruch napsal(a):
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
And I have not checked the RPM documentation
There is this section:
https://rpm-software-management.github.io/rpm/manual/spec.html#build-scriptl...
also recommending RPM_ macros for scriptlets:
I acknowledge what Panu wrote, but I think there are also countervailing reasons to prefer the macro: - it is shorter and generally more consistent with the rest of the spec file, which will have many many macros and only occasionally a shell variable, so the macro is more aesthetic. - but the big thing is that the macro is safe wrt. typos, while the variable not so much. It's just too easy to make a stupid typo in the variable name and do bad things™ in a local build.
${RPM_BUILD_ROOT:?} would be a better way to spell the variable reference, but that's too long expect people to use it.
So maybe we should have a macro that would expand to the env var: %global BUILDROOT ${RPM_BUILD_ROOT:?} and recommend that people use that instead?
Zbyszek
On 6/5/24 18:22, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Jun 04, 2024 at 09:31:47AM +0200, Vít Ondruch wrote:
Dne 04. 06. 24 v 9:27 Vít Ondruch napsal(a):
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
And I have not checked the RPM documentation
There is this section:
https://rpm-software-management.github.io/rpm/manual/spec.html#build-scriptl...
also recommending RPM_ macros for scriptlets:
I acknowledge what Panu wrote, but I think there are also countervailing reasons to prefer the macro:
it is shorter and generally more consistent with the rest of the spec file, which will have many many macros and only occasionally a shell variable, so the macro is more aesthetic.
but the big thing is that the macro is safe wrt. typos, while the variable not so much. It's just too easy to make a stupid typo in the variable name and do bad things™ in a local build.
${RPM_BUILD_ROOT:?} would be a better way to spell the variable reference, but that's too long expect people to use it.
Yeah the thing is those macros can and will never go away because everybody instinctively prefers them for consistency within the spec, shorter to type and all.
So maybe we should have a macro that would expand to the env var: %global BUILDROOT ${RPM_BUILD_ROOT:?} and recommend that people use that instead?
That'd be a third variant of the same thing, probably causing even more confusion, and specs using that would be incompatible with everything else. Let's not.
Making the macros expand to the corresponding environment variables is a is a sound strategy though, and what we're doing with %_smp_mflags already:
%_smp_mflags -j${RPM_BUILD_NCPUS}
It can cause some breakage though so care needs to be taken. And just now, we've already rocked the boat more than is entirely healthy for a single release, so further experiments in this area will have to wait for some other time.
- Panu -
- Panu -
On Thu, Jun 06, 2024 at 09:53:48AM +0300, Panu Matilainen wrote:
On 6/5/24 18:22, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Jun 04, 2024 at 09:31:47AM +0200, Vít Ondruch wrote:
Dne 04. 06. 24 v 9:27 Vít Ondruch napsal(a):
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
> or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
And I have not checked the RPM documentation
There is this section:
https://rpm-software-management.github.io/rpm/manual/spec.html#build-scriptl...
also recommending RPM_ macros for scriptlets:
I acknowledge what Panu wrote, but I think there are also countervailing reasons to prefer the macro:
it is shorter and generally more consistent with the rest of the spec file, which will have many many macros and only occasionally a shell variable, so the macro is more aesthetic.
but the big thing is that the macro is safe wrt. typos, while the variable not so much. It's just too easy to make a stupid typo in the variable name and do bad things™ in a local build.
${RPM_BUILD_ROOT:?} would be a better way to spell the variable reference, but that's too long expect people to use it.
Yeah the thing is those macros can and will never go away because everybody instinctively prefers them for consistency within the spec, shorter to type and all.
So maybe we should have a macro that would expand to the env var: %global BUILDROOT ${RPM_BUILD_ROOT:?} and recommend that people use that instead?
That'd be a third variant of the same thing, probably causing even more confusion, and specs using that would be incompatible with everything else. Let's not.
Making the macros expand to the corresponding environment variables is a is a sound strategy though, and what we're doing with %_smp_mflags already:
%_smp_mflags -j${RPM_BUILD_NCPUS}
It can cause some breakage though so care needs to be taken. And just now, we've already rocked the boat more than is entirely healthy for a single release, so further experiments in this area will have to wait for some other time.
I don't think we could ever change %buildroot to be ${RPM_BUILD_ROOT:?}, because the variable may be used in context where shell variable expansion is not supported… (E.g. a trivial "find '%{buildroot}' -name '*.a' -delete")
Zbyszek
V Thu, Jun 06, 2024 at 08:36:25AM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
On Thu, Jun 06, 2024 at 09:53:48AM +0300, Panu Matilainen wrote:
On 6/5/24 18:22, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Jun 04, 2024 at 09:31:47AM +0200, Vít Ondruch wrote:
Dne 04. 06. 24 v 9:27 Vít Ondruch napsal(a):
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote: > Hi Panu, > > On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote: > > > or better yet, ${RPM_BUILD_ROOT}. > > Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
And I have not checked the RPM documentation
There is this section:
https://rpm-software-management.github.io/rpm/manual/spec.html#build-scriptl...
also recommending RPM_ macros for scriptlets:
I acknowledge what Panu wrote, but I think there are also countervailing reasons to prefer the macro:
it is shorter and generally more consistent with the rest of the spec file, which will have many many macros and only occasionally a shell variable, so the macro is more aesthetic.
but the big thing is that the macro is safe wrt. typos, while the variable not so much. It's just too easy to make a stupid typo in the variable name and do bad things™ in a local build.
${RPM_BUILD_ROOT:?} would be a better way to spell the variable reference, but that's too long expect people to use it.
Yeah the thing is those macros can and will never go away because everybody instinctively prefers them for consistency within the spec, shorter to type and all.
So maybe we should have a macro that would expand to the env var: %global BUILDROOT ${RPM_BUILD_ROOT:?} and recommend that people use that instead?
That'd be a third variant of the same thing, probably causing even more confusion, and specs using that would be incompatible with everything else. Let's not.
Making the macros expand to the corresponding environment variables is a is a sound strategy though, and what we're doing with %_smp_mflags already:
%_smp_mflags -j${RPM_BUILD_NCPUS}
It can cause some breakage though so care needs to be taken. And just now, we've already rocked the boat more than is entirely healthy for a single release, so further experiments in this area will have to wait for some other time.
I don't think we could ever change %buildroot to be ${RPM_BUILD_ROOT:?}, because the variable may be used in context where shell variable expansion is not supported… (E.g. a trivial "find '%{buildroot}' -name '*.a' -delete")
Moreover, the scripts can be interpreted with a different shell than bash. A shell where ${RPM_BUILD_ROOT:?} is invalid or has a different meaning.
That's actually a reason why to prefer manually typing shell variables in the scripts: Shells know how to safely expand variables. rpm-build cannot know how to escape the expanded value.
-- Petr
* Vít Ondruch:
Dne 04. 06. 24 v 8:11 Panu Matilainen napsal(a):
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
I wish this nugget was captured somewhere on more prominent place. Because what you say does not really correspond with what we have in guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_using_buildroot_...
That's a bit unfortunate because %{optflags} and $RPM_OPT_FLAGS are necessary language-agnostic, which makes them less precise. And descripting %{optflags} as optimization flags has been misleading for around twenty years now.
Thanks, Florian
for the record: It seems there are only a few packages using it: https://sourcegraph.com/search?q=context:global+repo:%5Esrc.fedoraproject.or...
--
Michal Schorm Software Engineer Core Services - Databases Team Red Hat
--
On Tue, Jun 4, 2024 at 8:12 AM Panu Matilainen pmatilai@redhat.com wrote:
On 6/3/24 17:18, Eike Rathke wrote:
Hi Panu,
On Monday, 2024-06-03 15:55:09 +0300, Panu Matilainen wrote:
%{_buildrootdir} is nothing packages should be referring to, in any circumstance really. It's a potentially shared directory among arbitrary packages (in the traditional it's ~/rpmbuild/BUILDROOT/ ) and putting anything there can conflict with other packages being built on the system. %{buildroot} is what packages should be referring to,
So using %{buildroot} instead of %{_buildrootdir} would be an actual replacement?
Depends on the case. If, as I suspect below and you confirm in the other email this was never meant to be packaged, then %{buildroot} of course is not the right place.
or better yet, ${RPM_BUILD_ROOT}.
Why better?
In general, the RPM_* environment variables available to build scriptlets are what should be used instead of macros. Because, macros are processed while parsing the spec, which is different from actually *building* it. For one, using the environment improves srpm reproducibility because the local gunk like number of CPUs, the concrete path of %buildroot etc are only visible the scriptlets where actually used.
It's a subtle thing, took me 10+ years with rpm to grok the recommendation I'd seen long, long, long ago.
That said, these cases kinda appear to be something not intended to be packaged at all, just "I need to put this file somewhere". That's one of the many use-cases we introduced the intermediate %builddir in 4.20, under which the %{buildroot} itself is.
But that's _only_ available as of 4.20, so not in f39,f40, correct?
Yes. The next best thing would be using %{_builddir}/%{buildsubdir}, aka $PWD of the build scriptlets, this is available in all rpm versions for packages using %setup.
- Panu -
Eike
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue