Automatic BuildRequires

[Richard W.M. Jones]

[I'm sure this isn't the first time this has occurred to someone, or
even been done -- but couldn't find anything for it in Google ...]

http://et.redhat.com/~rjones/auto-buildrequires/

This tries to find the correct set of BuildRequires automatically when
running 'rpmbuild'.  Just replace the ordinary rpmbuild command with
auto-br-rpmbuild, and it will print out a suggested set of
BuildRequires lines at the end.  For example:

$ auto-br-rpmbuild -ba mingw32-iconv.spec
[...]
Checking for unpackaged file(s): /usr/lib/rpm/check-files
/home/rjones/rpmbuild/BUILDROOT/mingw32-iconv-1.12-5.fc10.x86_64
Wrote: /home/rjones/rpmbuild/SRPMS/mingw32-iconv-1.12-5.fc10.src.rpm
Wrote: /home/rjones/rpmbuild/RPMS/noarch/mingw32-iconv-1.12-5.fc10.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.XpFhTF
+ umask 022
+ cd /home/rjones/rpmbuild/BUILD
+ cd libiconv-1.12
+ rm -rf /home/rjones/rpmbuild/BUILDROOT/mingw32-iconv-1.12-5.fc10.x86_64
+ exit 0
BuildRequires: bash = 3.2.29.fc10.x86_64
BuildRequires: binutils = 2.18.50.0.9.7.fc10.x86_64
BuildRequires: ccache = 2.4.13.fc9.x86_64
BuildRequires: coreutils = 6.12.16.fc10.x86_64
BuildRequires: diffutils = 2.8.1.21.fc9.x86_64
BuildRequires: file = 4.26.3.fc10.x86_64
BuildRequires: findutils = 1:4.4.0.1.fc10.x86_64
BuildRequires: gawk = 3.1.5.18.fc10.x86_64
BuildRequires: gcc-gfortran = 4.3.2.6.x86_64
BuildRequires: gettext = 0.17.8.fc10.x86_64
BuildRequires: glibc-common = 2.8.90.14.x86_64
BuildRequires: grep = 2.5.1a.61.fc10.x86_64
BuildRequires: gzip = 1.3.12.7.fc10.x86_64
BuildRequires: make = 1:3.81.14.fc10.x86_64
BuildRequires: mingw32-binutils = 2.18.50_20080109_2.8.fc9.x86_64
BuildRequires: mingw32-filesystem = 34.1.fc9.noarch
BuildRequires: mingw32-gcc-c++ = 4.3.2.8.fc9.x86_64
BuildRequires: mingw32-gcc = 4.3.2.8.fc9.x86_64
BuildRequires: mingw32-gettext = 0.17.6.fc9.noarch
BuildRequires: mingw32-iconv = 1.12.4.fc9.noarch
BuildRequires: mingw32-runtime = 3.15.1.1.fc9.noarch
BuildRequires: mingw32-w32api = 3.12.1.fc9.noarch
BuildRequires: net-tools = 1.60.91.fc10.x86_64
BuildRequires: sed = 4.1.5.10.fc9.x86_64
BuildRequires: tar = 2:1.20.3.fc10.x86_64

As you can see it's not perfect.  It doesn't ignore 'core' packages
which are always expected in a mock/koji build.  Also because autoconf
scripts tend to touch the C++ and Fortran compilers, even when they
are not used, it always suggests those packages.  Also it doesn't
exclude some tools like tar which are touched by rpmbuild itself.

The implementation is a simple LD_PRELOAD script that analyzes open(2)
and execve(2) system calls, a Perl script that does the analysis, and
some shell scripts to hang it all together.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top

[Ignacio Vazquez-Abrams]

On Thu, 2008-11-06 at 11:01 +0000, Richard W.M. Jones wrote:
>  [I'm sure this isn't the first time this has occurred to
> someone, or
>  even been done -- but couldn't find anything for it in Google ...]
>  
>  http://et.redhat.com/~rjones/auto-buildrequires/
>  
>  This tries to find the correct set of BuildRequires automatically when
>  running 'rpmbuild'.  Just replace the ordinary rpmbuild command with
>  auto-br-rpmbuild, and it will print out a suggested set of
>  BuildRequires lines at the end.  For example: 
I take it that it needs to be run in a system/root that already has the
appropriate packages installed?

-- 
Ignacio Vazquez-Abrams <ivazqueznet(a)gmail.com&gt;

PLEASE don't CC me; I'm already subscribed

[Richard W.M. Jones]

On Thu, Nov 06, 2008 at 06:15:38AM -0500, Ignacio Vazquez-Abrams wrote:
>  On Thu, 2008-11-06 at 11:01 +0000, Richard W.M. Jones wrote:
>  > [I'm sure this isn't the first time this has occurred to someone, or
>  > even been done -- but couldn't find anything for it in Google ...]
>  > 
>  > http://et.redhat.com/~rjones/auto-buildrequires/
>  > 
>  > This tries to find the correct set of BuildRequires automatically when
>  > running 'rpmbuild'.  Just replace the ordinary rpmbuild command with
>  > auto-br-rpmbuild, and it will print out a suggested set of
>  > BuildRequires lines at the end.  For example:
>  
>  I take it that it needs to be run in a system/root that already has the
>  appropriate packages installed? 
Yes, you just run it on a normal system.  The main idea is so you
don't have to keep submitting Koji jobs because you missed out some
BuildRequires.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Dennis Gilmore]

On Thursday 06 November 2008 05:16:50 am Richard W.M. Jones wrote:
>  On Thu, Nov 06, 2008 at 06:15:38AM -0500, Ignacio Vazquez-Abrams
> wrote:
>  > On Thu, 2008-11-06 at 11:01 +0000, Richard W.M. Jones wrote:
>  > > [I'm sure this isn't the first time this has occurred to someone, or
>  > > even been done -- but couldn't find anything for it in Google ...]
>  > >
>  > > http://et.redhat.com/~rjones/auto-buildrequires/
>  > >
>  > > This tries to find the correct set of BuildRequires automatically when
>  > > running 'rpmbuild'.  Just replace the ordinary rpmbuild command with
>  > > auto-br-rpmbuild, and it will print out a suggested set of
>  > > BuildRequires lines at the end.  For example:
>  >
>  > I take it that it needs to be run in a system/root that already has the
>  > appropriate packages installed?
>
>  Yes, you just run it on a normal system.  The main idea is so you
>  don't have to keep submitting Koji jobs because you missed out some
>  BuildRequires. 
Thats what mock on your local system is for.  Please don't abuse
koji by doing 
that.  if you have it right on one arch using scratch builds in koji to test 
all arches is appropriate. but not to work out missing BuildRequires.  Again 
please be thoughtful with the resources fedora provides  they are not infinite.

Dennis

[Richard W.M. Jones]

On Sun, Nov 09, 2008 at 03:49:42PM -0600, Dennis Gilmore wrote:
>  On Thursday 06 November 2008 05:16:50 am Richard W.M. Jones wrote:
>  > On Thu, Nov 06, 2008 at 06:15:38AM -0500, Ignacio Vazquez-Abrams wrote:
>  > > On Thu, 2008-11-06 at 11:01 +0000, Richard W.M. Jones wrote:
>  > > > [I'm sure this isn't the first time this has occurred to someone,
> or
>  > > > even been done -- but couldn't find anything for it in Google ...]
>  > > >
>  > > > http://et.redhat.com/~rjones/auto-buildrequires/
>  > > >
>  > > > This tries to find the correct set of BuildRequires automatically when
>  > > > running 'rpmbuild'.  Just replace the ordinary rpmbuild command
> with
>  > > > auto-br-rpmbuild, and it will print out a suggested set of
>  > > > BuildRequires lines at the end.  For example:
>  > >
>  > > I take it that it needs to be run in a system/root that already has the
>  > > appropriate packages installed?
>  >
>  > Yes, you just run it on a normal system.  The main idea is so you
>  > don't have to keep submitting Koji jobs because you missed out some
>  > BuildRequires.
>  Thats what mock on your local system is for. 
Or the same for mock ...

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Horst H. von Brand]

Dennis Gilmore <dennis(a)ausil.us&gt; wrote:

[...]

>  Thats what mock on your local system is for.  Please don't abuse
> koji by
>  doing that.  if you have it right on one arch using scratch builds in
>  koji to test all arches is appropriate. but not to work out missing
>  BuildRequires.  Again please be thoughtful with the resources fedora
>  provides they are not infinite. 
Last time I tried to use mock it spent 10 times as much resources setting
up the same environment over and over than doing useful work. Gave up.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                    Fono: [phone]
Universidad Tecnica Federico Santa Maria             [phone]
Casilla 110-V, Valparaiso, Chile 2340000       Fax:  [phone]

[Jarod Wilson]

On Tuesday 11 November 2008 14:48:15 Horst H. von Brand wrote:
>  Dennis Gilmore <dennis(a)ausil.us&gt; wrote:
>
>  [...]
>
>  > Thats what mock on your local system is for.  Please don't abuse koji by
>  > doing that.  if you have it right on one arch using scratch builds in
>  > koji to test all arches is appropriate. but not to work out missing
>  > BuildRequires.  Again please be thoughtful with the resources fedora
>  > provides they are not infinite.
>
>  Last time I tried to use mock it spent 10 times as much resources setting
>  up the same environment over and over than doing useful work. Gave up. 
*cough* mock --no-clean *cough*

-- 
Jarod Wilson
jarod(a)redhat.com

[Jesse Keating]

On Tue, 2008-11-11 at 16:48 -0300, Horst H. von Brand wrote:
>  
>  Last time I tried to use mock it spent 10 times as much resources setting
>  up the same environment over and over than doing useful work. Gave up. 
Use mock caching of packages, use caching of buildroots, use ramdisk
chroots, etc...

A local mock build once packages et al are cached is going to be far
faster than waiting for an upload to koji and a build in that
environment, because koji is using mock too.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Michael Schwendt]

On Thu, 6 Nov 2008 11:01:42 +0000, Richard W.M. Jones wrote:

>  [I'm sure this isn't the first time this has occurred to
> someone, or
>  even been done -- but couldn't find anything for it in Google ...]
>  
>  http://et.redhat.com/~rjones/auto-buildrequires/ 
There's one slightly similar one, which is several years old:

https://www.europe.redhat.com/documentation/rhl6.2/rhl_6.2_sw_repository/...

Download it here for example:
http://www.filewatcher.com/m/InDependence-1.0-6.src.rpm.16197.0.0.html

[Matt Domsch]

On Thu, Nov 06, 2008 at 07:23:02PM +0100, Michael Schwendt wrote:
>  On Thu, 6 Nov 2008 11:01:42 +0000, Richard W.M. Jones wrote:
>  
>  > [I'm sure this isn't the first time this has occurred to someone, or
>  > even been done -- but couldn't find anything for it in Google ...]
>  > 
>  > http://et.redhat.com/~rjones/auto-buildrequires/
>  
>  There's one slightly similar one, which is several years old:
>  
>
> https://www.europe.redhat.com/documentation/rhl6.2/rhl_6.2_sw_repository/...
>  
>  Download it here for example:
>  http://www.filewatcher.com/m/InDependence-1.0-6.src.rpm.16197.0.0.html 
this might  be useful on conjunction with the occasional "full rawhide
rebuilds using rpmbuild on a system with Everything installed" run
that someone (sorry, I forget who did this) does.  That has the added
benefit of picking up features that configure auto-discovers but if
missing BRs, won't discover.

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

[Matt Domsch]

On Sun, Nov 09, 2008 at 02:49:46PM -0600, Matt Domsch wrote:
>  On Thu, Nov 06, 2008 at 07:23:02PM +0100, Michael Schwendt wrote:
>  > On Thu, 6 Nov 2008 11:01:42 +0000, Richard W.M. Jones wrote:
>  > 
>  > > [I'm sure this isn't the first time this has occurred to someone, or
>  > > even been done -- but couldn't find anything for it in Google ...]
>  > > 
>  > > http://et.redhat.com/~rjones/auto-buildrequires/
>  > 
>  > There's one slightly similar one, which is several years old:
>  > 
>  >
> https://www.europe.redhat.com/documentation/rhl6.2/rhl_6.2_sw_repository/...
>  > 
>  > Download it here for example:
>  > http://www.filewatcher.com/m/InDependence-1.0-6.src.rpm.16197.0.0.html
>  
>  this might  be useful on conjunction with the occasional "full rawhide
>  rebuilds using rpmbuild on a system with Everything installed" run
>  that someone (sorry, I forget who did this) does.  That has the added
>  benefit of picking up features that configure auto-discovers but if
>  missing BRs, won't discover. 
It was Karsten Hopp back in March 2008.
https://www.redhat.com/archives/fedora-devel-list/2008-March/msg01257.html

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

[Karsten Hopp]

Matt Domsch schrieb:
>  On Sun, Nov 09, 2008 at 02:49:46PM -0600, Matt Domsch wrote:
> > On Thu, Nov 06, 2008 at 07:23:02PM +0100, Michael Schwendt wrote:
> >> On Thu, 6 Nov 2008 11:01:42 +0000, Richard W.M. Jones wrote:
> >>
> >>> [I'm sure this isn't the first time this has occurred to someone, or
> >>> even been done -- but couldn't find anything for it in Google ...]
> >>>
> >>> http://et.redhat.com/~rjones/auto-buildrequires/
> >> There's one slightly similar one, which is several years old:
> >>
> >>
> https://www.europe.redhat.com/documentation/rhl6.2/rhl_6.2_sw_repository/...
> >>
> >> Download it here for example:
> >> http://www.filewatcher.com/m/InDependence-1.0-6.src.rpm.16197.0.0.html
> > this might  be useful on conjunction with the occasional "full rawhide
> > rebuilds using rpmbuild on a system with Everything installed" run
> > that someone (sorry, I forget who did this) does.  That has the added
> > benefit of picking up features that configure auto-discovers but if
> > missing BRs, won't discover.
>  
>  It was Karsten Hopp back in March 2008.
>  https://www.redhat.com/archives/fedora-devel-list/2008-March/msg01257.html
>   
I've used auto-br-rpmbuild with my latest full-install mass rebuild and grepped the
BuildRequires from the logs.
They are available at
	http://karsten.fedorapeople.org/buildrequires/

[Richard W.M. Jones]

On Thu, Dec 18, 2008 at 01:41:01PM +0100, Karsten Hopp wrote:
>  Matt Domsch schrieb:
> > On Sun, Nov 09, 2008 at 02:49:46PM -0600, Matt Domsch wrote:
> >> On Thu, Nov 06, 2008 at 07:23:02PM +0100, Michael Schwendt wrote:
> >>> On Thu, 6 Nov 2008 11:01:42 +0000, Richard W.M. Jones wrote:
> >>>
> >>>> [I'm sure this isn't the first time this has occurred to someone,
> or
> >>>> even been done -- but couldn't find anything for it in Google ...]
> >>>>
> >>>> http://et.redhat.com/~rjones/auto-buildrequires/
> >>> There's one slightly similar one, which is several years old:
> >>>
> >>>
> https://www.europe.redhat.com/documentation/rhl6.2/rhl_6.2_sw_repository/...
> >>>
> >>> Download it here for example:
> >>> http://www.filewatcher.com/m/InDependence-1.0-6.src.rpm.16197.0.0.html
> >> this might  be useful on conjunction with the occasional "full rawhide
> >> rebuilds using rpmbuild on a system with Everything installed" run
> >> that someone (sorry, I forget who did this) does.  That has the added
> >> benefit of picking up features that configure auto-discovers but if
> >> missing BRs, won't discover.
> >
> > It was Karsten Hopp back in March 2008.
> > https://www.redhat.com/archives/fedora-devel-list/2008-March/msg01257.html
> >
>
>  I've used auto-br-rpmbuild with my latest full-install mass rebuild and grepped the
>  BuildRequires from the logs.
>  They are available at
>  	http://karsten.fedorapeople.org/buildrequires/ 
At some point I'm going to push this project into a public place, like
Fedora Hosted.  There's a series of patches that people have sent me
which need to be integrated ...

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

On Thu, Dec 18, 2008 at 12:44:47PM +0000, Richard W.M. Jones wrote:
>  At some point I'm going to push this project into a public place,
> like
>  Fedora Hosted.  There's a series of patches that people have sent me
>  which need to be integrated ... 
https://fedorahosted.org/fedora-infrastructure/ticket/1070

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Jerry James]

On Thu, Dec 18, 2008 at 5:41 AM, Karsten Hopp <karsten(a)redhat.com&gt; wrote:
>  I've used auto-br-rpmbuild with my latest full-install mass
> rebuild and
>  grepped the
>  BuildRequires from the logs.
>  They are available at
>         http://karsten.fedorapeople.org/buildrequires/ 
I'm not sure what to do with this information.  For example, take a
look at one of my packages: automaton.  It's a Java package.  Yet your
logs show gstreamer-devel as a BuildRequires.  How did that get in
there?  And what's going on with the 3 different versions of glibc, on
2 arches?

I've checked a couple of my other packages and had a similar reaction.
 There are BuildRequires listed that make no sense to me.  Can you
give more detail on how these lists were generated?  Thanks,
-- 
Jerry James
http://loganjerry.googlepages.com/

[Michael Schwendt]

On Fri, 19 Dec 2008 15:42:35 -0700, Jerry wrote:

>  On Thu, Dec 18, 2008 at 5:41 AM, Karsten Hopp wrote:
>  > I've used auto-br-rpmbuild with my latest full-install mass rebuild and
>  > grepped the
>  > BuildRequires from the logs.
>  > They are available at
>  >        http://karsten.fedorapeople.org/buildrequires/
>  
>  I'm not sure what to do with this information.  For example, take a
>  look at one of my packages: automaton.  It's a Java package.  Yet your
>  logs show gstreamer-devel as a BuildRequires.  How did that get in
>  there? 
A bug. compface also lists gstreamer-devel as a BR, which isn't true.

There are more problems. For example:

xmms-sid (an input plugin for XMMS) lists lots of packages, which
are completely unrelated to the plugin and xmms. Not limited to
wine-core, xulrunner, qt3, libvirt-cim, inn, kdelibs*, mysql-libs,
octave, ogre, ...

Same for sylpheed.
Same for pth.
Same for audacious-plugin-fc.

[Richard W.M. Jones]

On Fri, Dec 19, 2008 at 03:42:35PM -0700, Jerry James wrote:
>  On Thu, Dec 18, 2008 at 5:41 AM, Karsten Hopp
> <karsten(a)redhat.com&gt; wrote:
>  > I've used auto-br-rpmbuild with my latest full-install mass rebuild and
>  > grepped the
>  > BuildRequires from the logs.
>  > They are available at
>  >        http://karsten.fedorapeople.org/buildrequires/
>  
>  I'm not sure what to do with this information.  For example, take a
>  look at one of my packages: automaton.  It's a Java package.  Yet your
>  logs show gstreamer-devel as a BuildRequires.  How did that get in
>  there?  And what's going on with the 3 different versions of glibc, on
>  2 arches?
>  
>  I've checked a couple of my other packages and had a similar reaction.
>   There are BuildRequires listed that make no sense to me.  Can you
>  give more detail on how these lists were generated?  Thanks, 
Possibly a bug, possibly because something in the configure script of
the application touches a file in gstreamer-devel.  Take a look at:

http://et.redhat.com/~rjones/auto-buildrequires/

There'll be a fedorahosted project of this soon.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top

[Jerry James]

On Sat, Dec 20, 2008 at 1:25 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  Possibly a bug, possibly because something in the configure script
> of
>  the application touches a file in gstreamer-devel.  Take a look at:
>
>  http://et.redhat.com/~rjones/auto-buildrequires/
>
>  There'll be a fedorahosted project of this soon.
>
>  Rich.
>
>  --
>  Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
>  virt-top is 'top' for virtual machines.  Tiny program with many
>  powerful monitoring features, net stats, disk stats, logging, etc.
>  http://et.redhat.com/~rjones/virt-top 
I would like to debug this.  That would be much easier if there was
some way to tell the script to not "rm $logfile" when it is done.
I'll cheat and edit the script for now, but please add an option to do
that.

Okay, cheating done.  The gstreamer-devel dependency is being pulled
in because /usr/lib/rpm/gstreamer.prov is being opened.  Since
rpmbuild itself does that, it appears that the script will tell you
that gstreamer-devel is a BuildRequires for every package, right?

This probably explains why the script is also telling me that both
perl and python are BuildRequires for my Java package.
-- 
Jerry James
http://loganjerry.googlepages.com/

[Richard W.M. Jones]

On Sun, Dec 21, 2008 at 04:39:02PM -0700, Jerry James wrote:
>  I would like to debug this.  That would be much easier if there was
>  some way to tell the script to not "rm $logfile" when it is done.
>  I'll cheat and edit the script for now, but please add an option to do
>  that. 
Yup, that would be a useful option.

>  Okay, cheating done.  The gstreamer-devel dependency is being pulled
>  in because /usr/lib/rpm/gstreamer.prov is being opened.  Since
>  rpmbuild itself does that, it appears that the script will tell you
>  that gstreamer-devel is a BuildRequires for every package, right?
>  
>  This probably explains why the script is also telling me that both
>  perl and python are BuildRequires for my Java package. 
.. and Fortran too.  Most configure scripts look for it, but it's not
really used that often.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v