Discussion: Macros for Python Specfiles

[Bohuslav Kabrda]

Hi all,
as mentioned in [1], I'd like to start a discussion about proposed Python macros.
The full macro file can be seen at [2] and an example diff of converted spec at [3].

The important macros from the proposal are:

%py_default_filter # will filter all *.so file provides from
/usr/lib{64}/pythonX.Y/site-packages
%py[3]_build # will run setup.py build with proper args/flags
%py[3]_install # will run setup.py install with proper args

I already proposed this to fpc [4], but it seems that there is a disagreement [5] about
the %py_{build,install} macros. This is why I'd like to start a broader discussion
about them here.
Thank you for your opinions.

Slavek.

-- 
Regards,
Bohuslav "Slavek" Kabrda.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=953704
[2] https://bugzilla.redhat.com/attachment.cgi?id=739290
[3] https://bugzilla.redhat.com/attachment.cgi?id=739291
[4] https://fedorahosted.org/fpc/ticket/281
[5] https://bugzilla.redhat.com/show_bug.cgi?id=953704#c10

[Nick Coghlan]

On 05/06/2013 10:43 PM, Bohuslav Kabrda wrote:
>  Hi all,
>  as mentioned in [1], I'd like to start a discussion about proposed Python macros.
>  The full macro file can be seen at [2] and an example diff of converted spec at [3].
>
>  The important macros from the proposal are:
>
>  %py_default_filter # will filter all *.so file provides from
> /usr/lib{64}/pythonX.Y/site-packages
>  %py[3]_build # will run setup.py build with proper args/flags
>  %py[3]_install # will run setup.py install with proper args
>
>  I already proposed this to fpc [4], but it seems that there is a disagreement [5] about
> the %py_{build,install} macros. This is why I'd like to start a broader discussion
> about them here.
>  Thank you for your opinions. 
The potential advantage I see to the py_build and py_install macros is 
that we're actively trying to move away from always relying on setup.py 
upstream. Thus, the idea of a helper script that calls setup.py if it is 
present, or looks up the right build command in pymeta.cfg if *that* is 
present sounds reasonable to me.

Similarly, we're going to try to move away from invoking setup.py at 
install time, and instead install from wheel files with a post-install 
hook registered in the package metadata.

This will hopefully become clearer once I finish the next draft of PEP 
426 (Python's distribution metadata 2.0). You can get a sneak peek at 
https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt, but be 
aware that that is a *very* rough draft (I'm in the middle of a major 
rewrite, after we realised the old metadata was fundamentally broken and 
not useful to anyone. I still have a *ton* of stuff to do in terms of 
clearly defining the bidirectional mapping between legacy metadata and 
metadata 2.0, adding the post install hook specification, etc).

The two parts likely to be of most interest to those on this list:

Development activities: 
https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt#cl-124
Dependencies: 
https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt#cl-577

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)

[Toshio Kuratomi]

On Tue, May 07, 2013 at 01:57:01PM +1000, Nick Coghlan wrote:
>  On 05/06/2013 10:43 PM, Bohuslav Kabrda wrote:
>  >Hi all,
>  >as mentioned in [1], I'd like to start a discussion about proposed Python
> macros.
>  >The full macro file can be seen at [2] and an example diff of converted spec at [3].
>  >
>  >The important macros from the proposal are:
>  >
>  >%py_default_filter # will filter all *.so file provides from
> /usr/lib{64}/pythonX.Y/site-packages 
This one has been approved

>  >%py[3]_build # will run setup.py build with proper args/flags
>  >%py[3]_install # will run setup.py install with proper args
>  > 
I have reservations about these.  The first one is general to these types of
macros so I'll make the case not to use these macros at all here.
Subsequent portions of this message will deal with problems I have with the
macros should we choose to go this route.

RPM spec files are mostly shell scripts.  This makes it easy for system
administrators to make the jump from locally installing a package to
building an rpm to becoming a valued Fedora packager because, at its heart,
an rpm spec file is just recording the steps that they would have taken
anyway.

Macros form another layer on top of the shell scripts that system
administrators have to learn; a layer that they can't then directly apply
back to their local installs.

As the Zen of python says, "Explicit is better than implicit." In this case
the macros are moving us away from being explicit for only a little gain.
(I'll acknowledge and talk about the gain in the next section).  Moreover,
for every command line that we're replacing, we are replacing them with
multiple macros, three of which are publically visible and all of which
would need to be understood by someone who wanted to know what the macros do
and modify their behaviour:

The current command line for building:
   %{__python} setup.py build

The macros to implement this for the new style:
  %{__python} setup.py build
  %py_build %py2_build
  %py2_build %py_rt_build %__python
  %py_rt_build() %{expand:\
    CFLAGS="$RPM_OPT_FLAGS" %{!?1:%{__python}}%{?1:%{1}} %{py_setup}
    %{?py_setup_args} build %{?py_build_args}\
    }


>  >I already proposed this to fpc [4], but it seems that there is a
> disagreement [5] about the %py_{build,install} macros. This is why I'd like to start a
> broader discussion about them here.
>  >Thank you for your opinions.
>  
>  The potential advantage I see to the py_build and py_install macros
>  is that we're actively trying to move away from always relying on
>  setup.py upstream. 
<nod> This is the advantage that I see to macroizing as well.  While writing
out the command line makes it easy for individual packagers to experiment
with changes to the common workflow and then enhance the guidelines by
submitting the results of their experiments as modifications to the default
command line invocations in the guidelines, macroizing makes it easy for a
central decision maker to update how all packages do things.

>  Thus, the idea of a helper script that calls
>  setup.py if it is present, or looks up the right build command in
>  pymeta.cfg if *that* is present sounds reasonable to me.
>   
If we're going to hide the command line invocation behind a macro, I did
propose in the bug report that we use a helper script rather than pure
macros to implement it:
  https://bugzilla.redhat.com/show_bug.cgi?id=953704#c10

Pure rpm macros are hard to read and understand and differ enough from shell
that someone attempting to carry their knowledge from that domain will get
lost.  A script would alleviate that.

Could you tell us more about pymeta.cfg?  is it the pymeta.json that I see
mentioned in the draft PEP?  Another data format of the same information?
Or something different?

>  Similarly, we're going to try to move away from invoking setup.py
> at
>  install time, and instead install from wheel files with a
>  post-install hook registered in the package metadata.
>   
So.. in talking about using wheel, I had a look at what we can do now.  This
is what the landscape looked like to me a month ago when I tried it out:

# This stays the same as it is now
Source0: sdist-1.0.tar.gz

# New BuildRequires.  python-wheel isn't yet packaged.
# python-pip needs to have the equivalent of the --root flag; this has
# recently been updated in Fedora to a version that has this capability
BuildRequires: python-wheel
BuildRequires: python-pip >= 1.3.0

# This also stays the same as it is now
%prep
%setup -q -n %{srcname}-%{version}
%patch0 [..]

%build
# here's the first difference.  We build a wheel instead of just the
# compiled code
%{__python} setup.py bdist_wheel

%install
# And then we install the wheel into an expanded form on the filesystem
pip install --root %{buildroot} dist/*.whl

I actually got stuck on the install section for a while (python-wheel
doesn't have a --root capable install command and python-pip was too old in
Fedora at the time) so I haven't looked further than that to see if there
were different files on disk as a result of installing from a wheel... my
understanding is that that is one of the promises of the wheel format (for
instance, locale files ending up in %{_datadir}/locale instead of in
%{python_sitelib} ).

What I see so far:
* The build and install commands look macro-able
* This uses setup.py but from your other comment, I guess that there's
  a different way to do this that uses an alternate tool?
* This seemed to work without any support from setup.py but I was using a
  very simple package as my first try at this.  We probably need to try this
  with more complex package -- in particular I wonder whether we will be
  able to use this for all packages, most packages, or just some packages.
  That affects whether we want to make this something that is chosen
  centrally vs chosen by the individual package maintainers.
* If the package builds using setup.py and builds either using the old
  strategy or the new strategy, what are the benefits of the new strategy?
* As mentioned earlier, I heard that one of the benefits of the new strategy
  was that it would place certain types of files into FHS compliant locations.
  This is great but it seems like it won't always be easy or desirable to
  macroize that in the %files section.  If the package maintainer has to
  adjust the %files section explicitly, it might be better for them to be in
  control of switching the build strategy as well.


>  This will hopefully become clearer once I finish the next draft of
>  PEP 426 (Python's distribution metadata 2.0).  
<nod>  My suggestion is that if we do want to macroize this to make it easy
to centrally switch the strategy used to build and install python modules we
are putting hte cart before the horse here.  We should first figure out what
the new strategy is and write that up.  Then figure out how to write macros
that allow us to switch between the two.  Writing the macros first and then
hoping that they cover all of the things that need to be changed in order to
effect the change is either going to lead to an incomplete solution (which
will require packagers to modify their spec files multiple times for the
changing guidelines) or overengineering.as we make it account for potential
quirks in the tooling that never appear in the future.

>  
>  The two parts likely to be of most interest to those on this list:
>  
>  Development activities:
>  https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt#cl-124
>  Dependencies:
>  https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt#cl-577
>   
(This section is getting into a tangent from the original topic).

Read through those sections -- one thing that I noticed is that how to use
the version information in a dependency is not spoken about.  Something that
would be nice from a distribution POV is to treat versions differently if
you are running a program (including testing) versus deploying it.  This is
something that pkg_resources gets wrong.

Example:

python-foo-1.0 is upstream and python-foo-1.0-1 is the Fedora package.

python-foo is a large framework with several different python modules that
don't all need to be used together.

A security issue is found in the package.  upstream releases python-foo-1.2
with the fix as well as some API changes to specific modules.  Fedora
applies a backport of the security fix to their python-foo-1.0-2 package.

The pybar application makes use of the python-foo modules.  They only use
the modules whose API has not changed between python-foo-1.0 and
python-foo-2.0.  Due to the security update, they update their Requirement
to python-foo >= 2.0.

now -- if I'm a developer or a sysadmin installing from pypi; I want pip
install to (which processes the dependencies recursively) to get
python-foo-1.2.

If I'm a packager for Fedora, I want pybar to use the python-foo-1.0-2
Fedora package when I build and run because I've checked that the API usage
is okay.

If I'm the end user, I want the Fedora pybar package that I've installed to
work with the Fedora python-foo-1.0-2 package.

With pkg_resources, this required patching of the version information in
the setup.py by the packager to account for this other use (bug was fixed
downstream).

A higher level way to think about it is to be able to separate out version
information as it relates to API compatibility from version information as
it relates to working around bugs.

-Toshio

[Nick Coghlan]

On 05/25/2013 03:13 AM, Toshio Kuratomi wrote:
>  As the Zen of python says, "Explicit is better than
> implicit." In this case
>  the macros are moving us away from being explicit for only a little gain.
>  (I'll acknowledge and talk about the gain in the next section).  Moreover,
>  for every command line that we're replacing, we are replacing them with
>  multiple macros, three of which are publically visible and all of which
>  would need to be understood by someone who wanted to know what the macros do
>  and modify their behaviour:
>
>  The current command line for building:
>      %{__python} setup.py build
>
>  The macros to implement this for the new style:
>     %{__python} setup.py build
>     %py_build %py2_build
>     %py2_build %py_rt_build %__python
>     %py_rt_build() %{expand:\
>       CFLAGS="$RPM_OPT_FLAGS" %{!?1:%{__python}}%{?1:%{1}} %{py_setup}
>       %{?py_setup_args} build %{?py_build_args}\
>       } 
Given the upstream flux, I'd say "don't change this yet". We're
punting 
on several of the issues that are specific to building from source to 
metadata 2.1 (since they involve bigger changes than just cleaning up 
the installation side of things).

>  Could you tell us more about pymeta.cfg?  is it the pymeta.json that
> I see
>  mentioned in the draft PEP?  Another data format of the same information?
>  Or something different? 
Because pymeta.json is a generated file, the question becomes "How do 
you generate pymeta.json, if all you have is a source tarball?".

For metadata 2.0, the command for that is probably going to be "python 
setup.py dist_info".

Killing the reliance on setup.py on the build side now won't be 
happening until metadata 2.1 at the earliest.

> > Similarly, we're going to try to move away from invoking
> setup.py at
> > install time, and instead install from wheel files with a
> > post-install hook registered in the package metadata.
> >
>
>  So.. in talking about using wheel, I had a look at what we can do now.  This
>  is what the landscape looked like to me a month ago when I tried it out:
>
>  # This stays the same as it is now
>  Source0: sdist-1.0.tar.gz
>
>  # New BuildRequires.  python-wheel isn't yet packaged.
>  # python-pip needs to have the equivalent of the --root flag; this has
>  # recently been updated in Fedora to a version that has this capability
>  BuildRequires: python-wheel
>  BuildRequires: python-pip >= 1.3.0 
pip 1.4 should include the necessary bits to handle the wheel generation 
(http://www.pip-installer.org/en/latest/usage.html#pip-wheel).

>  # This also stays the same as it is now
>  %prep
>  %setup -q -n %{srcname}-%{version}
>  %patch0 [..]
>
>  %build
>  # here's the first difference.  We build a wheel instead of just the
>  # compiled code
>  %{__python} setup.py bdist_wheel
>
>  %install
>  # And then we install the wheel into an expanded form on the filesystem
>  pip install --root %{buildroot} dist/*.whl
>
>  I actually got stuck on the install section for a while (python-wheel
>  doesn't have a --root capable install command and python-pip was too old in
>  Fedora at the time) so I haven't looked further than that to see if there
>  were different files on disk as a result of installing from a wheel... my
>  understanding is that that is one of the promises of the wheel format (for
>  instance, locale files ending up in %{_datadir}/locale instead of in
>  %{python_sitelib} ). 
Yeah, support for sysconfig schemes is the main advantage wheel offers 
over the previous egg format (although it does in turn depend on 
upstream projects identify files correctly).

>  What I see so far:
>  * The build and install commands look macro-able
>  * This uses setup.py but from your other comment, I guess that there's
>     a different way to do this that uses an alternate tool? 
Eventually we want to have the build commands be entry points that 
reference Python callables, but I've decided it makes more sense to 
postpone that aspect and focus on the installation side for now.

>  * This seemed to work without any support from setup.py but I was
> using a
>     very simple package as my first try at this.  We probably need to try this
>     with more complex package -- in particular I wonder whether we will be
>     able to use this for all packages, most packages, or just some packages.
>     That affects whether we want to make this something that is chosen
>     centrally vs chosen by the individual package maintainers. 
I think it should be package-by-package for now. Some packages (like 
Twisted) are known not to support installation from the current 
incarnation of the wheel format because they need to run code following 
the install. "./setup.py install" currently lets them do that, but 
wheels won't support postinstallation hooks until metadata 2.0 is available.

>  * If the package builds using setup.py and builds either using the
> old
>     strategy or the new strategy, what are the benefits of the new strategy? 
For us, probably not a lot because we use separate build servers anyway, 
with binary RPMs for deployment systems. The payoff is bigger upstream 
where wheels act as a cross-platform binary distribution format.

>  * As mentioned earlier, I heard that one of the benefits of the new
> strategy
>     was that it would place certain types of files into FHS compliant locations.
>     This is great but it seems like it won't always be easy or desirable to
>     macroize that in the %files section.  If the package maintainer has to
>     adjust the %files section explicitly, it might be better for them to be in
>     control of switching the build strategy as well.
>
>
> > This will hopefully become clearer once I finish the next draft of
> > PEP 426 (Python's distribution metadata 2.0).
>
>  <nod>  My suggestion is that if we do want to macroize this to make it easy
>  to centrally switch the strategy used to build and install python modules we
>  are putting hte cart before the horse here.  We should first figure out what
>  the new strategy is and write that up.  Then figure out how to write macros
>  that allow us to switch between the two.  Writing the macros first and then
>  hoping that they cover all of the things that need to be changed in order to
>  effect the change is either going to lead to an incomplete solution (which
>  will require packagers to modify their spec files multiple times for the
>  changing guidelines) or overengineering.as we make it account for potential
>  quirks in the tooling that never appear in the future. 
Yeah, I agree - wait and see is the better path here.

> > The two parts likely to be of most interest to those on this
> list:
> >
> > Development activities:
> > https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt#cl-124
> > Dependencies:
> > https://bitbucket.org/ncoghlan/misc/src/default/pep-0426.txt#cl-577
> >
>
>  (This section is getting into a tangent from the original topic).
>
>  Read through those sections -- one thing that I noticed is that how to use
>  the version information in a dependency is not spoken about.  Something that
>  would be nice from a distribution POV is to treat versions differently if
>  you are running a program (including testing) versus deploying it.  This is
>  something that pkg_resources gets wrong. 
I'll be publishing updated drafts of PEP 426 (metadata 2.0) and PEP 440 
(the associated versioning spec) this week. Something they try to do is 
make clear that the kinds of dependencies appropriate for library 
developers (which should have generous assumptions regarding forward 
compatibility) differ from those that are suitable for system 
integrators (who may pin particular tested versions)

>  Example:
>
>  python-foo-1.0 is upstream and python-foo-1.0-1 is the Fedora package.
>
>  python-foo is a large framework with several different python modules that
>  don't all need to be used together.
>
>  A security issue is found in the package.  upstream releases python-foo-1.2
>  with the fix as well as some API changes to specific modules.  Fedora
>  applies a backport of the security fix to their python-foo-1.0-2 package.
>
>  The pybar application makes use of the python-foo modules.  They only use
>  the modules whose API has not changed between python-foo-1.0 and
>  python-foo-2.0.  Due to the security update, they update their Requirement
>  to python-foo >= 2.0. 
I'm guessing you meant "python-foo >= 1.2" here?

>  now -- if I'm a developer or a sysadmin installing from pypi; I
> want pip
>  install to (which processes the dependencies recursively) to get
>  python-foo-1.2.
>
>  If I'm a packager for Fedora, I want pybar to use the python-foo-1.0-2
>  Fedora package when I build and run because I've checked that the API usage
>  is okay.
>
>  If I'm the end user, I want the Fedora pybar package that I've installed to
>  work with the Fedora python-foo-1.0-2 package.
>
>  With pkg_resources, this required patching of the version information in
>  the setup.py by the packager to account for this other use (bug was fixed
>  downstream).
>
>  A higher level way to think about it is to be able to separate out version
>  information as it relates to API compatibility from version information as
>  it relates to working around bugs. 
If the upstream project doesn't do maintenance releases, then there's 
still going to be some kind of modification of the metadata or the build 
commands required for cases where an application needs a bug fix that 
has been backported from the next feature release.

That said, I believe pip supports version overrides which let an 
integrator say "trust me, this version is fine, the upstream dependency 
metadata is wrong".

So, perhaps the moral of the story here is that going through pip 
instead of invoking setup.py directly lets you override quite a few bad 
default behaviours in setuptools, which means *pip* might be an 
appropriate choice as our Python build helper tool :)

That would also let us transparently upgrade to any new metabuild hooks 
we come up with upstream - such hooks will only happen if the pip 
developers agree to implement them upstream, so we may as well take 
advantage of that downstream.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)