Problems with scripts in a common spec file

[John Dennis]

With reference to the guidelines for Python packaging found here:

https://fedoraproject.org/wiki/Packaging:Python

Specifically the material concerning executables in /usr/bin.

In the "Example common spec file" section is this comment.

# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install, and in general we want the
# python3 version to be the default.

But this does not work if you only install the python2-XXX package 
instead of both the python2-XXX and python3-XXX packages. Here is why.

distutils will adjust the shbang interpreter line in installed scripts 
to be interpreter specific, e.g. you get one of:

#!/usr/bin/python2

or

#!/usr/bin/python3

But the guidelines require the py3 version of the script in the py2 
package. Thus if you install just the python2-XXX package you'll end up 
with a script that runs the /usr/bin/python3 interpreter whose sys.path 
only includes library locations for py3! Thus the script cannot execute 
because it's missing its libraries (because the libraries were installed 
in the py2 locations).

How is this supposed to be addressed?

One possible workaround is to add a requirement on the py3 package in 
the py2 package. But that is silly because it completely negates 
everything in the py2 package (everything will run as if it were py3 
after installing the py2 package and the py2 files will never be 
referenced).

Perhaps the issue is better rephrased as "How does one package Python 
when there are no version specific differences/requirements?" If this is 
covered in the guidelines it's not obvious and needs clarification. I 
assume this means using the generic %{_python} macros instead of dual 
use of the version specific macros so that everything corresponds to the 
system default version of Python. Correct?



-- 
John

[Tomas Orsava]

Hi,

those are very good questions to which you should be able to find 
answers on the Python RPM Porting Guide [0]. You are right that this 
should be better covered in the packaging guidelines, sadly the process 
of changing them is rather problematic so no one yet had the time to 
update them.

[0] http://python-rpm-porting.readthedocs.io/en/latest/

Kind regards,
Tomas



On 05/26/2016 12:18 AM, John Dennis wrote:
>  With reference to the guidelines for Python packaging found here:
>
>  https://fedoraproject.org/wiki/Packaging:Python
>
>  Specifically the material concerning executables in /usr/bin.
>
>  In the "Example common spec file" section is this comment.
>
>  # Must do the python2 install first because the scripts in /usr/bin are
>  # overwritten with every setup.py install, and in general we want the
>  # python3 version to be the default.
>
>  But this does not work if you only install the python2-XXX package 
>  instead of both the python2-XXX and python3-XXX packages. Here is why.
>
>  distutils will adjust the shbang interpreter line in installed scripts 
>  to be interpreter specific, e.g. you get one of:
>
>  #!/usr/bin/python2
>
>  or
>
>  #!/usr/bin/python3
>
>  But the guidelines require the py3 version of the script in the py2 
>  package. Thus if you install just the python2-XXX package you'll end 
>  up with a script that runs the /usr/bin/python3 interpreter whose 
>  sys.path only includes library locations for py3! Thus the script 
>  cannot execute because it's missing its libraries (because the 
>  libraries were installed in the py2 locations).
>
>  How is this supposed to be addressed?
>
>  One possible workaround is to add a requirement on the py3 package in 
>  the py2 package. But that is silly because it completely negates 
>  everything in the py2 package (everything will run as if it were py3 
>  after installing the py2 package and the py2 files will never be 
>  referenced).
>
>  Perhaps the issue is better rephrased as "How does one package Python 
>  when there are no version specific differences/requirements?" If this 
>  is covered in the guidelines it's not obvious and needs clarification. 
>  I assume this means using the generic %{_python} macros instead of 
>  dual use of the version specific macros so that everything corresponds 
>  to the system default version of Python. Correct?
>
>
>

[John Dennis]

On 05/26/2016 08:24 AM, Tomas Orsava wrote:
>  Hi,
>
>  those are very good questions to which you should be able to find
>  answers on the Python RPM Porting Guide [0]. You are right that this
>  should be better covered in the packaging guidelines, sadly the process
>  of changing them is rather problematic so no one yet had the time to
>  update them.
>
>  [0] http://python-rpm-porting.readthedocs.io/en/latest/ 
Thanks Tomas, I was aware of this document as well. However I believe 
both documents contain the same mistake.

Here is the problem:

Only the python3-XXX package installs the (Py3) script. Unless the 
python2-XXX package requires the script neither the script nor the Py3 
modules/packages required by the (Py3) script will be installed when you 
install either the python-XXX or python2-XXX package (assuming the 
virtual provides of python-XXX points to python2-XXX as is currently the 
case).

I think the python2-XXX package in the examples is missing something 
like this:

Requires: %{_bindir}/sample-exec

Make sense?

-- 
John

[Tomas Orsava]

Hi!

On 05/26/2016 07:38 PM, John Dennis wrote:
>  On 05/26/2016 08:24 AM, Tomas Orsava wrote:
> > Hi,
> >
> > those are very good questions to which you should be able to find
> > answers on the Python RPM Porting Guide [0]. You are right that this
> > should be better covered in the packaging guidelines, sadly the process
> > of changing them is rather problematic so no one yet had the time to
> > update them.
> >
> > [0] http://python-rpm-porting.readthedocs.io/en/latest/
>
>  Thanks Tomas, I was aware of this document as well. However I believe 
>  both documents contain the same mistake.
>
>  Here is the problem:
>
>  Only the python3-XXX package installs the (Py3) script. Unless the 
>  python2-XXX package requires the script neither the script nor the Py3 
>  modules/packages required by the (Py3) script will be installed when 
>  you install either the python-XXX or python2-XXX package (assuming the 
>  virtual provides of python-XXX points to python2-XXX as is currently 
>  the case).
>
>  I think the python2-XXX package in the examples is missing something 
>  like this:
>
>  Requires: %{_bindir}/sample-exec
>
>  Make sense?
>  
I believe there is a misunderstanding. In your first message you said 
"But the guidelines require the py3 version of the script in the py2 
package." That is incorrect.

The guidelines only require you to package the Python 3 version of the 
script. They are not very clear on where you should put it, but the only 
logical place is of course the Python 3 subpackage (or a new subpackage 
with only the executable script that relies on the Python 3 subpackage).

You should never put the Python 3 executable in the Python 2 subpackage.

Kind regards,
Tomas

[John Dennis]

On 05/27/2016 10:10 AM, Tomas Orsava wrote:
> > I think the python2-XXX package in the examples is missing
> something
> > like this:
> >
> > Requires: %{_bindir}/sample-exec
> >
> > Make sense?
> >
>  I believe there is a misunderstanding. In your first message you said
>  "But the guidelines require the py3 version of the script in the py2
>  package." That is incorrect.
>
>  The guidelines only require you to package the Python 3 version of the
>  script. They are not very clear on where you should put it, but the only
>  logical place is of course the Python 3 subpackage (or a new subpackage
>  with only the executable script that relies on the Python 3 subpackage).
>
>  You should never put the Python 3 executable in the Python 2 subpackage. 
That's fine. But unless one of two things are done the guidelines as 
they currently stand will leave you with a broken python2-XXX package. 
You either have to copy the Py2 version of the script to 
%{_bindir}/sample-exec-2.7 after the %py2_install runs instead of 
deleting the scripts as is currently recommended. -OR- you need to add a 
Requires on the %{_bindir}/sample-exec to the python2-XXX package as I 
suggested above.

If one of these 2 things are not done then someone installing just the 
python2-XXX version of the package (what I expect might be a common user 
action) will not have a script that can execute.

Also neither set of guidelines include examples of setting up links if 
you elect to use versioned scripts (e.g. %{_bindir}/sample-exec-2.7, 
%{_bindir}/sample-exec-3.5). The guidelines seem to suggest it's best to 
avoid versioned scripts which I would agree with, if that is the case 
then of the two options discussed above only the Requires fix will work.

-- 
John

[Tomas Orsava]

Hi!


On 05/27/2016 07:44 PM, John Dennis wrote:
>  On 05/27/2016 10:10 AM, Tomas Orsava wrote:
> >> I think the python2-XXX package in the examples is missing something
> >> like this:
> >>
> >> Requires: %{_bindir}/sample-exec
> >>
> >> Make sense?
> >>
> > I believe there is a misunderstanding. In your first message you said
> > "But the guidelines require the py3 version of the script in the py2
> > package." That is incorrect.
> >
> > The guidelines only require you to package the Python 3 version of the
> > script. They are not very clear on where you should put it, but the only
> > logical place is of course the Python 3 subpackage (or a new subpackage
> > with only the executable script that relies on the Python 3 subpackage).
> >
> > You should never put the Python 3 executable in the Python 2 subpackage.
>
>  That's fine. But unless one of two things are done the guidelines as 
>  they currently stand will leave you with a broken python2-XXX package. 
>  You either have to copy the Py2 version of the script to 
>  %{_bindir}/sample-exec-2.7 after the %py2_install runs instead of 
>  deleting the scripts as is currently recommended. -OR- you need to add 
>  a Requires on the %{_bindir}/sample-exec to the python2-XXX package as 
>  I suggested above.
>
>  If one of these 2 things are not done then someone installing just the 
>  python2-XXX version of the package (what I expect might be a common 
>  user action) will not have a script that can execute. 
That is as it should be. The
application should be provided in only one 
subpackage, not both, unless it behaves differently on Python 2 and 
Python 3 (example would be e.g. pip that does behave differently). If 
the user wants the application, they will have to install the Python 3 
subpackage.
>  Also neither set of guidelines include examples of setting up links if 
>  you elect to use versioned scripts (e.g. %{_bindir}/sample-exec-2.7, 
>  %{_bindir}/sample-exec-3.5). The guidelines seem to suggest it's best 
>  to avoid versioned scripts which I would agree with, if that is the 
>  case then of the two options discussed above only the Requires fix 
>  will work. 
The guidelines are indeed flawed. But the Python RPM Packaging Guide 
(not part of the guidelines) does include examples of this [0]. However, 
it's only explained in the 4th chapter—chapter for packages where the 
executable behaves differently depending on the Python version (2 or 3). 
In all other cases you should not use versioned executables as the 
executable shall be included in only one version.

[0] http://python-rpm-porting.readthedocs.io/en/latest/tools.html