Automated Python packaging?

[Peter Oliver]

Thereʼs a Python module Iʼd like to package for Fedora.  I made a start on Copr, but I rapidly got lost in a maze of missing dependencies and broken builds.

OTOH, running `pip install` takes seconds and everything just works.

I feel like someone must have come up with an easy way of building a chain of PyPI dependencies into a collection of RPMs.  Am I missing a trick?

[mabinm]

Hi Peter.You can build the kind of dependency-aware rpm you're looking for in two ways.Do a pip download of the python package with or without dependencies and have the %post scriptlet in the RPM spec file do a pip install of the wheel, tar.gz or egg into the target python installation, which will pull other dependent packages required by the python package file.Maybe no file at all needs to be included in the rpm itself, just the rpm post scriptlet doing the pip install.The fpm utility can be used to convert a pip package to an rpm package. You still may need to provide a postinstall script argument that does the pip install of dependencies. Michael MabinSent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: Peter Oliver  Date: 11/13/20  4:50 PM  (GMT-06:00) To: [e-mail] Subject: Automated Python packaging? Thereʼs a Python module Iʼd like to package for Fedora.  I made a start on Copr, but I rapidly got lost in a maze of missing dependencies and broken builds.OTOH, running `pip install` takes seconds and everything just works.I feel like someone must have come up with an easy way of building a chain of PyPI dependencies into a collection of RPMs.  Am I missing a trick?-- Peter Oliver_______________________________________________python-devel mailing list -- [e-mail] unsubscribe send an email to [e-mail] Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelinesList Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Lumír Balhar]

Hello, Michael.

On 11/14/20 1:50 AM, mabinm wrote:
> Hi Peter.
>
> You can build the kind of dependency-aware rpm you're looking for in 
> two ways.
>
> Do a pip download of the python package with or without dependencies 
> and have the %post scriptlet in the RPM spec file do a pip install of 
> the wheel, tar.gz or egg into the target python installation, which 
> will pull other dependent packages required by the python package file.

This is considered a very bad idea given the fact that it can completely 
destroy your system. This is similar to doing `sudo pip install 
<package>` which might rewrite files installed from RPMs to the global 
system site-packages — files that not only dnf but also other system 
libraries might depend on.

If something is packaged as RPM, package maintainers make sure that the 
package is compatible with the rest of the system. Installing something 
from the PyPI to the global site-packages is a bad idea.

Have a nice day.

Lumír
> Maybe no file at all needs to be included in the rpm itself, just the 
> rpm post scriptlet doing the pip install.
>
> The fpm utility can be used to convert a pip package to an rpm 
> package. You still may need to provide a postinstall script argument 
> that does the pip install of dependencies.
>
> Michael Mabin
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
> -------- Original message --------
> From: Peter Oliver 
> Date: 11/13/20 4:50 PM (GMT-06:00)
> To: [e-mail]
> Subject: Automated Python packaging?
>
> Thereʼs a Python module Iʼd like to package for Fedora.  I made a 
> start on Copr, but I rapidly got lost in a maze of missing 
> dependencies and broken builds.
>
> OTOH, running `pip install` takes seconds and everything just works.
>
> I feel like someone must have come up with an easy way of building a 
> chain of PyPI dependencies into a collection of RPMs.  Am I missing a 
> trick?
>
> -- 
> Peter Oliver
> _______________________________________________
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Michael Mabin]

> On Nov 14, 2020, at 9:31 AM, Lumír Balhar  wrote:
>
> Hello, Michael.
>
> On 11/14/20 1:50 AM, mabinm wrote:
> > Hi Peter.
> >
> > You can build the kind of dependency-aware rpm you're looking for in two ways.
> >
> > Do a pip download of the python package with or without dependencies and have the %post scriptlet in the RPM spec file do a pip install of the wheel, tar.gz or egg into the target python installation, which will pull other dependent packages required by the python package file.
>
> This is considered a very bad idea given the fact that it can completely destroy your system. This is similar to doing `sudo pip install <package>` which might rewrite files installed from RPMs to the global system site-packages — files that not only dnf but also other system libraries might depend on.
>
> If something is packaged as RPM, package maintainers make sure that the package is compatible with the rest of the system. Installing something from the PyPI to the global site-packages is a bad idea.
>
> Have a nice day.
>
> Lumír

Yes. I get this would not be desirable for the system python.
But for an alternative python location it would be fine.

Michael Mabin
> > Maybe no file at all needs to be included in the rpm itself, just the rpm post scriptlet doing the pip install.
> >
> > The fpm utility can be used to convert a pip package to an rpm package. You still may need to provide a postinstall script argument that does the pip install of dependencies.
> >
> > Michael Mabin
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> >
> > -------- Original message --------
> > From: Peter Oliver  
> > Date: 11/13/20 4:50 PM (GMT-06:00)
> > To: [e-mail] 
> > Subject: Automated Python packaging?
> >
> > Thereʼs a Python module Iʼd like to package for Fedora.  I made a start on Copr, but I rapidly got lost in a maze of missing dependencies and broken builds.
> >
> > OTOH, running `pip install` takes seconds and everything just works.
> >
> > I feel like someone must have come up with an easy way of building a chain of PyPI dependencies into a collection of RPMs.  Am I missing a trick?
> >
> > -- 
> > Peter Oliver
> > _______________________________________________
> > python-devel mailing list -- [e-mail] 
> > To unsubscribe send an email to [e-mail] 
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]... https://lists.fedoraproject.org/archives/list/[e-mail]
> >
> > python-devel mailing list -- [e-mail] 
> > To unsubscribe send an email to [e-mail] 
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]... https://lists.fedoraproject.org/archives/list/[e-mail]

[Lumír Balhar]

Hello.

Unfortunately, there is no way to package a bunch of packages from PyPI 
to RPMs at once. There is pyp2rpm [0] tool that might help you with 
creating individual packages but if your module has a lot of 
dependencies and some of them are missing in Fedora, you'll have to 
start with them.

Which module are we talking about?

Lumír

[0] https://github.com/fedora-python/pyp2rpm

On 11/13/20 11:50 PM, Peter Oliver wrote:
> Thereʼs a Python module Iʼd like to package for Fedora.  I made a start on Copr, but I rapidly got lost in a maze of missing dependencies and broken builds.
>
> OTOH, running `pip install` takes seconds and everything just works.
>
> I feel like someone must have come up with an easy way of building a chain of PyPI dependencies into a collection of RPMs.  Am I missing a trick?

[Peter Oliver]

On Sat, 14 Nov 2020, Lumír Balhar wrote:
> Which module are we talking about?

Itʼs https://pypi.org/project/nanoemoji/

[Lumír Balhar]

Its dependency tree is long:

nanoemoji==0.4.0
   - absl-py [required: >=0.9.0, installed: 0.11.0]
     - six [required: Any, installed: 1.15.0]
   - fonttools [required: >=4.17.0, installed: 4.17.0]
   - lxml [required: >=4.0, installed: 4.6.1]
   - ninja [required: >=1.10.0.post1, installed: 1.10.0.post2]
   - picosvg [required: >=0.7.1, installed: 0.7.1]
     - lxml [required: >=4.0, installed: 4.6.1]
     - skia-pathops [required: >=0.4.1, installed: 0.5.1.post1]
   - pillow [required: >=7.2.0, installed: 8.0.1]
   - regex [required: >=2020.4.4, installed: 2020.11.13]
   - ufo2ft [required: >=2.15.0, installed: 2.16.0]
     - booleanOperations [required: >=0.9.0, installed: 0.9.0]
       - fonttools [required: >=4.0.2, installed: 4.17.0]
       - pyclipper [required: >=1.1.0.post1, installed: 1.2.0]
     - compreffor [required: >=0.4.6, installed: 0.5.0]
       - fonttools [required: >=4, installed: 4.17.0]
     - cu2qu [required: >=1.6.6, installed: 1.6.7]
       - fonttools [required: >=3.32.0, installed: 4.17.0]
     - fonttools [required: >=4.11.0, installed: 4.17.0]
   - ufoLib2 [required: >=0.6.2, installed: 0.8.0]
     - attrs [required: >=19.2.0, installed: 20.3.0]
     - fonttools [required: >=4.0.0, installed: 4.17.0]

Some packages are already in Fedora (six, pillow, fonttools, lxml …) but 
if you want to package nanoemoji, packaging its missing dependencies 
will be unavoidable.

Lumír

On 11/16/20 12:10 AM, Peter Oliver wrote:
> On Sat, 14 Nov 2020, Lumír Balhar wrote:
> > Which module are we talking about?
>
> Itʼs https://pypi.org/project/nanoemoji/
>
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...