Thanks for the detailed response!
To start with your final questions:
> - Why is Python itself not installed in the /app prefix?
> - Where and how is pip install invoked?
The way that Flatpaks work is that there are *two* filesystem images, one mounted on /usr and one mounted on /app. The image mounted on /usr is called the "runtime" and contains libraries and other files shared between different applications. In Fedora, this is made out of standard distribution packages - glibc, libjpeg, and so forth. It also contains Python 3 and some modules.
The filesystem image at /app contains packages specific to a specific application - the application itself and libraries. In Fedora, we create the /app filesystem by rebuilding packages inside a module that is configured to have a special RPM macros package installed in the buildroot -
https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 (installed at /etc/rpm/macros.flatpak so that it has higher priority than the macros from /usr/lib/rpm)
(The advantage of this split over the docker layer system is that applications
don't need to be rebuilt every time the base layer is updated.)
macros.flatpak redefines %_prefix to /app, and then does various more and less hacky things to make that actual work within the Fedora RPM ecosystem - in particular for Python, it overrides
%python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is installed from the same package. This combination worked pretty well for F35 for Python.
pip install is being invoked is by %pyproject_install or %py_install_wheel out of an RPM build - the particular breakage that triggered this investigation was the Python bindings for zxing-cpp , but I think it would apply to pretty much anything that has moved to the %pyproject macros.
On Thu, May 26, 2022 at 4:04 AM Miro Hrončok <
mhroncok@redhat.com> wrote:
Having some standard way to configure a system by environment variable or drop in file would definitely be nice ... and sounds like something we could leverage.
Though I'm also wondering now if we could just change %py_install variants and %pyproject variants to have --prefix %{_prefix} on the install command line - that should have no effect for standard builds and make Flatpak builds work?
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to False.
RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my change
that reads this value to make it work (I have no problem doing that).
Those are the ideas that come to my mind about how to set that attribute:
- build your own forked Python package that sets this
- build your own forked pip package that defaults to this
We definitely have the *ability* to use a stream branch for python3 or python-pip, and build it in the flatpak-runtime package. But I'd really prefer to avoid that, since that creates a fork we need to maintain every time that these packages are updated in Fedora 36, and there's a possibility of missing security updates.
- set it via a Python module that is imported from
/usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth
However, forcing pip to use distutils is the opposite of future-proof.
When looking for a permanent solution, you need to avoid distutils entirely.
A pip with a backported patch (or a rebase) + this approach sounds most maintainable for Fedora 36. But if we think that the --prefix %{_prefix} approach actually works, then perhaps we can either:
A) Just update python-rpm-macros and pygobject-rpm-macros in F36
B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime module for F36 and reconverge for F37 - I'm less worried about carrying a stream branch for macros packages, since they are unlikely to have security fixes.