PEP 720 – Cross-compiling Python packages

PEP 720 – Cross-compiling Python packages

Author:

Filipe Laíns <lains at python.org>

PEP-Delegate:

Status:

Draft

Type:

Informational

Created:

01-Jul-2023

Python-Version:

3.12

---

Table of Contents

• Abstract
• Motivation
• Analysis
  • Introduction
  • Upstream support
    • Projects with decent cross-build support
  • Downstream approaches
    • Cross-build environment
    • Faking the target environment
  • Environment introspection
    • CPython (and similar)
  • Relevant Information
    • When should extensions be linked against libpython?
    • What are prefix, exec_prefix, base_prefix, and base_exec_prefix?
• Case studies
  • crossenv
  • conda-forge
  • Yocto Project
  • Buildroot
  • Pyodide
  • Beeware
  • python-for-android
  • kivy-ios
  • AidLearning
  • QPython
  • pyqtdeploy
  • Chaquopy
  • EDK II
  • ActivePython
  • Termux

Abstract

This PEP attempts to document the status of cross-compilation of downstream projects.

It should give an overview of the approaches currently used by distributors (Linux distros, WASM environment providers, etc.) to cross-compile downstream projects (3rd party extensions, etc.).

Motivation

We write this PEP to express the challenges in cross-compilation and act as a supporting document in future improvement proposals.

Analysis

Introduction

There are a couple different approaches being used to tackle this, with different levels of interaction required from the user, but they all require a significant amount of effort. This is due to the lack of standardized cross-compilation infrastructure on the Python packaging ecosystem, which itself stems from the complexity of cross-builds, making it a huge undertaking.

Upstream support

Some major projects like CPython, setuptools, etc. provide some support to help with cross-compilation, but it’s unofficial and at a best-effort basis. For example, the sysconfig module allows overwriting the data module name via the _PYTHON_SYSCONFIGDATA_NAME environment variable, something that is required for cross-builds, and setuptools accepts patches [1] to tweak/fix its logic to be compatible with popular “environment faking” workflows [2].

The lack of first-party support in upstream projects leads to cross-compilation being fragile and requiring a significant effort from users, but at the same time, the lack of standardization makes it harder for upstreams to improve support as there’s no clarity on how this feature should be provided.

[1]

At the time of writing (Jun 2023), setuptools’ compiler interface code, the component that most of affects cross-compilation, is developed on the pypa/distutils repository, which gets periodically synced to the setuptools repository.

[2]

We specifically mention popular workflows, because this is not standardized. Though, many of the most popular implementations (crossenv, conda-forge’s build system, etc.) work similarly, and this is what we are referring to here. For clarity, the implementations we are referring to here could be described as crossenv-style.

Projects with decent cross-build support

It seems relevant to point out that there are a few modern Python package build-backends with, at least, decent cross-compilation support, those being scikit-build and meson-python. Both these projects integrate external mature build-systems into Python packaging — CMake and Meson, respectively — so cross-build support is inherited from them.

Downstream approaches

Cross-compilation approaches fall in a spectrum that goes from, by design, requiring extensive user interaction to (ideally) almost none. Usually, they’ll be based on one of two main strategies, using a cross-build environment, or faking the target environment.

Cross-build environment

This consists of running the Python interpreter normally and utilizing the cross-build provided by the projects’ build-system. However, as we saw above, upstream support is lacking, so this approach only works for a small-ish set of projects. When this fails, the usual strategy is to patch the build-system code to build use the correct toolchain, system details, etc. [3].

Since this approach often requires package-specific patching, it requires a lot of user interaction.

Examples

python-for-android, kivy-ios, etc.

[3]

The scope of the build-system patching varies between users and usually depends on the their goal — some (eg. Linux distributions) may patch the build-system to support cross-builds, while others might hardcode compiler paths and system information in the build-system, to simply make the build work.

Faking the target environment

Aiming to drop the requirement for user input, a popular approach is trying to fake the target environment. It generally consists of monkeypatching the Python interpreter to get it to mimic the interpreter on the target system, which constitutes of changing many of the sys module attributes, the sysconfig data, etc. Using this strategy, build-backends do not need to have any cross-build support, and should just work without any code changes.

Unfortunately, though, it isn’t possible to truly fake the target environment. There are many reasons for this, one of the main ones being that it breaks code that actually needs to introspect the running interpreter. As a result, monkeypatching Python to look like target is very tricky — to achieve the less amount of breakage, we can only patch certain aspects of the interpreter. Consequently, build-backends may need some code changes, but these are generally much smaller than the previous approach. This is an inherent limitation of the technique, meaning this strategy still requires some user interaction.

Nonetheless, this strategy still works out-of-the-box with significantly more projects than the approach above, and requires much less effort in these cases. It is successful in decreasing the amount of user interaction needed, even though it doesn’t succeed in being generic.

Examples

crossenv, conda-forge, etc.

Environment introspection

As explained above, most build system code is written with the assumption that the target system is the same as where the build is occurring, so introspection is usually used to guide the build.

In this section, we try to document most of the ways this is accomplished. It should give a decent overview of of environment details that are required by build systems.

Snippet	Description	Variance
>>> importlib.machinery.EXTENSION_SUFFIXES [ '.cpython-311-x86_64-linux-gnu.so', '.abi3.so', '.so', ]	Extension (native module) suffixes supported by this interpreter.	This is implementation-defined, but it usually differs based on the implementation, system architecture, build configuration, Python language version, and implementation version — if one exists.
>>> importlib.machinery.SOURCE_SUFFIXES ['.py']	Source (pure-Python) suffixes supported by this interpreter.	This is implementation-defined, but it usually doesn’t differ (outside exotic implementations or systems).
>>> importlib.machinery.all_suffixes() [ '.py', '.pyc', '.cpython-311-x86_64-linux-gnu.so', '.abi3.so', '.so', ]	All module file suffixes supported by this interpreter. It should be the union of all importlib.machinery.*_SUFFIXES attributes.	This is implementation-defined, but it usually differs based on the implementation, system architecture, build configuration, Python language version, and implementation version — if one exists. See the entries above for more information.
>>> sys.abiflags ''	ABI flags, as specified in PEP 3149.	Differs based on the build configuration.
>>> sys.api_version 1013	C API version.	Differs based on the Python installation.
>>> sys.base_prefix /usr	Prefix of the installation-wide directories where platform independent files are installed.	Differs based on the platform, and installation.
>>> sys.base_exec_prefix /usr	Prefix of the installation-wide directories where platform dependent files are installed.	Differs based on the platform, and installation.
>>> sys.