PEP 710 – Recording the provenance of installed packages
- Author:
- Fridolín Pokorný <fridolin.pokorny at gmail.com>
- Sponsor:
- Donald Stufft <donald at stufft.io>
- PEP-Delegate:
- Paul Moore <p.f.moore at gmail.com>
- Discussions-To:
- Discourse thread
- Status:
- Draft
- Type:
- Standards Track
- Topic:
- Packaging
- Created:
- 27-Mar-2023
- Post-History:
- 03-Dec-2021, 30-Jan-2023, 14-Mar-2023, 03-Apr-2023
Abstract
This PEP describes a way to record the provenance of installed Python distributions.
The record is created by an installer and is available to users in
the form of a JSON file provenance_url.json in the .dist-info directory.
The mentioned JSON file captures additional metadata to allow recording a URL to a
distribution package together with the installed distribution hash.
This proposal is built on top of PEP 610 following its corresponding
canonical PyPA spec and complements direct_url.json
with provenance_url.json for when packages are identified by a name, and
optionally a version.
Motivation
Installing a Python Project involves downloading a Distribution Package from a Package Index and extracting its content to an appropriate place. After the installation process is done, information about the release artifact used as well as its source is generally lost. However, there are use cases for keeping records of distributions used for installing packages and their provenance.
Python wheels can be built with different compiler flags or supporting different wheel tags. In both cases, users might get into a situation in which multiple wheels might be considered by installers (possibly from different package indexes) and immediately finding out which wheel file was actually used during the installation might be helpful. This way, developers can use information about wheels to debug issues making sure the desired wheel was actually installed. Another use case could be tools reporting software installed, such as tools reporting a SBOM (Software Bill of Materials), that might give more accurate reports. Yet another use case could be reconstruction of the Python environment by pinning each installed package to a specific distribution artifact consumed from a Python package index.
Rationale
The motivation described in this PEP is an extension of Recording the Direct URL Origin of installed distributions specification. In addition to recording provenance information for packages installed using a direct URL, installers should also do so for packages installed by name (and optionally version) from Python package indexes.
The idea described in this PEP originated in a tool called micropipenv
that is used to install
distribution packages in containerized
environments (see the reported issue thoth-station/micropipenv#206).
Currently, the assembled containerized application does not implicitly carry
information about the provenance of installed distribution packages
(unless these are installed from full URLs and recorded via direct_url.json).
This requires container image suppliers to link
container images with the corresponding build process, its configuration and
the application source code for checking requirements files in cases when
software present in containerized environments needs to be audited.
The subsequent discussion in the Discourse thread also brought up
pip’s new --report option that can
generate a detailed JSON report about
the installation process. This option could help with the provenance problem
this PEP approaches. Nevertheless, this option needs to be explicitly passed
to pip to obtain the provenance information, and includes additional metadata that
might not be necessary for checking the provenance (such as Python version
requirements of each distribution package). Also, this option is
specific to pip as of the writing of this PEP.
Note the current spec for recording installed packages defines a RECORD file that
records installed files, but not the distribution artifact from which these
files were obtained. Auditing installed artifacts can be performed
based on matching the entries listed in the RECORD file. However, this
technique requires a pre-computed database of files each artifact provides or a
comparison with the actual artifact content. Both approaches are relatively
expensive and time consuming operations which could be eliminated with the
proposed provenance_url.json file.
Recording provenance information for installed distribution packages, both those obtained from direct URLs and by name/version from an index, can simplify auditing Python environments in general, beyond just the specific use case for containerized applications mentioned earlier. A community project pip-audit raised their possible interest in pypa/pip-audit#170.
Specification
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
The provenance_url.json file SHOULD be created in the .dist-info
directory by installers when installing a Distribution Package
specified by name (and optionally by Version Specifier).
This file MUST NOT be created when installing a distribution package from a requirement specifying a direct URL reference (including a VCS URL).
Only one of the files provenance_url.json and direct_url.json (from
Recording the Direct URL Origin of installed distributions specification and the corresponding specification of
the Direct URL Data Structure),
may be present in a given .dist-info directory; installers MUST NOT add
both.
The provenance_url.json JSON file MUST be a dictionary, compliant with
RFC 8259 and UTF-8 encoded.
If present, it MUST contain exactly two keys. The first MUST be url, with
type string. The second key MUST be archive_info with a value defined
below.
The value of the url key MUST be the URL from which the distribution
package was downloaded. If a wheel is built from a source distribution, the
url value MUST be the URL from which the source distribution was
downloaded. If a wheel is downloaded and installed directly, the url field
MUST be the URL from which the wheel was downloaded. As in the Direct URL
Data Structure specification, the url
value MUST be stripped of any sensitive authentication information for security
reasons.
The user:password section of the URL MAY however be composed of environment variables, matching the following regular expression:
\$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?