Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 751 – A file format to record Python dependencies for installation reproducibility

Author:
Brett Cannon <brett at python.org>
Status:
Final
Type:
Standards Track
Topic:
Packaging
Created:
24-Jul-2024
Post-History:
25-Jul-2024 30-Oct-2024 15-Jan-2025
Replaces:
665
Resolution:
31-Mar-2025
Table of Contents

Important

This PEP is a historical document. The up-to-date, canonical spec, pylock.toml Specification, is maintained on the PyPA specs page.

×

See the PyPA specification update process for how to propose changes.

Abstract

This PEP proposes a new file format for specifying dependencies to enable reproducible installation in a Python environment. The format is designed to be human-readable and machine-generated. Installers consuming the file should be able to calculate what to install without the need for dependency resolution at install-time.

Motivation

Currently, no standard exists to create an immutable record, such as a lock file, which specifies what direct and indirect dependencies should be installed into a virtual environment.

Considering there are at least five well-known solutions to this problem in the community (PDM, pip freeze, pip-tools, Poetry, and uv), there seems to be an appetite for lock files in general.

Those tools also vary in what locking scenarios they support. For instance, pip freeze and pip-tools only generate single-use lock files for the current environment while PDM, Poetry, and uv can/try to lock for multiple environments and use-cases at once. There’s also concerns around the lack of secure defaults in the face of supply chain attacks by some tools (e.g. including hashes for files).

The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially leaving users unsupported (e.g. Dependabot only supporting select tools, same for cloud providers who can do dependency installations on your behalf, etc.). It also impacts portability between tools, which causes vendor lock-in. By not having compatibility and interoperability it fractures tooling around lock files where both users and tools have to choose what lock file format to use upfront, making it costly to use/switch to other formats (e.g. tooling around auditing a lock file). Rallying around a single format removes this cost/barrier.

The closest the community has to a standard are pip’s requirements files which all the aforementioned tools either use directly as their file format or export to (i.e. requirements.txt). Unfortunately, the format is not a standard but is supported by convention. It’s also designed very much for pip’s needs, limiting its flexibility and ease of use (e.g. it’s a bespoke file format). Lastly, it is not secure by default (e.g. file hash support is entirely an opt-in feature, you have to tell pip to not look for other dependencies outside of what’s in the requirements file, etc.).

Note

Much of the motivation from PEP 665 also applies to this PEP.

Rationale

The file format proposed by this PEP is designed to be human-readable. This is so that the contents of the file can be audited by a human to make sure no undesired dependencies end up being included in the lock file.

The file format is also designed to not require a resolver at install time. This greatly simplifies reasoning about what would be installed when consuming a lock file. It should also lead to faster installs which are much more frequent than creating a lock file.

The data in the file should be consumable by tools not written in Python. This allows for e.g. cloud hosting providers to write their own tool to perform installations in their preferred programming language. This introduces the concept of lockers which write the lock file, and installers which install from a lock file (they can be the same tool).

The file format should promote good security defaults. As the format is not meant to be human-writable, this means having tools provide security-related details is reasonable and not a costly burden.

The contents of a lock file should be able to replace the vast majority of uses of requirements files when used as a lock file (e.g. what pip-tools and pip freeze emit). This means the file format specified by this PEP can, at minimum, act as an export target for tools which have their own internal lock file format.

Lock files can be single-use and multi-use. Single-use lock files are things like requirements.txt files, which serve a single use-case/purpose (hence why it isn’t uncommon for a project to have multiple requirements files, each for a different use-case). Multi-use lock files represent multiple use-cases within a single file, often expressed through extras and Dependency Groups. As such, this PEP supports additions to Environment Markers that allows for specifying extras and dependency groups as appropriate. This allows for a single lock file to support those cases. This not only cuts down on the number of potential lock files, but it also makes it easier when a single package is meant to be consistent across all use-cases (with multiple single-use lock files it requires coordinating across multiple lock files). This support also means that this PEP supports all package installation-related data that can be recorded in a pyproject.toml file. The hope is that this support will allow some tools to drop their internal lock file entirely and solely rely on what this PEP specifies.

Specification

File Name

A lock file MUST be named pylock.toml or match the regular expression r"^pylock\.([^.]+)\.toml$" if a name for the lock file is desired or if multiple lock files exist. The use of the .toml file extension is to make syntax highlighting in editors easier and to reinforce the fact that the file format is meant to be human-readable. The prefix and suffix of a named file MUST be lowercase when possible, for easy detection and removal, e.g.:

if len(filename) > 11 and filename.startswith("pylock.") and filename.endswith(".toml"):
    name = filename.removeprefix("pylock.").removesuffix(".toml")

The expectation is that services that automatically install from lock files will search for:

  1. The lock file with the service’s name and doing the default install
  2. A multi-use pylock.toml with a dependency group with the name of the service
  3. The default install of pylock.toml

E.g. a cloud host service named “spam” would first look for pylock.spam.toml to install from, and if that file didn’t exist then install from pylock.toml and look for a dependency group named “spam” to use if present.

The lock file(s) SHOULD be located in the directory as appropriate for the scope of the lock file. Locking against a single pyproject.toml, for instance, would place the pylock.toml in the same directory. If the lock file covered multiple projects in a monorepo, then the expectation is the pylock.toml file would be in the directory that held all the projects being locked.

File Format