Using third-party Python packages

Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these are vendored in third_party/python, while others are installed at runtime via pip.

The dependencies of Mach itself can be found at python/sites/mach.txt. Mach commands may have additional dependencies which are specified at python/sites/<site>.txt.

For example, the following Mach command would have its 3rd-party dependencies declared at python/sites/foo.txt.

@Command(
    "foo-it",
    virtualenv_name="foo",
)
# ...
def foo_it_command():
    import specific_dependency

The format of <site>.txt files are documented further in the MachEnvRequirements class.

Adding a Python package

There’s two ways of using 3rd-party Python dependencies:

Note

For dependencies that meet both restrictions (dependency of Mach/build, and has native code), see the Mach/Build Native 3rd-party Dependencies section below.

pip install the package

To add a pip install-d package dependency, add it to your site’s python/sites/<site>.txt manifest file:

...
pypi:new-package==<version>

If you’d like to lock dependencies and validate hashes, you can alternatively specify a path to a requirements.txt file:

...
requirements-txt:path/to/requirements.txt

The requirements.txt file can be generated using any tool you like, but it must include hashes for all listed packages.

Note

Some tasks are not permitted to use external resources, and for those we can publish packages to an internal PyPI mirror. See how to upload to internal PyPI for more details.

Vendoring Python packages

To vendor a Python package run ./mach vendor python --add <package>~=<major>.<minor>. This will add your dependency to third_party/python/pyproject.toml then begin the re-vendoring process for all dependencies. The pyproject.toml is used by uv to create a lockfile (uv.lock) that ensures all the dependencies are compatible. This lockfile is then used to generate a third_party/python/requirements.txt which is then used by pip to download all dependencies into the third_party/python directory.

Note

The dependency you are attempting to add may not be compatible with what’s already vendored. In this case, the lockfile generation/dependency resolution will fail with an error message along the lines of No solution found when resolving dependencies:. You may be able to get around this by pinning your dependency to a newer or older version. If that doesn’t work you can try modifying the pin(s) of the already vendored dependency(ies) that are causing the conflict(s).

Beware that this is a rather painful process. Changing the version of an already vendored dependency may break functionality somewhere in the codebase. This means that even if you get uv to make a compatible lockfile, you may have caused a breakage somewhere else that uv cannot foresee. It is your responsibility to fix anything you break, otherwise your changes will be rejected or backed out if the issue isn’t discovered until after landing.

If you change pins for packages to workaround issues, please add comments in the third_party/python/pyproject.toml for each necessary pin indicating why it’s needed and which dependency(ies) need it. Doing so will make it much easier for the next person that comes along trying to do the same thing.

After the ./mach vendor python completes successfully, you’ll need to add that package and any new transitive dependencies (you’ll see them added in third_party/python/requirements.txt) to the associated site’s dependency manifest in python/sites/<site>.txt:

...
vendored:third_party/python/new-package
vendored:third_party/python/new-package-dependency-foo
vendored:third_party/python/new-package-dependency-bar
...

To remove a vendored package run ./mach vendor python --remove <package>. This re-creates the lockfile with that dependency removed (along with any transitive dependencies that aren’t shared) and re-vendor everything.

Note

  • You can add or remove multiple packages at the same time: ./mach vendor python --add <package_one> --add <package_two>

  • If desired, you can add/remove dependencies manually in the third_party/python/pyproject.toml. Once you’ve made your changes, just run ./mach vendor python without the --add and/or --remove arguments.

After the ./mach vendor python completes successfully you’ll need to remove the package and transitive dependencies from all the site manifest files (python/sites/<site>.txt) that used the removed package(s).

Note

The following policy applies to ALL vendored packages:

  • Vendored PyPI libraries MUST NOT be modified

  • Vendored libraries