What Are Python Wheels and Why Should You Care?

Wheels Make Things Go Fast

Above, you saw a comparison of an installation that fetches a prebuilt wheel and one that downloads a sdist. Wheels make the end-to-end installation of Python packages faster for two reasons:

1. All else being equal, wheels are typically smaller in size than source distributions, meaning they can move faster across a network.
2. Installing from wheels directly avoids the intermediate step of building packages off of the source distribution.

It’s almost guaranteed that the chardet install occurred in a fraction of the time required for uWSGI. However, that’s arguably an unfair apples-to-oranges comparison since chardet is a significantly smaller and less complex package. With a different command, you can create a more direct comparison that will demonstrate just how much of a difference wheels make.

You can make pip ignore its inclination towards wheels by passing the --no-binary option:

$ time python -m pip install \
      --no-cache-dir \
      --force-reinstall \
      --no-binary=:all: \
      cryptography

This command times the installation of the cryptography package, telling pip to use a source distribution even if a suitable wheel is available. Including :all: makes the rule apply to cryptography and all of its dependencies.

On my machine, this takes around thirty-two seconds from start to finish. Not only does the install take a long time, but building cryptography also requires that you have the OpenSSL development headers present and available to Python.

Now you can reinstall cryptography, but this time make sure that pip uses wheels from PyPI. Because pip will prefer a wheel, this is similar to just calling pip install with no arguments at all. But in this case, you can make the intent explicit by requiring a wheel with --only-binary:

$ time python -m pip install \
      --no-cache-dir \
      --force-reinstall \
      --only-binary=cryptography \
      cryptography

This option takes just over four seconds, or one-eighth the time that it took when using only source distributions for cryptography and its dependencies.

What Is a Python Wheel?

A Python .whl file is essentially a ZIP (.zip) archive with a specially crafted filename that tells installers what Python versions and platforms the wheel will support.

A wheel is a type of built distribution. In this case, built means that the wheel comes in a ready-to-install format and allows you to skip the build stage required with source distributions.

A wheel filename is broken down into parts separated by hyphens:

{dist}-{version}(-{build})?-{python}-{abi}-{platform}.whl

Each section in {brackets} is a tag, or a component of the wheel name that carries some meaning about what the wheel contains and where the wheel will or will not work.

Here’s an illustrative example using a cryptography wheel:

cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl

cryptography distributes multiple wheels. Each wheel is a platform wheel, meaning it supports only specific combinations of Python versions, Python ABIs, operating systems, and machine architectures. You can break down the naming convention into parts:

• cryptography is the package name.

• 2.9.2 is the package version of cryptography. A version is a PEP 440-compliant string such as 2.9.2, 3.4, or 3.9.0.a3.

• cp35 is the Python tag and denotes the Python implementation and version that the wheel demands. The cp stands for CPython, the reference implementation of Python, while the 35 denotes Python 3.5. This wheel wouldn’t be compatible with Jython, for instance.

• abi3 is the ABI tag. ABI stands for application binary interface. You don’t really need to worry about what it entails, but abi3 is a separate version for the binary compatibility of the Python C API.

• macosx_10_9_x86_64 is the platform tag, which happens to be quite a mouthful. In this case it can be broken down further into sub-parts:

  • macosx is the macOS operating system.
  • 10_9 is the macOS developer tools SDK version used to compile the Python that in turn built this wheel.
  • x86_64 is a reference to x86-64 instruction set architecture.

The final component isn’t technically a tag but rather the standard .whl file extension. Combined, the above components indicate the target machine that this cryptography wheel is designed for.

Now let’s turn to a different example. Here’s what you saw in the above case for chardet:

chardet-3.0.4-py2.py3-none-any.whl

You can break this down into its tags:

• chardet is the package name.
• 3.0.4 is the package version of chardet.
• py2.py3 is the Python tag, meaning the wheel supports Python 2 and 3 with any Python implementation.
• none is the ABI tag, meaning the ABI isn’t a factor.
• any is the platform. This wheel will work on virtually any platform.

The py2.py3-none-any.whl segment of the wheel name is common. This is a universal wheel that will install with Python 2 or 3 on any platform with any ABI. If the wheel ends in none-any.whl, then it’s very likely a pure-Python package that doesn’t care about a specific Python ABI or CPU architecture.

Another example is the jinja2 templating engine. If you navigate to the downloads page for the Jinja 3.x alpha release, then you’ll see the following wheel:

Jinja2-3.0.0a1-py3-none-any.whl

Notice the lack of py2 here. This is a pure-Python project that will work on any Python 3.x version, but it’s not a universal wheel because it doesn’t support Python 2. Instead, it’s called a pure-Python wheel.

Here are a few more examples of .whl names distributed for some popular open source packages:

Now that you have a thorough understanding of what wheels are, it’s time to talk about what good they serve.

Advantages of Python Wheels

Here’s a testament to wheels from the Python Packaging Authority (PyPA):

Not all developers have the right tools or experiences to build these components written in these compiled languages, so Python created the wheel, a package format designed to ship libraries with compiled artifacts. In fact, Python’s package installer, pip, always prefers wheels because installation is always faster, so even pure-Python packages work better with wheels. (Source)

A fuller description is that wheels benefit both users and maintainers of Python packages alike in a handful of ways:

• Wheels install faster than source distributions for both pure-Python packages and extension modules.

• Wheels are smaller than source distributions. For example, the six wheel is about one-third the size of the corresponding source distribution. This differential becomes even more important when you consider that a pip install for a single package may actually kick off downloading a chain of dependencies.

• Wheels cut setup.py execution out of the equation. Installing from a source distribution runs whatever is contained in that project’s setup.py. As pointed out by PEP 427, this amounts to arbitrary code execution. Wheels avoid this altogether.

• There’s no need for a compiler to install wheels that contain compiled extension modules. The extension module comes included with the wheel targeting a specific platform and Python version.

• pip automatically generates .pyc files in the wheel that match the right Python interpreter.

• Wheels provide consistency by cutting many of the variables involved in installing a package out of the equation.

You can use a project’s Download files tab on PyPI to view the different distributions that are available. For example, pandas distributes a wide array of wheels.

Telling pip What to Download

It’s possible to exert fine-grained control over pip and tell it which format to prefer or avoid. You can use the --only-binary and --no-binary options to do this. You saw these used in an earlier section on installing the cryptography package, but it’s worth taking a closer look at what they do:

$ pushd "$(mktemp -d)"
$ python -m pip download --only-binary :all: --dest . --no-cache six
Collecting six
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
  Saved ./six-1.14.0-py2.py3-none-any.whl
Successfully downloaded six

In this example, you change to a temporary directory to store the download with pushd "$(mktemp -d)". You use pip download rather than pip install so that you can inspect the resulting wheel, but you can replace download with install while keeping the same set of options.

You download the six module with several flags:

• --only-binary :all: tells pip to constrain itself to using wheels and ignore source distributions. Without this option, pip will only prefer wheels but will fall back to source distributions in some scenarios.
• --dest . tells pip to download six to the current directory.
• --no-cache tells pip not to look in its local download cache. You use this option just to illustrate a live download from PyPI since it’s likely you do have a six cache somewhere.

I mentioned earlier that a wheel file is essentially a .zip archive. You can take this statement literally and treat wheels as such. For instance, if you want to view a wheel’s contents, you can use unzip:

$ unzip -l six*.whl
Archive:  six-1.14.0-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    34074  01-15-2020 18:10   six.py
     1066  01-15-2020 18:10   six-1.14.0.dist-info/LICENSE
     1795  01-15-2020 18:10   six-1.14.0.dist-info/METADATA
      110  01-15-2020 18:10   six-1.14.0.dist-info/WHEEL
        4  01-15-2020 18:10   six-1.14.0.dist-info/top_level.txt
      435  01-15-2020 18:10   six-1.14.0.dist-info/RECORD
---------                     -------
    37484                     6 files

six is a special case: it’s actually a single Python module rather than a complete package. Wheel files can also be significantly more complex, as you’ll see later on.

In contrast to --only-binary, you can use --no-binary to do the opposite:

$ python -m pip download --no-binary :all: --dest . --no-cache six
Collecting six
  Downloading six-1.14.0.tar.gz (33 kB)
  Saved ./six-1.14.0.tar.gz
Successfully downloaded six
$ popd

The only change in this example is the switch to --no-binary :all:. This tells pip to ignore wheels even if they’re available and instead download a source distribution.

When might --no-binary be useful? Here are a few cases:

• The corresponding wheel is broken. This is an irony of wheels. They’re designed to make things break less often, but in some cases a wheel can be misconfigured. In this case, downloading and building the source distribution for yourself may be a working alternative.

• You want to apply a small change or patch file to the project and then install it. This is an alternative to cloning the project from its version control system URL.

You can also use the flags described above with pip install. Additionally, instead of :all:, which will apply the --only-binary rule not just to the package you’re installing but to all of its dependencies, you can pass --only-binary and --no-binary a list of specific packages to apply that rule to.

Here are a few examples for installing the URL library yarl. It contains Cython code and depends on multidict, which contains pure C code. There are several options to strictly use or strictly ignore wheels for yarl and its dependencies:

$ # Install `yarl` and use only wheels for yarl and all dependencies
$ python -m pip install --only-binary :all: yarl

$ # Install `yarl` and use wheels only for the `multidict` dependency
$ python -m pip install --only-binary multidict yarl

$ # Install `yarl` and don't use wheels for yarl or any dependencies
$ python -m pip install --no-binary :all: yarl

$ # Install `yarl` and don't use wheels for the `multidict` dependency
$ python -m pip install --no-binary multidict yarl

In this section, you got a glimpse of how to fine-tune the distribution types that pip install will use. While a regular pip install should work with no options, it’s helpful to know these options for special cases.

The manylinux Wheel Tag

Linux comes in many variants and flavors, such as Debian, CentOS, Fedora, and Pacman. Each of these may use slight variations in shared libraries, such as libncurses, and core C libraries, such as glibc.

If you’re writing a C/C++ extension, then this could create a problem. A source file written in C and compiled on Ubuntu Linux isn’t guaranteed to be executable on a CentOS machine or an Arch Linux distribution. Do you need to build a separate wheel for each and every Linux variant?

Luckily, the answer is no, thanks to a specially designed set of tags called the manylinux platform tag family. There are currently three variations:

1. manylinux1 is the original format specified in PEP 513.

2. manylinux2010 is an update specified in PEP 571 that upgrades to CentOS 6 as the underlying OS on which the Docker images are based. The rationale is that CentOS 5.11, which is where the list of allowed libraries in manylinux1 comes from, reached EOL in March 2017 and stopped receiving security patches and bug fixes.

3. manylinux2014 is an update specified in PEP 599 that upgrades to CentOS 7 since CentOS 6 is scheduled to reach EOL in November 2020.

You can find an example of manylinux distributions within the pandas project. Here are two (out of many) from the list of available pandas downloads from PyPI:

pandas-1.0.3-cp37-cp37m-manylinux1_x86_64.whl
pandas-1.0.3-cp37-cp37m-manylinux1_i686.whl

In this case, pandas has built manylinux1 wheels for CPython 3.7 supporting both x86-64 and i686 architectures.

At its core, manylinux is a Docker image built off a certain version of the CentOS operating system. It comes bundled with a compiler suite, multiple versions of Python and pip, and an allowed set of shared libraries.

As of mid-2020, manylinux1 is still the predominant manylinux tag. One reason for this might just be habit. Another might be that support on the client (user) side for manylinux2010 and above is limited to more recent versions of pip:

In other words, if you’re a package developer building manylinux2010 wheels, then someone using your package will need pip 19.0 (released in January 2019) or later to let pip find and install manylinux2010 wheels from PyPI.

Luckily, virtual environments have become more common, meaning that developers can update a virtual environment’s pip without touching the system pip. This isn’t always the case, however, and some Linux distributions still ship with outdated versions of pip.

All that is to say, if you’re installing Python packages on a Linux host, then consider yourself fortunate if the package maintainer has gone out of their way to create manylinux wheels. This will almost guarantee a hassle-free installation of the package regardless of your specific Linux variant or version.