PEP 748 – A Unified TLS API for Python

Author:: Joop van de Pol <joop.vandepol at trailofbits.com>, William Woodruff <william at yossarian.net>
Sponsor:: Alyssa Coghlan <ncoghlan at gmail.com>
Discussions-To:: Discourse thread
Status:: Draft
Type:: Standards Track
Created:: 27-Jun-2024
Python-Version:: 3.14
Post-History:: 17-Apr-2024
Replaces:: 543

Table of Contents

Abstract

This PEP defines a standard TLS interface in the form of a collection of protocol classes. This interface will allow Python implementations and third-party libraries to provide bindings to TLS libraries other than OpenSSL.

These bindings can be used by tools that expect the interface provided by the Python standard library, with the goal of reducing the dependence of the Python ecosystem on OpenSSL.

Rationale

It has become increasingly clear that robust and user-friendly TLS support is an extremely important part of the ecosystem of any popular programming language. For most of its lifetime, this role in the Python ecosystem has primarily been served by the ssl module, which provides a Python API to the OpenSSL library.

Because the ssl module is distributed with the Python standard library, it has become the overwhelmingly most popular method for handling TLS in Python. A majority of Python libraries, both in the standard library and on the Python Package Index, rely on the ssl module for their TLS connectivity.

Unfortunately, the preeminence of the ssl module has had a number of tied the entire Python ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even in situations where it may provide a worse user experience than alternative TLS implementations, which imposes a cognitive burden and makes it hard to provide “platform-native” experiences.

Problems

The fact that the ssl module is built into the standard library has meant that all standard-library Python networking libraries are entirely reliant on the OpenSSL that the Python implementation has been linked against. This leads to the following issues:

It is difficult to take advantage of new, higher-security TLS without recompiling Python to get a new OpenSSL. While there are third-party bindings to OpenSSL (e.g. pyOpenSSL), these need to be shimmed into a format that the standard library understands, forcing projects that want to use them to maintain substantial compatibility layers.
Windows distributions of Python need to be shipped with a copy of OpenSSL. This puts the CPython development team in the position of being OpenSSL redistributors, potentially needing to ship security updates to the Windows Python distributions when OpenSSL vulnerabilities are released.
macOS distributions of Python need either to be shipped with a copy of OpenSSL or linked against the system OpenSSL library. Apple has formally deprecated linking against the system OpenSSL library, and even if they had not, that library version has been unsupported by upstream for nearly one year as of the time of writing. The CPython development team has started shipping newer OpenSSLs with the Python available from python.org, but this has the same problem as with Windows.
Users may wish to integrate with TLS libraries other than OpenSSL for other reasons, such as maintenance burden versus a system-provided implementation, or because OpenSSL is simply too large and unwieldy for their platform (e.g. for embedded Python). Those users are left with the requirement to use third-party networking libraries that can interact with their preferred TLS library or to shim their preferred library into the OpenSSL-specific ssl module API.

Additionally, the ssl module as implemented today limits the ability of CPython itself to add support for alternative TLS implementations, or remove OpenSSL support entirely, should either of these become necessary or useful. The ssl module exposes too many OpenSSL-specific function calls and features to easily map to an alternative TLS implementation.

Proposal

This PEP proposes to introduce a few new Protocol Classes in Python 3.14 to provide TLS functionality that is not so strongly tied to OpenSSL. It also proposes to update standard library modules to use only the interface exposed by these protocol classes wherever possible. There are three goals here:

To provide a common API surface for both core and third-party developers to target their TLS implementations to. This allows TLS developers to provide interfaces that can be used by most Python code, and allows network developers to have an interface that they can target that will work with a wide range of TLS implementations.
To provide an API that has few or no OpenSSL-specific concepts leak through. The ssl module today has a number of warts caused by leaking OpenSSL concepts through to the API: the new protocol classes would remove those specific concepts.
To provide a path for the core development team to make OpenSSL one of many possible TLS implementations, rather than requiring that it be present on a system in order for Python to have TLS support.

The proposed interface is laid out below.

Interfaces

There are several interfaces that require standardization. Those interfaces are:

Configuring TLS, currently implemented by the SSLContext class in the ssl module.
Providing an in-memory buffer for doing in-memory encryption or decryption with no actual I/O (necessary for asynchronous I/O models), currently implemented by the SSLObject class in the ssl module.
Wrapping a socket object, currently implemented by the SSLSocket class in the ssl module.
Applying TLS configuration to the wrapping objects in (2) and (3). Currently this is also implemented by the SSLContext class in the ssl module.
Specifying TLS cipher suites. There is currently no code for doing this in the standard library: instead, the standard library uses OpenSSL cipher suite strings.
Specifying application-layer protocols that can be negotiated during the TLS handshake.
Specifying TLS versions.
Reporting errors to the caller, currently implemented by the SSLError class in the ssl module.
Specifying certificates to load, either as client or server certificates.
Specifying which trust database should be used to validate certificates presented by a remote peer.
Finding a way to get hold of these interfaces at run time.

For the sake of simplicity, this PEP proposes to remove interfaces (3) and (4), and replace them by a simpler interface that returns a socket which ensures that all communication through the socket is protected by TLS. In other words, this interface treats concepts such as socket initialization, the TLS handshake, Server Name Indication (SNI), etc., as an atomic part of creating a client or server connection. However, in-memory buffers are still supported, as they are useful for asynchronous communication.

Obviously, (5) doesn’t require a protocol class: instead, it requires a richer API for configuring supported cipher suites that can be easily updated with supported cipher suites for different implementations.

(9) is a thorny problem, because in an ideal world the private keys associated with these certificates would never end up in-memory in the Python process (that is, the TLS library would collaborate with a Hardware Security Module (HSM) to provide the private key in such a way that it cannot be extracted from process memory). Thus, we need to provide an extensible model of providing certificates that allows concrete implementations the ability to provide this higher level of security, while also allowing a lower bar for those implementations that cannot. This lower bar would be the same as the status quo: that is, the certificate may be loaded from an in-memory buffer, from a file on disk, or additionally referenced by some arbitrary ID corresponding to a system certificate store.

(10) also represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. Some implementations have specific trust store formats that only they can use (such as the OpenSSL CA directory format that is created by c_rehash), and others may not allow you to specify a trust store that does not include their default trust store. On the other hand, most implementations will support some form of loading custom DER- or PEM-encoded certificates.

For this reason, we need to provide a model that assumes very little about the form that trust stores take, while maintaining type-compatibility with other implementations. The sections “Certificate”, “Private Keys”, and “Trust Store” below go into more detail about how this is achieved.

Finally, this API will split the responsibilities currently assumed by the SSLContext object: specifically, the responsibility for holding and managing configuration and the responsibility for using that configuration to build buffers or sockets.

This is necessary primarily for supporting functionality like Server Name Indication (SNI). In OpenSSL (and thus in the ssl module), the server has the ability to modify the TLS configuration in response to the client telling the server what hostname it is trying to reach. This is mostly used to change the certificate chain so as to present the correct TLS certificate chain for the given hostname. The specific mechanism by which this is done is by returning a new SSLContext object with the appropriate configuration as part of a user-provided SNI callback function.

This is not a model that maps well to other TLS implementations, and puts a burden on users to write callback functions. Instead, we propose that the concrete implementations handle SNI transparently for every user after receiving the relevant certificates.

For this reason, we split the responsibility of SSLContext into two separate objects, which are each split into server and client versions. The TLSServerConfiguration and TLSClientConfiguration objects act as containers for a TLS configuration: the ClientContext and ServerContext objects are instantiated with a TLSClientConfiguration and TLSServerConfiguration object, respectively, and are used to create buffers or sockets. All four objects would be immutable.

Note

The following API declarations uniformly use type hints to aid reading.

Configuration

The TLSServerConfiguration and TLSClientConfiguration concrete classes define objects that can hold and manage TLS configuration. The goals of these classes are as follows:

To provide a method of specifying TLS configuration that avoids the risk of errors in typing (this excludes the use of a simple dictionary).
To provide an object that can be safely compared to other configuration objects to detect changes in TLS configuration, for use with the SNI callback.

These classes are not protocol classes, primarily because they are not expected to have implementation-specific behavior. The responsibility for transforming a TLSServerConfiguration or TLSClientConfiguration object into a useful set of configurations for a given TLS implementation belongs to the Context objects discussed below.

These classes have one other notable property: they are immutable. This is a desirable trait for a few reasons. The most important one is that immutability by default is a good engineering practice. As a side benefit, it allows these objects to be used as dictionary keys, which is potentially useful for specific TLS implementations and their SNI configuration. On top of this, it frees implementations from needing to worry about their configuration objects being changed under their feet, which allows them to avoid needing to carefully synchronize changes between their concrete data structures and the configuration object.

These objects are extendable: that is, future releases of Python may add configuration fields to these objects as they become useful. For backwards-compatibility purposes, new fields are only appended to these objects. Existing fields will never be removed, renamed, or reordered. They are split between client and server to minimize API confusion.

The TLSClientConfiguration class would be defined by the following code:

class TLSClientConfiguration:
    __slots__ = (
        "_certificate_chain",
        "_ciphers",
        "_inner_protocols",
        "_lowest_supported_version",
        "_highest_supported_version",
        "_trust_store",
    )

    def __init__(
        self,
        certificate_chain: SigningChain | None = None,
        ciphers: Sequence[CipherSuite] | None = None,
        inner_protocols: Sequence[NextProtocol | bytes] | None = None,
        lowest_supported_version: TLSVersion | None = None,
        highest_supported_version: TLSVersion | None = None,
        trust_store: TrustStore | None = None,
    ) -> None:
        if inner_protocols is None:
            