Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 729 – Typing governance process

Author:
Jelle Zijlstra <jelle.zijlstra at gmail.com>, Shantanu Jain <hauntsaninja at gmail.com>
Discussions-To:
Discourse thread
Status:
Active
Type:
Process
Topic:
Governance, Typing
Created:
19-Sep-2023
Post-History:
04-Oct-2023, 20-Sep-2023
Resolution:
20-Nov-2023
Table of Contents

Abstract

This PEP proposes a new way to govern the Python type system: a council that is responsible for maintaining and developing the Python type system. The council will maintain a specification and conformance test suite and will initially be appointed by the Python Steering Council.

Motivation

The Python type system was created by PEP 484, almost ten years ago. The type system is now widely used, and typing has become an important tool for writing good, maintainable Python code. Many changes have been made to the type system to cover more use cases and improve usability. Several type checkers have been created, each with their own strengths. The type annotation syntax has driven several major innovations in the Python ecosystem, such as the popular dataclasses module, runtime type checking and validation by packages such as Pydantic, and static compilation by tools such as mypyc.

However, as the type system has grown, several interrelated problems with the current way to manage the type system have become apparent.

PEPs are the only specification

The Python type system was initially created by a PEP (PEP 484), and changes to the type system are still made by PEPs. The specification for the Python type system, to the extent there is one, consists of this series of PEPs. But Standards Track PEPs aren’t meant to be living documents or specifications; they are change proposals.

An example may illustrate the problem here. Around the same time as the introduction of the typing module by PEP 484, PEP 3156 introduced the asyncio module, another major new feature that has been instrumental to the success of Python 3. Both modules have evolved greatly since their initial creation and inspired changes to the core language.

However, asyncio and typing are different in an essential aspect: a user who uses asyncio interacts only with the standard library itself, while a user of typing has to also think about an external tool, the type checker. The Python language reference covers the symbols in the typing module, but does not (and should not) go into detail on how the full type system should be interpreted by type checkers. That material currently exists only in the PEPs.

This problem is shared by the packaging ecosystem, which attempts to solve it by maintaining a separate set of PyPA specifications.

It’s hard to evolve the specification

Because the PEPs are the only specification we have, anything that could be seen as a change to the specification would theoretically require a new PEP. But that is often too heavy a process for a small change. Sometimes changes are made directly to old PEPs instead, but that goes against the idea that accepted and implemented PEPs become historical documents that should no longer be changed.

Some concrete examples include:

  • PEP 484 explicitly says that typing.NoReturn cannot be used in argument annotations. Nevertheless, type checkers have long accepted such usage.
  • A 2023 discussion noted that PEP 561’s description of partial stubs is unclear, and major type checkers did not implement it exactly as specified.
  • The widely used third-party typing_extensions package provides backports of new type system features. Type checkers are expected to treat symbols in this module the same as symbols in typing, but this is not explicitly specified in any of the PEPs.

The type system is underspecified

While the PEPs provide a specification, they are often not sufficiently precise (sometimes intentionally so). This is especially true as the combinatorial complexity of the type system has grown.

It ends up falling to individual type checkers to decide how to navigate underspecified areas. In cases where type checkers informally coordinate, this results in de facto standards that aren’t clearly recorded anywhere, making the type system less accessible to newcomers. For example:

The Steering Council is not well-placed to solve the above problems

The SC has the entire language in its remit, and is not well-placed to make decisions that are purely about the type system – if only because they don’t have the time to deal with type system arcana alongside their other responsibilities. This is similar in spirit to the reasons why the Steering Council sometimes uses PEP delegation.

Endorsements

This PEP was endorsed by maintainers of all major type checkers, including Rebecca Chen (pytype), Eric Traut (Pyright), and privately by maintainers of mypy and Pyre.

Specification

We propose the creation of a new group, the Typing Council. This group will be responsible for developing and maintaining the Python type system, and for solving the above problems.

The “operations and process” section describes how this group would operate and be governed.

The more exciting “projects” section describes solutions to the above problems that the Typing Council could shepherd.

Mandate

The Typing Council’s mandate is to ensure that the Python type system is:

  • Useful: The type system should serve common use cases. As identified by PEP 484, the primary use case is static analysis, but there are others, such as runtime type checking, static compilation, IDE support, and documentation. The Typing Council should consider all of these use cases when making decisions, and be open to supporting additional use cases as they come up.
  • Usable: The type system should be easy to use for Python developers. It should be ergonomic to write well-typed Python code that is accepted by type checkers. There should be good documentation for the type system.
  • Stable: As the type system matures, users should be able to rely on their typed code continuing to work and be able to trust their mental model for the type system. Changes should be made with care and in a way that minimizes disruption. Nevertheless, the type system should be able to evolve, and it does not make sense to use the same compatibility guidelines for type checker behavior as for Python itself. Of course, the existence and runtime behavior of objects in the typing module does follow Python’s standard compatibility policy in PEP 387.

Operations and process

The council would have three to five members, comprised of prominent community members, such as Python core developers and maintainers of major type checkers. The members should include people affiliated with a variety of projects related to type checking, which may include type checkers, CPython, typeshed, or other projects.

The initial members of the council are: