Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 688 – Making the buffer protocol accessible in Python

Author:
Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To:
Discourse thread
Status:
Final
Type:
Standards Track
Topic:
Typing
Created:
23-Apr-2022
Python-Version:
3.12
Post-History:
23-Apr-2022, 25-Apr-2022, 06-Oct-2022, 26-Oct-2022
Resolution:
07-Mar-2023
Table of Contents

Important

This PEP is a historical document. The up-to-date, canonical documentation can now be found at Emulating buffer types.

×

See PEP 1 for how to propose changes.

Abstract

This PEP proposes a Python-level API for the buffer protocol, which is currently accessible only to C code. This allows type checkers to evaluate whether objects implement the protocol.

Motivation

The CPython C API provides a versatile mechanism for accessing the underlying memory of an object—the buffer protocol introduced in PEP 3118. Functions that accept binary data are usually written to handle any object implementing the buffer protocol. For example, at the time of writing, there are around 130 functions in CPython using the Argument Clinic Py_buffer type, which accepts the buffer protocol.

Currently, there is no way for Python code to inspect whether an object supports the buffer protocol. Moreover, the static type system does not provide a type annotation to represent the protocol. This is a common problem when writing type annotations for code that accepts generic buffers.

Similarly, it is impossible for a class written in Python to support the buffer protocol. A buffer class in Python would give users the ability to easily wrap a C buffer object, or to test the behavior of an API that consumes the buffer protocol. Granted, this is not a particularly common need. However, there has been a CPython feature request for supporting buffer classes written in Python that has been open since 2012.

Rationale

Current options

There are two known workarounds for annotating buffer types in the type system, but neither is adequate.

First, the current workaround for buffer types in typeshed is a type alias that lists well-known buffer types in the standard library, such as bytes, bytearray, memoryview, and array.array. This approach works for the standard library, but it does not extend to third-party buffer types.

Second, the documentation for typing.ByteString currently states:

This type represents the types bytes, bytearray, and memoryview of byte sequences.

As a shorthand for this type, bytes can be used to annotate arguments of any of the types mentioned above.

Although this sentence has been in the documentation since 2015, the use of bytes to include these other types is not specified in any of the typing PEPs. Furthermore, this mechanism has a number of problems. It does not include all possible buffer types, and it makes the bytes type ambiguous in type annotations. After all, there are many operations that are valid on bytes objects, but not on memoryview objects, and it is perfectly possible for a function to accept bytes but not memoryview objects. A mypy user reports that this shortcut has caused significant problems for the psycopg project.

Kinds of buffers

The C buffer protocol supports many options, affecting strides, contiguity, and support for writing to the buffer. Some of these options would be useful in the type system. For example, typeshed currently provides separate type aliases for writable and read-only buffers.

However, in the C buffer protocol, most of these options cannot be queried directly on the type object. The only way to figure out whether an object supports a particular flag is to actually ask for the buffer. For some types, such as memoryview, the supported flags depend on the instance. As a result, it would be difficult to represent support for these flags in the type system.

Specification

Python-level buffer protocol

We propose to add two Python-level special methods, __buffer__ and __release_buffer__. Python classes that implement these methods are usable as buffers from C code. Conversely, classes implemented in C that support the buffer protocol acquire synthesized methods accessible from Python code.

The __buffer__ method is called to create a buffer from a Python object, for example by the memoryview() constructor. It corresponds to the bf_getbuffer C slot. The Python signature for this method is def __buffer__(self, flags: int, /) -> memoryview: .... The method must return a memoryview object. If the bf_getbuffer slot is invoked on a Python class with a __buffer__ method, the interpreter extracts the underlying Py_buffer from the memoryview returned by the method and returns it to the C caller. Similarly, if Python code calls the __buffer__ method on an instance of a C class that implements bf_getbuffer, the returned buffer is wrapped in a memoryview for consumption by Python code.

The __release_buffer__ method should be called when a caller no longer needs the buffer returned by __buffer__. It corresponds to the bf_releasebuffer C slot. This is an optional part of the buffer protocol. The Python signature for this method is def __release_buffer__(self, buffer: memoryview, /) -> None: .... The buffer to be released is wrapped in a memoryview. When this method is invoked through CPython’s buffer API (for example, through calling memoryview.release on a memoryview returned by __buffer__), the passed memoryview is the same object as was returned by __buffer__. It is also possible to call __release_buffer__ on a C class that implements bf_releasebuffer.

If __release_buffer__ exists on an object, Python code that calls __buffer__ directly on the object must call __release_buffer__ on the same object when it is done with the buffer. Otherwise, resources used by the object may not be reclaimed. Similarly, it is a programming error to call __release_buffer__ without a previous call to __buffer__, or to call it multiple times for a single call to __buffer__. For objects that implement the C buffer protocol, calls to __release_buffer__ where the argument is not a memoryview wrapping the same object will raise an exception. After a valid call to __release_buffer__, the memoryview is invalidated (as if its release() method had been called), and any subsequent calls to __release_buffer__ with the same memoryview will raise an exception. The interpreter will ensure that misuse of the Python API will not break invariants at the C level – for example, it will not cause memory safety violations.

inspect.BufferFlags

To help implementations of __buffer__, we add inspect.BufferFlags, a subclass of enum.IntFlag. This enum contains all flags defined in the C buffer protocol. For example, inspect.BufferFlags.SIMPLE has the same value as the PyBUF_SIMPLE constant.

collections.abc.Buffer

We add a new abstract base classes, collections.abc.Buffer, which requires the __buffer__ method. This class is intended primarily for use in type annotations:

def need_buffer(b: Buffer) -> memoryview:
    return memoryview(b)

need_buffer(b"xy")  # ok
need_buffer("xy")  # rejected by static type checkers

It can also be used in isinstance and issubclass checks:

>>> from collections.abc import Buffer
>>> isinstance(b"xy", Buffer)
True
>>> issubclass(bytes, Buffer)
True
>>> issubclass(memoryview, Buffer)
True
>>> isinstance("xy", Buffer)
False
>>> issubclass(str, Buffer)
False

In the typeshed stub files, the class should be defined as a Protocol, following the precedent of other simple ABCs in collections.abc such as collections.abc.Iterable or collections.abc.Sized.

Example

The following is an example of a Python class that implements the buffer protocol:

import contextlib
import inspect

class MyBuffer:
    def __init__(self, data