Buffer Protocol

Legacy Wiki Page

This page was migrated from the old MoinMoin-based wiki. Information may be outdated or no longer applicable. For current documentation, see python.org.

Buffer Protocol

This page proposes (now begins to document) a design for a Jython equivalent to the CPython buffer protocol. A good place to start is this worked example (pdf) that motivated the current design.

What is the Buffer API?

The Jython Buffer API is an interface you can use in Java when accessing or implementing certain built-in types, or your own. It is the basis of the type memoryview with which it shares many features.

In CPython, certain objects are based on an underlying memory array or buffer. The CPython designers judged it desirable to be able to access that buffer directly, without intermediate copying. CPython provides this at the C level in the form of the buffer protocol. This is used heavily in the implementation of some core types and standard library modules.

The capability is just as useful in Jython, and has been implemented for version 2.7.

Accessing an Object that has the Buffer API

Objects flag their willingness to provide a buffer by implementing the following interface:

   1 public interface BufferProtocol {
   2     Buffer getBuffer(int flags);   
   3 }

Here is a simple example of the API used, adapted from the implementation of bytearray:

   1     protected void init(BufferProtocol value) throws PyException {
   2         // Get the buffer view
   3         try (PyBuffer view = value.getBuffer(PyBUF.FULL_RO)) {
   4             // Create storage for the bytes
   5             this.storage = new byte[view.getLen()];
   6             this.size = storage.length;
   7             this.offset = 0
   8             // Have the view drop them in
   9             view.copyTo(this.storage, this.offset);
  10         }
  11     }

The reason for the try-with-resources construct is to ensure the view is released when the client has finished with it. (Some objects change behaviour for the lifetime of the exported view. Notably, a bytearray cannot be resized while exporting.)

The flags argument indicates the buffer characteristics wanted by the consumer, and what kind of data organisation the consumer can cope with. The return is an implementation of PyBuffer appropriate to the storage organisation of the exporter. In the example, the consumer says it can cope with any organisation (FULL), but that it will only be reading (RO).

The business about exporter storage organisation is only relevant if you are going to access the actual byte array. Here, the client says it can cope with any organisation, because it is only going to use the buffer’s own copyTo method. Every buffer implementation has to understand its own storage and provide copyTo, charAt, storeAt, and several other methods.

This interface is quite rich, but the Javadoc is thorough. In fact, the interface is defined in two stages: a part that is independent of the type of unit in the buffer, and a part that is definite that the units are bytes.

The type-agnostic part of the interface is PyBUF:

   1 public interface PyBUF {
   2 
   3     boolean isReadonly();
   4 
   5     // Access to buffer (client responsible for indexing)
   6     int getNdim();
   7     int[] getShape();
   8     int getItemsize();
   9     int getLen();
  10     int[] getStrides();
  11     int[] getSuboffsets();
  12     boolean isContiguous(char order);
  13 
  14     // Constants taken from CPython object.h in v3.3
  15     static final int MAX_NDIM = 64;
  16     static final int WRITABLE = 0x0001;
  17     static final int SIMPLE = 0;
  18     static final int FORMAT = 0x0004;
  19     static final int ND = 0x0008;
  20     static final int STRIDES = 0x0010 | ND;
  21     static final int C_CONTIGUOUS = 0x0020 | STRIDES