Developing PyArrow#
Coding Style#
We follow a similar PEP8-like coding style to the pandas project. To fix style issues, use the
pre-commit command:
$ pre-commit run --show-diff-on-failure --color=always --all-files python
Unit Testing#
We are using pytest to develop our unit test suite. After building the project you can run its unit tests like so:
$ pushd arrow/python
$ python -m pytest pyarrow
$ popd
Package requirements to run the unit tests are found in
requirements-test.txt and can be installed if needed with pip install -r
requirements-test.txt.
If you get import errors for pyarrow._lib or another PyArrow module when
trying to run the tests, run python -m pytest arrow/python/pyarrow and check
if the editable version of pyarrow was installed correctly.
The project has a number of custom command line options for its test suite. Some tests are disabled by default, for example. To see all the options, run
$ python -m pytest pyarrow --help
and look for the “custom options” section.
Note
There are a few low-level tests written directly in C++. These tests are
implemented in pyarrow/src/arrow/python/python_test.cc,
but they are also wrapped in a pytest-based
test module
run automatically as part of the PyArrow test suite.
Test Groups#
We have many tests that are grouped together using pytest marks. Some of these
are disabled by default. To enable a test group, pass --$GROUP_NAME,
e.g. --parquet. To disable a test group, prepend disable, so
--disable-parquet for example. To run only the unit tests for a
particular group, prepend only- instead, for example --only-parquet.
The test groups currently include:
dataset: Apache Arrow Dataset testsflight: Flight RPC testsgandiva: tests for Gandiva expression compiler (uses LLVM, deprecated since version 24.0.0)hdfs: tests that use libhdfs to access the Hadoop filesystemhypothesis: tests that use thehypothesismodule for generating random test cases. Note that--hypothesisdoesn’t work due to a quirk with pytest, so you have to pass--enable-hypothesislarge_memory: Test requiring a large amount of system RAMorc: Apache ORC testsparquet: Apache Parquet testss3: Tests for Amazon S3tensorflow: Tests that involve TensorFlow
Type Checking#
PyArrow provides type stubs (*.pyi files) for static type checking. These
stubs are located in the pyarrow-stubs/ directory and are automatically
included in the distributed wheel packages.
Running Type Checkers#
We support multiple type checkers. Their configurations are in
pyproject.toml.
mypy
To run mypy on the PyArrow codebase:
$ cd arrow/python
$ mypy
The mypy configuration is in the [tool.mypy] section of pyproject.toml.
pyright
To run pyright:
$ cd arrow/python
$ pyright
The pyright configuration is in the [tool.pyright] section of pyproject.toml.
ty
To run ty (note: currently only partially configured):
$ cd arrow/python
$ ty check
Maintaining Type Stubs#
Type stubs for PyArrow are maintained in the