Data Types and In-Memory Data Model#

Apache Arrow defines columnar array data structures by composing type metadata with memory buffers, like the ones explained in the documentation on Memory and IO. These data structures are exposed in Python through a series of interrelated classes:

  • Type Metadata: Instances of pyarrow.DataType, which describe the type of an array and govern how its values are interpreted

  • Schemas: Instances of pyarrow.Schema, which describe a named collection of types. These can be thought of as the column types in a table-like object.

  • Arrays: Instances of pyarrow.Array, which are atomic, contiguous columnar data structures composed from Arrow Buffer objects

  • Record Batches: Instances of pyarrow.RecordBatch, which are a collection of Array objects with a particular Schema

  • Tables: Instances of pyarrow.Table, a logical table data structure in which each column consists of one or more pyarrow.Array objects of the same type.

We will examine these in the sections below in a series of examples.

Type Metadata#

Apache Arrow defines language agnostic column-oriented data structures for array data. These include:

  • Fixed-length primitive types: numbers, booleans, date and times, fixed size binary, decimals, and other values that fit into a given number

  • Variable-length primitive types: binary, string

  • Nested types: list, map, struct, and union

  • Dictionary type: An encoded categorical type (more on this later)

Each data type in Arrow has a corresponding factory function for creating an instance of that type object in Python:

>>> import pyarrow as pa
>>> t1 = pa.int32()
>>> t2 = pa.string()
>>> t3 = pa.binary()
>>> t4 = pa.binary(10)
>>> t5 = pa.timestamp('ms')
>>> t1
DataType(int32)
>>> print(t1)
int32
>>> print(t4)
fixed_size_binary[10]
>>> print(t5)
timestamp[ms]

Note

Different data types might use a given physical storage. For example, int64, float64, and timestamp[ms] all occupy 64 bits per value.

These objects are metadata; they are used for describing the data in arrays, schemas, and record batches. In Python, they can be used in functions where the input data (e.g. Python objects) may be coerced to more than one Arrow type.

The Field type is a type plus a name and optional user-defined metadata:

>>> f0 = pa.field('int32_field', t1)
>>> f0
pyarrow.Field<int32_field: int32>
>>> f0.name
'int32_field'
>>> f0.type
DataType(int32)

Arrow supports nested value types like list, map, struct, and union. When creating these, you must pass types or fields to indicate the data types of the types’ children. For example, we can define a list of int32 values with:

>>> t6 = pa.list_(t1)
>>> t6
ListType(list<item: int32>)

A struct is a collection of named fields:

>>> fields = [
...     pa.field('s0', t1),
...     pa.field('s1', t2),
...     pa.field('s2', t4),
...     pa.field('s3', t6),
... ]
>>> t7 = pa.struct(fields)
>>> print(t7)
struct<s0: int32, s1: string, s2: fixed_size_binary[10], s3: list<item: int32>>

For convenience, you can pass (name, type) tuples directly instead of Field instances:

>>> t8 = pa.struct([('s0', t1), ('s1', t2), ('s2', t4), ('s3', t6)])
>>> print(t8)
struct<s0: int32, s1: string, s2: fixed_size_binary[10], s3: list<item: int32>>
>>> t8 == t7
True

See Data Types API for a full listing of data type functions.

Schemas#

The Schema type is similar to the struct array type; it defines the column names and types in a record batch or table data structure. The pyarrow.schema() factory function makes new Schema objects in Python:

>>> my_schema = pa.schema([('field0', t1),
...                        ('field1', t2),
...                        ('field2', t4),
...                        ('field3', t6)])
>>> my_schema
field0: int32
field1: string
field2: fixed_size_binary[10]
field3: list<item: int32>
  child 0, item: int32

In some applications, you may not create schemas directly, only using the ones that are embedded in IPC messages.

Schemas are immutable, which means you can’t update an existing schema, but you can create a new one with updated values using Schema.set().

>>> updated_field = pa.field('field0_new', pa.int64())
>>> my_schema2 = my_schema.set(0, updated_field)
>>> my_schema2
field0_new: int64
field1: string
field2: fixed_size_binary[10]
field3: list<item: int32>
  child 0, item: int32

Arrays#

For each data type, there is an accompanying array data structure for holding memory buffers that define a single contiguous chunk of columnar array data. When you are using PyArrow, this data may come from IPC tools, though it can also be created from various types of Python sequences (lists, NumPy arrays, pandas data).

A simple way to create arrays is with pyarrow.array, which is similar to the numpy.array function. By default PyArrow will infer the data type for you:

>>> arr = pa.array([1, 2, None, 3])
>>> arr
<pyarrow.lib.Int64Array object at ...>
[
  1,
  2,
  null,
  3
]

But you may also pass a specific data type to override type inference:

>>> pa.array([1, 2], type=pa.uint16())
<pyarrow.lib.UInt16Array object at ...>
[
  1,
  2
]

The array’s type attribute is the corresponding piece of type metadata:

>>> arr.type
DataType(int64)

Each in-memory array has a known length and null count (which will be 0 if there are no null values):

>>> len(arr)
4
>>> arr.null_count
1

Scalar values can be selected with normal indexing. pyarrow.array converts None values to Arrow nulls; we return the special pyarrow.NA value for nulls:

>>> arr[0]
<pyarrow.Int64Scalar: 1>
>>> arr[2]
<pyarrow.Int64Scalar: None>

Arrow data is immutable, so values can be selected but not assigned.

Arrays can be sliced without copying:

>>> arr[1:3]
<pyarrow.lib.Int64Array object at ...>
[
  2,
  null
]

None values and NAN handling#

As mentioned in the above section, the Python object None is always converted to an Arrow null element on the conversion to pyarrow.Array. For the float NaN value which is either represented by the Python object float('nan') or numpy.nan we normally convert it to a valid float value during the conversion. If an integer input is supplied to pyarrow.array that contains np.nan, ValueError is raised.

To handle better compatibility with Pandas, we support interpreting NaN values as null elements. This is enabled automatically on all from_pandas function and can be enabled on the other conversion functions by passing from_pandas=True as a function parameter.

List arrays#

pyarrow.array is able to infer the type of simple nested data structures like lists:

>>> nested_arr = pa.array([[], None, [1, 2], [None, 1]])
>>> print(nested_arr.type)
list<item: int64>

ListView arrays#

pyarrow.array can create an alternate list type called ListView:

>>> nested_arr = pa.array([[], None, [1, 2], [None, 1]], type=pa.list_view(pa.int64()))
>>> print(nested_arr.type)
list_view<item: int64>

ListView arrays have a different set of buffers than List arrays. The ListView array has both an offsets and sizes buffer, while a List array only has an offsets buffer. This allows for ListView arrays to specify out-of-order offsets:

>>> values = [1, 2, 3, 4, 5, 6]
>>> offsets = [4, 2, 0]
>>> sizes = [2, 2, 2]
>>> arr = pa.ListViewArray.from_arrays(offsets, sizes, values)
>>> arr
<pyarrow.lib.ListViewArray object at ...>
[
  [
    5,
    6
  ],
  [
    3,
    4
  ],
  [
    1,
    2
  ]
]

See the format specification for more details on ListView Layout.

Struct arrays#

pyarrow.array is able to infer the schema of a struct type from arrays of dictionaries:

>>> pa.array([{'x': 1, 'y': True}, {'z': 3.4, 'x': 4}])
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: int64
  [
    1,
    4
  ]
-- child 1 type: bool
  [
    true,
    null
  ]
-- child 2 type: double
  [
    null,
    3.4
  ]

Struct arrays can be initialized from a sequence of Python dicts or tuples. For tuples, you must explicitly pass the type:

>>> ty = pa.struct([('x', pa.int8()),
...                 ('y', pa.bool_())])
>>> pa.array([{'x': 1, 'y': True}, {'x': 2, 'y': False}], type=ty)
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: int8
  [
    1,
    2
  ]
-- child 1 type: bool
  [
    true,
    false
  ]
>>> pa.array([(3, True), (4, False)], type=ty)
<pyarrow.lib.StructArray object at ...>
-- is_valid: all not null
-- child 0 type: int8
  [
    3,
    4
  ]
-- child 1 type: bool
  [
    true,
    false
  ]

When initializing a struct array, nulls are allowed both at the struct level and at the individual field level. If initializing from a sequence of Python dicts, a missing dict key is handled as a null value:

>>> pa.array([{'x': 1}, None, {'y': None}], type=ty)
<pyarrow.lib.StructArray object at ...>
-- is_valid:
  [
    true,
    false,
    true
  ]
-- child 0 type: int8
  [
    1,
    0,
    null
  ]
-- child 1 type: bool
  [
    null,
    false,
    null
  ]

You can also construct a struct array from existing arrays for each of the struct’s components. In this case, data storage will be shared with the individual arrays, and no copy is involved:

>>> xs = pa.array([5, 6, 7], type=