Python's Array: Working With Numeric Data Efficiently

Arrays in Computer Science

To understand arrays better, it helps to zoom out a bit and look at them through the lens of theory. This will clarify some baseline terminology, including:

• Abstract data types
• Data structures
• Data types

Computer science models collections of data as abstract data types (ADTs) that support certain operations like insertion or deletion of elements. These operations must satisfy additional constraints that describe the abstract data type’s unique behaviors.

The word abstract in this context means these data types leave the implementation details up to you, only defining the expected semantics or the set of available operations that an ADT must support. As a result, you can often represent one abstract data type using a few alternative data structures, which are concrete implementations of the same conceptual approach to organizing data.

Programming languages usually provide a few data structures in the form of built-in data types as a convenience so that you don’t have to implement them yourself. This means you can focus on solving more abstract problems instead of starting from scratch every time. For example, the Python dict data type is a hash table data structure that implements the dictionary abstract data type.

To reiterate the meaning of these terms, abstract data types define the desired semantics, data structures implement them, and data types represent data structures in programming languages as built-in syntactic constructs.

Some of the most common examples of abstract data types include these:

• Dictionaries
• Graphs
• Lists
• Queues
• Sets
• Stacks
• Trees

In some cases, you can build more specific kinds of abstract data types on top of existing ADTs by incorporating additional constraints. For instance, you can build a stack by modifying the queue or the other way around.

As you can see, the list of ADTs doesn’t include arrays. That’s because the array is a specific data structure representing the list abstract data type. The list ADT dictates what operations the array must support and which behaviors it should exhibit. If you’ve worked with the Python list, then you should already have a pretty good idea of what the list in computer science is all about.

The list abstract data type is a linear collection of values forming an ordered sequence of elements. These elements follow a specific arrangement, meaning that each element has a position relative to the others, identified by a numeric index that usually starts at zero. The list has a variable but finite length. It may or may not contain values of different types, as well as duplicates.

The interface of the list abstract data type resembles Python’s list, typically including the following operations:

Now that you understand where the array data structure fits into the bigger picture, it’s time to take a closer look at it.

The Array as a Data Structure

To visualize the traditional array data structure graphically, imagine a series of identically sized rectangular boxes lined up side by side, each holding a distinct value of a common type. For example, here’s what an array of integers could look like:

This array has seven elements, indexed from zero to six, all of which are integer numbers. Note that some of these values occur more than once.

At first glance, this looks just like the standard list in Python. After all, both represent the list abstract data type covered earlier. What makes the array data structure special is a few unique features. In particular, an array must be:

• Homogeneous: All elements in an array share a common data type, allowing them to have a uniform size. This is different from Python lists, which are allowed to contain elements of mixed types.
• Contiguous: Array elements are adjacent to each other in the computer memory, occupying a contiguous block of memory space. No other data is allowed between the beginning and the end of an array. Such a layout corresponds to the way your computer memory is logically organized.
• Fixed-Size: Because other data may follow immediately after the block of memory occupied by your array, expanding the block would overwrite that data, potentially causing severe errors or malfunctions. Therefore, when you allocate memory for an array, you must know its size up front without the possibility to change it later.

The first two traits make the array a random-access data structure, letting you instantly retrieve an element at any given index without having to traverse the whole array. When you know the memory address of the first element in the array, you can calculate the address of the desired element by applying a straightforward math formula:

To find the memory address A_i of an element located at index i in the array, you can take the address of your array, which is the same as the address of the first element A₀ at index zero, and then add an offset. Because all elements have equal sizes, you can obtain the memory address offset by multiplying the target index i by the single element’s size in bytes.

This formula works as long as all elements have an identical size and appear next to each other. The downside of such a memory layout is the array’s fixed size, which becomes apparent once you wish to append, insert, or remove elements. However, you can only simulate resizing by making another array, copying existing elements to it, and disregarding your old array.

As a result, you end up frequently asking the operating system for an ever larger block of memory, and you waste time moving data from one place to another. It’s what makes insertions and deletions so inefficient with pure arrays.

One way to mitigate this limitation is to build an abstraction on top of your array that eagerly allocates more memory than necessary and keeps track of the current number of elements against the array’s capacity. When the underlying array becomes full, you can reallocate and copy the data to a larger array. That’s roughly what the Python list does under the surface to cope with dynamic resizing for you.

What about elements of the same type but variable sizes, such as strings?

Strings can have several representations in the computer memory. For example, the length-prefixed representation allocates an extra byte to encode the number of characters that follow. Another popular representation is the null-terminated string, which is essentially an array of bytes terminated by a special sentinel value called the null character. With such a representation, you can create an array of pointers to strings:

The array depicted above consists of pointers, or numeric addresses, each indicating a specific memory area where the given string begins. Even though the individual elements can have variable sizes and be scattered all over your computer’s memory, their corresponding pointers are plain integers of uniform size that fit into an array.

As it turns out, the array data structure isn’t the only way to implement the list abstract data type. Next up, you’ll learn about a popular alternative—the linked list.

Arrays vs Linked Lists

At this point, you know that a list in computer science is an abstract data type and an array is one of its specific implementations. But why would you want to have more than one implementation, and how do you know which one to use?

The choice of a given data structure is vital for performance and memory efficiency. Most data structures offer a trade-off between the two, and they usually provide better time-space complexity for certain operations at the expense of others. Therefore, you should always pick a suitable data structure based on the requirements of the specific problem that you’re trying to solve.

For example, you can implement the list abstract data type using either an array or a linked list. Both are sequences of elements, but arrays will be more suitable in scenarios involving frequent lookups, while linked lists will work better when you anticipate a greater number of insertions or deletions.

Arrays always offer instant access to arbitrary elements but require extra time to add a new element. Conversely, linked lists allow for quick insertions but may take longer to find an element by index. That trade-off is the result of how both data structures organize data in memory.

If you’re interested in how the linked list achieves efficient insertions and deletions, then check out the relevant section of the corresponding article on Wikipedia to learn more. Now, the time has finally come to reveal whether Python has arrays or not.

Arrays in Python

Although a Python list represents the list abstract data type, its implementation isn’t the typical array or linked list. When you look at the CPython source code, you’ll find that the underlying PyListObject is a dynamic array of pointers to objects, allowing Python’s list to combine the benefits of both data structures. It’s cleverly optimized, ensuring quick lookups as well as insertions under most circumstances.

Python takes the idea of keeping an array of pointers to character strings, which was outlined before, a step further. It does so by retaining extra information about the data type of a value being pointed to from an internal array.

This makes Python’s list capable of storing heterogeneous elements of arbitrary types. On the other hand, this and the over-allocation make the Python list less memory-efficient than the classic array that you’d typically find in lower-level programming languages like C.

To sum up, Python internally relies on the array data structure but doesn’t directly expose it to you. The closest you can get to an array in Python is with the standard library’s array or NumPy’s ndarray data types. However, neither one of them provides a true array, as they can only hold numeric values.

On the Real Python website, you’ll find tons of resources on NumPy if you’d like to learn more about that library. And up next, you’ll read about the array module.

Choose the Type of Your Array Elements

The first step before creating an instance of the array class from the array module is to decide on the common numeric type for its elements. Note that all array elements must have the same fixed numeric type. Once you choose from a few well-known numeric types, you can’t change your mind, so choose wisely. Because the numeric type determines the range of possible values, you won’t be able to store numbers outside that range in your array.

Unlike a Python list, which accepts arbitrary kinds of objects, a Python array can only accommodate numbers that fit a precise numeric type defined at the time of its creation. To use a real-life analogy, you can think of a Python list as a conveyor belt at a grocery store checkout counter. In contrast, a Python array is more like a carton specifically designed to hold eggs of a certain size and nothing else.

Python provides comparatively few numeric types through its syntax. They are:

• Integers
• Floating-point numbers
• Complex numbers

Integers in Python have arbitrary precision, making them theoretically unbounded. Floating-point numbers and, by extension, complex numbers have a hard limit defined in the sys.float_info constant.

The Python standard library also ships with more specific types, including fractions and decimal numbers, which are built on top of the primitive types listed above. Beyond that, you’re on your own. While the numeric types available in Python cover the majority of high-level use cases, you might face some challenges when you start dealing with low-level binary data.

To address these challenges, the array module gives you much more granular control over the range of numbers. It exposes numeric types from the C programming language that are unavailable in standard Python. For example, you can use this module to declare an array of long integers or single-precision floats. Additionally, you can specify whether these numbers should be signed or unsigned, whereas all numbers in pure Python are always signed.

There are thirteen numeric types in the array module, each represented by a single-letter type code, which you can reveal by accessing the typecodes variable:

>>> import array
>>> array.typecodes
'bBuhHiIlLqQfd'

Uppercase letters correspond to unsigned numbers, while lowercase letters represent signed ones. Their specific meanings are as follows:

As you can see, there are five pairs of integers consisting of the signed and unsigned flavor, two consistently signed floating-point types, and one wide character type that’s been deprecated since Python 3.3.

The more bytes you allocate, the greater the range of numbers you can represent. For example, a signed short requires two bytes or sixteen bits in memory, which gives you a range of numbers between -32,768 and 32,767. With its unsigned counterpart, you lose the ability to represent negative numbers, but you gain positive numbers up to 65,535. In other words, the range of representable values shifts upward.

Generally, knowing the number of bytes and whether the numeric type is signed lets you find the minimum and maximum values like so:

>>> def get_range(num_bytes, signed=True):
...     num_bits = num_bytes * 8
...     if signed:
...         return -2 ** (num_bits - 1), 2 ** (num_bits - 1) - 1
...     else:
...         return 0, (2 ** num_bits) - 1
...
>>> get_range(num_bytes=2, signed=False)
(0, 65535)

>>> get_range(num_bytes=2, signed=True)
(-32768, 32767)

>>> get_range(num_bytes=4, signed=False)
(0, 4294967295)

>>> get_range(num_bytes=4, signed=True)
(-2147483648, 2147483647)

>>> get_range(num_bytes=8, signed=False)
(0, 18446744073709551615)

>>> get_range(num_bytes=8, signed=True)
(-9223372036854775808, 9223372036854775807)

The resulting tuples represent the minimum and maximum values included in each range of numbers.

Note that the table you saw earlier indicates the minimum number of bytes required to store a value of each data type. In practice, the actual number of bytes may be higher, depending on your system architecture and how the Python interpreter was compiled. For example, the int data type will occupy four bytes instead of just two on most modern computers:

>>> from array import array
>>> array("i").itemsize
4
>>> array("i").typecode
'i'

You can check a concrete array’s .itemsize attribute to find out exactly how many bytes one of its elements will consume in memory. Similarly, you can read the array’s .typecode attribute to show the letter that you used to create that array. Both attributes are read-only.

If you’re loading data from an external source into an array, then you should pick the type that exactly matches the corresponding format. Sometimes, it may not be possible to map a nonstandard numeric type onto one of the type codes in the array module. For example, OpenEXR files often use 16-bit floating-point numbers, while some WAV files encode data using 24-bit signed integers. Later, you’ll learn how to cope with such situations.

Otherwise, if you’re writing data of your own, then use the smallest type that can fit all possible values to optimize the memory use. Say you had an array of people’s ages ranging from 0 to 200. Because age can’t be negative, you may choose a type code corresponding to an unsigned byte, which can hold values from 0 to 255, giving you ample space.

By the way, if anyone you know lives to be older than two hundred years, then stop worrying about data types and ask them about their secret!

Once you determine which numeric type best suits your needs, you can finally create an array.

Create an Empty Array to Populate Later

The array’s constructor expects only one required argument, which must specify the desired type of elements. When you omit the optional second argument, then you’ll define an empty array of numbers. For example, the following code creates an empty array of unsigned shorts, which are 2-byte integers without a sign:

>>> from array import array
>>> empty = array("H")
>>> len(empty)
0

Unlike the classic array data structure with a fixed length, Python’s array data type is dynamically sized. Therefore, you can add elements to an empty array later, and it’ll automatically resize itself accordingly.

Before learning how to populate an existing array with numbers, you’ll explore other ways of creating array instances in Python.

Initialize a New Array Using Another Iterable

Instead of creating an empty array, you can alternatively populate your new array with elements from an initializer value passed as the second argument to the constructor. The initializer must be either a bytes instance or an iterable, provided that it consists of one of the following types:

• Integers
• Floats
• Characters

Unlike some other programming languages, Python has no dedicated type for representing individual characters. An iterable of characters means a Python string or any other collection comprising strings of length one. Also, note that Python arrays don’t support complex numbers, even though they’re another numeric type in the language.

Below are a few examples demonstrating how you can use the Python array to interpret the same byte sequence in different ways. This lets you control how to read binary data from a file, for example. To test this out, generate sample bytes by encoding the snake emoji with UTF-8:

>>> snake_emoji = "\N{snake}".encode("utf-8")

>>> snake_emoji
b'\xf0\x9f\x90\x8d'

>>> len(snake_emoji)
4

>>> for byte in snake_emoji:
...     print(format(byte, "b"))
...
11110000
10011111
10010000
10001101

The resulting byte sequence consists of four bytes, whose bits are depicted in the code block above. You can pass this byte sequence to a new array as the initializer value. Depending on the type code of the array, you’ll get wildly different results:

>>> array("b", snake_emoji)
array('b', [-16, -97, -112, -115])

>>> array("B", snake_emoji)
array('B', [240, 159, 144, 141])

>>> array("h", snake_emoji)
array('h', [-24592, -29296])

>>> array("H", snake_emoji)
array('H', [40944, 36240])

>>> array("i", snake_emoji)
array('i', [-1919901712])

>>> array("I", snake_emoji)
array('I', [2375065584])

>>> array("f", snake_emoji)
array('f', [-8.913188736296437e-31])

When you declare an array of bytes using either the b or B type code, you get a new array of exactly four numbers corresponding to the individual byte values of your initializer. With wider types, such as the 2-byte short or the 4-byte integer and float, you get fewer elements in the array, but their magnitudes become larger.

Additionally, the arrays of signed types denoted with lowercase letter codes make Python interpret the numbers as negative. In this case, all four bytes of your initializer parameter have their most significant bit set to one, which becomes the sign bit.

If you already have another iterable container of integers, floats, or characters, then you can pass it to the array’s constructor to initialize the array with those values:

>>> array("i", [2 ** i for i in range(8)])
array('i', [1, 2, 4, 8, 16, 32, 64, 128])

>>> array("d", (1.4, 2.8, 4, 5.6))
array('d', [1.4, 2.8, 4.0, 5.6])

>>> array("u", "café")
array('u', 'café')

>>> array("u", ["c", "a", "f", "é"])
array('u', 'café')

The first array contains eight integers, which are powers of two that you calculate in a list comprehension. The second array has four double-precision floating-point numbers derived from a tuple of numbers. The last two arrays represent the same characters obtained from a string and a list of individual single-character strings, respectively.

Internally, each Python array is a linear collection of numbers. The textual representation of a non-empty array depicts those numbers, except for the sequence of Unicode characters, which are rendered as a Python string instead. Nevertheless, characters are encoded as numerical code points under the hood, even though you don’t immediately see them.

You can initialize Python arrays in various ways depending on your needs. Moreover, you can clone an existing array, allowing you to work with a copy of the data without affecting your original array.

Use an Existing Array as a Prototype

As a special case, you can provide another Python array as the initializer value to copy its elements. After all, an array is just an iterable of numbers that you can pass to the constructor:

>>> original = array("i", [1, 2, 3])
>>> cloned = array("i", original)
>>> cloned == original
True
>>> cloned is original
False

First, you create an array of three integers, and then you feed it to a new array. The cloned array consists of the same values as the original prototype but exists as a separate entity in the memory. Both arrays compare as equal while being distinct objects, so you can modify them independently of each other.

Note that both arrays contain elements of the same type. As long as the target array declares a superset type of the input array, or more generally, if your numbers can comfortably fit on the available range of values, then you can initialize the array:

>>> array("B", array("Q", range(256)))
array('B', [0, 1, 2, (...), 253, 254, 255])

Here, you initialize an array of unsigned bytes with another array whose elements use at least eight times more memory per number! However, because the actual values are small enough to fit on the target data type, there are no issues.

Something interesting happens when you feed an array of integers into a new array of floating-point numbers:

>>> array("f", array("i", [1, 2, 3]))
array('f', [1.0, 2.0, 3.0])

The single- and double-precision floats are supersets of the input type, so Python automatically promotes your numbers to the broader type. As a side effect, you can freely mix integers and floats when initializing an array of floating-point numbers, as Python will eventually convert them to an appropriate representation:

>>> array("d", (1, 2.8, 4, 5.6))
array('d', [1.0, 2.8, 4.0, 5.6])

Your input tuple consists of Python integers and floats, all of which eventually get converted to the double-precision floating-point numbers.

However, this type conversion won’t work the other way around because it would result in a potential information loss about the fractional part of the floating-point numbers:

>>> array("i", array("f", [3.14, 2.72]))
Traceback (most recent call last):
  ...
TypeError: 'float' object cannot be interpreted as an integer

Trying to initialize an array of integers with floating-point numbers fails. There are many more things that can go wrong when you misuse the Python array, so you’ll next learn how to cope with them.

Avoid Common Pitfalls in Creating Arrays

Remember that all elements that you stuff into an array, either through your initializer parameter or the interface that you’ll soon learn, must be compatible with the array’s declared type code. Otherwise, you’ll expose yourself to a host of problems.

First of all, when you supply an initializer containing a value of an unsupported data type, such as an iterable of complex numbers or Python strings instead of characters, then you’ll get a TypeError:

>>> array("d", [3 + 2j, 3 - 2j])
Traceback (most recent call last):
  ...
TypeError: must be real number, not complex

>>> array("u", ["café", "tea"])
Traceback (most recent call last):
  ...
TypeError: array item must be unicode character