Python Polars: A Lightning-Fast DataFrame Library

Getting to Know Polars

Polars combines the flexibility and user-friendliness of Python with the speed and scalability of Rust, making it a compelling choice for a wide range of data processing tasks. So, what makes Polars stand out among the crowd? There are many reasons, one of the most prominent being that Polars is lightning fast.

The core of Polars is written in Rust, a language that operates at a low level with no external dependencies. Rust is memory-efficient and gives you performance on par with C or C++, making it a great language to underpin a data analysis library. Polars also ensures that you can utilize all available CPU cores in parallel, and it supports large datasets without requiring all data to be in memory.

Another standout feature of Polars is its intuitive API. If you’re already familiar with libraries like pandas, then you’ll feel right at home with Polars. The library provides a familiar yet unique interface, making it easy to transition to Polars. This means you can leverage your existing knowledge and codebase while taking advantage of Polars’ performance gains.

Polars’ query engine leverages Apache Arrow to execute vectorized queries. Exploiting the power of columnar data storage, Apache Arrow is a development platform designed for fast in-memory processing. This is yet another rich feature that gives Polars an outstanding performance boost.

These are just a few key details that make Polars an attractive data processing library, and you’ll get to see these in action throughout this tutorial. Up next, you’ll get an overview of how to install Polars.

Installing Python Polars

Before installing Polars, make sure you have Python and pip installed on your system. Polars supports Python versions 3.7 and above. To check your Python version, open a terminal or command prompt and run the following command:

$ python --version

If you have Python installed, then you’ll see the version number displayed below the command. If you don’t have Python 3.7 or above installed, follow these instructions to get the correct version.

Polars is available on PyPI, and you can install it with pip. Open a terminal or command prompt, create a new virtual environment, and then run the following command to install Polars:

(venv) $ python -m pip install polars

This command will install the latest version of Polars from PyPI onto your machine. To verify that the installation was successful, start a Python REPL and import Polars:

>>> import polars as pl

If the import runs without error, then you’ve successfully installed Polars. You now have the core of Polars installed on your system. This is a lightweight installation of Polars that allows you to get started without extra dependencies.

Polars has other rich features that allow you to interact with the broader Python ecosystem and external data sources. To use these features, you need to install Polars with the feature flags that you’re interested in. For example, if you want to convert Polars DataFrames to pandas DataFrames and NumPy arrays, then run the following command when installing Polars:

(venv) $ python -m pip install "polars[numpy, pandas]"

This command installs the Polars core and the functionality that you need to convert Polars DataFrames to pandas and NumPy objects. You can find the list of optional dependencies that you can install with Polars in the documentation. Alternatively, you can run the following command to install Polars with all the optional dependencies:

(venv) $ python -m pip install "polars[all]"

This is the best way to go if you feel like you’ll utilize a wide range of Polars features. Otherwise, if you’d like to keep your environment as lightweight as possible, you should only install the optional dependencies that you need.

With Polars installed, you’re now ready to dive in. In the next section, you’ll get an overview of Polars’ core functionalities with DataFrames, expressions, and contexts. You’ll get a feel for Polars syntax and start to see why the library is so powerful.

Getting Started With Polars DataFrames

Like most other data processing libraries, the core data structure used in Polars is the DataFrame. A DataFrame is a two-dimensional data structure composed of rows and columns. The columns of a DataFrame are made up of series, which are one-dimensional labeled arrays.

You can create a Polars DataFrame in a few lines of code. In the following example, you’ll create a Polars DataFrame from a dictionary of randomly generated data representing information about houses. Be sure you have NumPy installed before running this example:

>>> import numpy as np
>>> import polars as pl

>>> num_rows = 5000
>>> rng = np.random.default_rng(seed=7)

>>> buildings_data = {
...      "sqft": rng.exponential(scale=1000, size=num_rows),
...      "year": rng.integers(low=1995, high=2023, size=num_rows),
...      "building_type": rng.choice(["A", "B", "C"], size=num_rows),
...  }
>>> buildings = pl.DataFrame(buildings_data)
>>> buildings
shape: (5_000, 3)
┌─────────────┬──────┬───────────────┐
│ sqft        ┆ year ┆ building_type │
│ ---         ┆ ---  ┆ ---           │
│ f64         ┆ i64  ┆ str           │
╞═════════════╪══════╪═══════════════╡
│ 707.529256  ┆ 1996 ┆ C             │
│ 1025.203348 ┆ 2020 ┆ C             │
│ 568.548657  ┆ 2012 ┆ A             │
│ 895.109864  ┆ 2000 ┆ A             │
│ …           ┆ …    ┆ …             │
│ 408.872783  ┆ 2009 ┆ C             │
│ 57.562059   ┆ 2019 ┆ C             │
│ 3728.088949 ┆ 2020 ┆ C             │
│ 686.678345  ┆ 2011 ┆ C             │
└─────────────┴──────┴───────────────┘

In this example, you first import numpy and polars with aliases of np and pl, respectively. Next, you define num_rows, which determines how many rows will be in the randomly generated data. To generate random numbers, you call default_rng() from NumPy’s random module. This returns a generator that can produce a variety of random numbers according to different probability distributions.

You then define a dictionary with the entries sqft, year, and building_type, which are randomly generated arrays of length num_rows. The sqft array contains floats, year contains integers, and the building_type array contains strings. These will become the three columns of a Polars DataFrame.

To create the Polars DataFrame, you call pl.DataFrame(). The class constructor for a Polars DataFrame accepts two-dimensional data in various forms, a dictionary in this example. You now have a Polars DataFrame that’s ready to use!

When you display buildings in the console, a nice string representation of the DataFrame is displayed. The string representation first prints the shape of the data as a tuple with the first entry telling you the number of rows and the second the number of columns in the DataFrame.

You then see a tabular preview of the data that shows the column names and their data types. For instance, year has type float64, and building_type has type str. Polars supports a variety of data types that are primarily based on the implementation from Arrow.

Polars DataFrames are equipped with many useful methods and attributes for exploring the underlying data. If you’re already familiar with pandas, then you’ll notice that Polars DataFrames use mostly the same naming conventions. You can see some of these methods and attributes in action on the DataFrame that you created in the previous example:

>>> buildings.schema
{'sqft': Float64, 'year': Int64, 'building_type': Utf8}

>>> buildings.head()
shape: (5, 3)
┌─────────────┬──────┬───────────────┐
│ sqft        ┆ year ┆ building_type │
│ ---         ┆ ---  ┆ ---           │
│ f64         ┆ i64  ┆ str           │
╞═════════════╪══════╪═══════════════╡
│ 707.529256  ┆ 1996 ┆ C             │
│ 1025.203348 ┆ 2020 ┆ C             │
│ 568.548657  ┆ 2012 ┆ A             │
│ 895.109864  ┆ 2000 ┆ A             │
│ 206.532754  ┆ 2011 ┆ A             │
└─────────────┴──────┴───────────────┘

>>> buildings.describe()
shape: (9, 4)
┌────────────┬─────────────┬───────────┬───────────────┐
│ describe   ┆ sqft        ┆ year      ┆ building_type │
│ ---        ┆ ---         ┆ ---       ┆ ---           │
│ str        ┆ f64         ┆ f64       ┆ str           │
╞════════════╪═════════════╪═══════════╪═══════════════╡
│ count      ┆ 5000.0      ┆ 5000.0    ┆ 5000          │
│ null_count ┆ 0.0         ┆ 0.0       ┆ 0             │
│ mean       ┆ 994.094456  ┆ 2008.5258 ┆ null          │
│ std        ┆ 1016.641569 ┆ 8.062353  ┆ null          │
│ min        ┆ 1.133256    ┆ 1995.0    ┆ A             │
│ max        ┆ 9307.793917 ┆ 2022.0    ┆ C             │
│ median     ┆ 669.370932  ┆ 2009.0    ┆ null          │
│ 25%        ┆ 286.807549  ┆ 2001.0    ┆ null          │
│ 75%        ┆ 1343.539279 ┆ 2015.0    ┆ null          │
└────────────┴─────────────┴───────────┴───────────────┘

You first look at the schema of the DataFrame with buildings.schema. Polars schemas are dictionaries that tell you the data type of each column in the DataFrame, and they’re necessary for the lazy API that you’ll explore later.

Next, you get a preview of the first five rows of the DataFrame with buildings.head(). You can pass any integer into .head(), depending on how many of the top rows you want to see, and the default number of rows is five. Polars DataFrames also have a .tail() method that allows you to view the bottom rows.

Lastly, you call buildings.describe() to get summary statistics for each column in the DataFrame. This is one of the best ways to get a quick feel for the nature of the dataset that you’re working with. Here’s what each row returned from .describe() means:

• count is the number of observations or rows in the dataset.
• null_count is the number of missing values in the column.
• mean is the arithmetic mean, or average, of the column.
• std is the standard deviation of the column.
• min is the minimum value of the column.
• max is the maximum value of the column.
• median is the median value, or fiftieth percentile, of the column.
• 25% is the twenty-fifth percentile, or first quartile, of the column.
• 75% is the seventy-fifth percentile, or third quartile, of the column.

As an example interpretation, the mean year in the data is between 2008 and 2009, with a standard deviation of just above eight years. The building_type column is missing most of the summary statistics because it consists of categorical values represented by strings.

Now that you’ve seen the basics of creating and interacting with Polars DataFrames, you can start trying more sophisticated queries and get a feel for the library’s power. To do this, you’ll need to understand contexts and expressions, which are the topics of the next section.

Polars Contexts and Expressions

Contexts and expressions are the core components of Polars’ unique data transformation syntax. Expressions refer to computations or transformations that are performed on data columns, and they allow you to apply various operations on the data to derive new results. Expressions include mathematical operations, aggregations, comparisons, string manipulations, and more.

A context refers to the specific environment or situation in which an expression is evaluated. In other words, a context is the fundamental action that you want to perform on your data. Polars has three main contexts:

• Selection: Selecting columns from a DataFrame
• Filtering: Reducing the DataFrame size by extracting rows that meet specified conditions
• Groupby/aggregation: Computing summary statistics within subgroups of the data

You can think of contexts as verbs and expressions as nouns. Contexts determine how the expressions are evaluated and executed, just as verbs determine the actions performed by nouns in language. To get started working with expressions and contexts, you’ll work with the same randomly generated data as before. Here’s the code to create the buildings DataFrame again:

>>> import numpy as np
>>> import polars as pl

>>> num_rows = 5000
>>> rng = np.random.default_rng(seed=7)

>>> buildings_data =