Polars (software)

Polars
Original author(s)	Ritchie Vink
Developer(s)	Community
Repository	github.com/pola-rs/polars/
Written in	Rust
Operating system	Cross-platform
Type	Technical computing
License	MIT License
Website	pola.rs

Polars is an open-source software library for data manipulation. Polars is built with an OLAP query engine implemented in Rust using Apache Arrow Columnar Format as the memory model. Although built using Rust, there are Python, Node.js, R, and SQL API interfaces to use Polars.

History

The first code to be committed was made on June 23, 2020.^[1] Ritchie Vink and Chiel Peters co-founded a company to develop Polars, after working together at the company Xomnia for five years. In 2023, Vink and Peters successfully closed a seed round of approximately $4 million, which was led by Bain Capital Ventures.

Features

The core object in Polars is the DataFrame, similar to other data processing software libraries.^[2] Contexts and expressions are important concepts to Polars' syntax. A context is the specific environment in which an expression is evaluated. Meanwhile, an expression refers to computations or transformations that are performed on data columns.

Polars has three main contexts:

selection: choosing columns from a DataFrame
filtering: subset a DataFrame by keeping rows that meet specified conditions
group by/aggregation: calculating summary statistics within subgroups of the data

Compared to pandas

Feature differences

Given that Polars was designed to work on a single machine, this prompts many comparisons with the similar data manipulation software, pandas.^[3] One big advantage that Polars has over pandas is performance, where Polars is 5 to 10 times faster than pandas on similar tasks. Additionally, pandas requires around 5 to 10 times as much RAM as the size of the dataset, which compares to the 2 to 4 times needed for Polars. Polars is also designed to use lazy evaluation (where a query optimizer will use the most efficient evaluation after looking at all steps) compared with pandas using eager evaluation (where steps are performed immediately). Some research on comparing pandas and Polars completing data analysis tasks show that Polars is more memory-efficient than pandas.^[4]

Syntax differences

Polars and pandas have similar syntax for reading in data using a read_csv() method, but have different syntax for calculating a rolling mean.^[5]

Code using pandas:

import pandas as pd

# Read in data
df_temp = pd.read_csv(
    "temp_record.csv", index_col="date", parse_dates=True, dtype={"temp": int}
)

# Explore data
print(df_temp.dtypes)
print(df_temp.head())

# Calculate rolling mean
df_temp.rolling(2).mean()

Code using Polars:

import polars as pl

# Read in data
df_temp = pl.read_csv(
    "temp_record.csv", try_parse_dates=True, dtypes={"temp": int}
).set_sorted("date")

# Explore data
print(df_temp.dtypes)
print(df_temp.head())

# Calculate rolling average
df_temp.rolling("date", period="2d").agg(pl.mean("temp"))