Tools & Libraries 20 min read

Beyond Pandas: A Practical Guide to Polars and DuckDB for Python Data Science

Master the modern Python data stack by learning when and how to use Polars and DuckDB alongside pandas. This hands-on guide covers real code examples, performance benchmarks, and hybrid workflow patterns for 2026.

Editorial Team

Introduction: The Python Data Landscape Has Shifted (And That's a Good Thing)

If you've been working with Python for data science, pandas has probably been the backbone of your workflow for years. And honestly, for good reason — pandas essentially defined how Python handles tabular data. But here's the thing: in 2026, the landscape looks fundamentally different from even a couple of years ago. Two libraries have emerged as serious, production-ready alternatives: Polars, a blazing-fast DataFrame library built on Rust, and DuckDB, an in-process analytical SQL engine that people often call "the SQLite of analytics."

This isn't another "pandas is dead" hot take. Far from it.

Pandas 3.0 dropped in January 2026 with some massive improvements — Copy-on-Write semantics, better string handling, and more. But the reality is that modern data science demands tools optimized for different jobs, and the smartest practitioners I've seen are building hybrid stacks that leverage the strengths of all three libraries.

In this guide, we'll take a hands-on, practical approach to Polars and DuckDB. You'll learn when to reach for each tool, see real code examples side by side, get a feel for the performance differences that actually matter in production, and walk away with a concrete strategy for weaving them into your existing Python data workflows.

Why Pandas Alone Isn't Enough Anymore

Before we dive into the alternatives, let's be upfront about where pandas struggles — not to bash it, but to understand exactly what problems Polars and DuckDB were designed to solve.

The Memory Problem

Pandas loads entire datasets into memory as Python objects, typically requiring 2-5x the file size in RAM. A 2 GB CSV file? It can easily balloon to 8-10 GB in a pandas DataFrame. For datasets that fit comfortably in memory, this is totally fine. But as datasets grow — and let's be real, they always grow — you hit a wall pretty fast.

The Single-Thread Bottleneck

Most pandas operations run on a single CPU core. On modern machines with 8, 16, or even 64 cores, that means you're leaving the vast majority of your compute power just sitting there idle. Groupby operations, joins, and aggregations on large datasets can take minutes when they really should take seconds.

The API Consistency Issues

Despite the improvements in pandas 3.0, the library carries decades of API decisions that can trip you up. The distinction between apply, map, and transform; the sometimes-confusing axis parameter; the ambiguity between in-place and copy operations (improved but not entirely gone) — these all add cognitive load that newer libraries sidestep by designing from scratch.

None of these are dealbreakers for every use case. But if you're working with datasets above a few hundred megabytes, processing data in production pipelines, or you simply want faster iteration in your notebooks, Polars and DuckDB offer some really compelling solutions.

Polars: The Speed Demon DataFrame Library

Polars is a DataFrame library written in Rust with first-class Python bindings. It was created by Ritchie Vink in 2020 and hit its stable 1.0 release in 2024. As of early 2026, Polars is at version 1.38, with a 2.0 roadmap in discussion. And this is important — it's not a pandas wrapper or a pandas-like API shim. It's an entirely independent implementation with its own execution engine, query optimizer, and memory model.

Installing and Getting Started

pip install polars

# Optional: install with all features (cloud support, spreadsheet I/O, etc.)
pip install 'polars[all]'

Let's start with a simple example to get a feel for how Polars differs from pandas:

import polars as pl

# Create a DataFrame
df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
    "department": ["Engineering", "Marketing", "Engineering", "Marketing", "Engineering"],
    "salary": [95000, 72000, 88000, 78000, 102000],
    "years_experience": [5, 3, 4, 6, 8]
})

print(df)
# shape: (5, 4)
# ┌─────────┬─────────────┬────────┬──────────────────┐
# │ name    ┆ department  ┆ salary ┆ years_experience │
# │ ---     ┆ ---         ┆ ---    ┆ ---              │
# │ str     ┆ str         ┆ i64    ┆ i64              │
# ╞═════════╪═════════════╪════════╪══════════════════╡
# │ Alice   ┆ Engineering ┆ 95000  ┆ 5                │
# │ Bob     ┆ Marketing   ┆ 72000  ┆ 3                │
# │ Charlie ┆ Engineering ┆ 88000  ┆ 4                │
# │ Diana   ┆ Marketing   ┆ 78000  ┆ 6                │
# │ Eve     ┆ Engineering ┆ 102000 ┆ 8                │
# └─────────┴─────────────┴────────┴──────────────────┘

Right away you'll notice Polars shows you the column types in the output. It's a small detail, but it saves a surprising amount of debugging time.

Expressions: The Heart of Polars

The biggest conceptual shift between pandas and Polars is the expression system. In pandas, you typically chain methods on DataFrames or Series. In Polars, you build expressions using pl.col() that describe what computation to perform, then pass those expressions to DataFrame methods like select, filter, with_columns, and group_by.

It feels a little weird at first if you're coming from pandas. But stick with it — it clicks fast.

# Select specific columns with transformations
result = df.select(
    pl.col("name"),
    pl.col("salary").alias("annual_salary"),
    (pl.col("salary") / 12).round(2).alias("monthly_salary"),
    (pl.col("salary") / pl.col("years_experience")).alias("salary_per_year_exp")
)

# Filter rows
senior_engineers = df.filter(
    (pl.col("department") == "Engineering") &
    (pl.col("years_experience") >= 5)
)

# Add new columns without dropping existing ones
df_enriched = df.with_columns(
    (pl.col("salary") * 1.1).round(0).cast(pl.Int64).alias("salary_after_raise"),
    pl.when(pl.col("years_experience") >= 5)
      .then(pl.lit("Senior"))
      .otherwise(pl.lit("Junior"))
      .alias("level")
)

This expression-based approach isn't just a style preference — it's what enables Polars to optimize and parallelize your queries automatically under the hood.

Lazy Evaluation: Let the Optimizer Do the Work

Polars offers two execution modes: eager (immediate execution, like pandas) and lazy (deferred execution with query optimization). Lazy evaluation is honestly one of Polars' most powerful features and the key to getting its best performance.

# Eager mode — executes immediately
result_eager = df.filter(pl.col("salary") > 80000).select("name", "salary")

# Lazy mode — builds a query plan, optimizes, then executes on .collect()
result_lazy = (
    df.lazy()
    .filter(pl.col("salary") > 80000)
    .select("name", "salary")
    .collect()  # Execution happens here
)

# Both produce the same result, but lazy mode can optimize the plan

When you use lazy mode, Polars builds a logical query plan and runs it through an optimizer before execution. The optimizer can:

  • Push predicates down — filters get applied as early as possible, reducing the data processed by later steps
  • Project columns — only the columns you actually use are read from disk
  • Eliminate common subexpressions — repeated computations are calculated just once
  • Optimize join order — joins get reordered for efficiency

This becomes especially powerful when you're reading from files:

# Scan a large CSV file lazily — nothing is loaded into memory yet
lazy_df = pl.scan_csv("large_dataset.csv")

# Build a complex query
result = (
    lazy_df
    .filter(pl.col("date") >= "2025-01-01")
    .group_by("category")
    .agg(
        pl.col("revenue").sum().alias("total_revenue"),
        pl.col("revenue").mean().alias("avg_revenue"),
        pl.col("order_id").n_unique().alias("unique_orders"),
        pl.col("quantity").sum().alias("total_quantity")
    )
    .sort("total_revenue", descending=True)
    .head(20)
    .collect()  # Only now does Polars read and process the file
)

# Polars will only read the columns it needs and apply filters during scan

GroupBy and Aggregations

GroupBy operations are where Polars really flexes compared to pandas, both in API clarity and raw performance:

import polars as pl
import numpy as np

# Create a larger example dataset
np.random.seed(42)
n = 100_000
sales_df = pl.DataFrame({
    "store_id": np.random.choice(["S001", "S002", "S003", "S004", "S005"], n),
    "product": np.random.choice(["Widget", "Gadget", "Doohickey", "Thingamajig"], n),
    "quantity": np.random.randint(1, 50, n),
    "unit_price": np.random.uniform(5.0, 100.0, n).round(2),
    "date": pl.date_range(
        pl.date(2024, 1, 1), pl.date(2025, 12, 31), eager=True
    ).sample(n, with_replacement=True)
})

# Multi-level groupby with multiple aggregations
summary = (
    sales_df.lazy()
    .with_columns(
        (pl.col("quantity") * pl.col("unit_price")).alias("revenue")
    )
    .group_by("store_id", "product")
    .agg(
        pl.col("revenue").sum().alias("total_revenue"),
        pl.col("revenue").mean().alias("avg_transaction"),
        pl.col("quantity").sum().alias("total_units_sold"),
        pl.len().alias("transaction_count"),
        pl.col("date").min().alias("first_sale"),
        pl.col("date").max().alias("last_sale"),
    )
    .sort("total_revenue", descending=True)
    .collect()
)

print(summary.head(10))

Joins in Polars

Polars supports all the standard join types with clean, straightforward syntax:

# Create two DataFrames
orders = pl.DataFrame({
    "order_id": [1, 2, 3, 4, 5],
    "customer_id": [101, 102, 101, 103, 104],
    "amount": [250.0, 180.0, 320.0, 95.0, 410.0]
})

customers = pl.DataFrame({
    "customer_id": [101, 102, 103, 105],
    "name": ["Alice", "Bob", "Charlie", "Eve"],
    "tier": ["Gold", "Silver", "Bronze", "Gold"]
})

# Inner join
inner = orders.join(customers, on="customer_id", how="inner")

# Left join — keeps all orders, even those without matching customers
left = orders.join(customers, on="customer_id", how="left")

# Join on different column names
orders_v2 = orders.rename({"customer_id": "cust_id"})
joined = orders_v2.join(
    customers,
    left_on="cust_id",
    right_on="customer_id",
    how="left"
)

DuckDB: SQL Power Inside Your Python Process

DuckDB takes a fundamentally different approach from both pandas and Polars. Instead of a DataFrame API, it gives you a full-featured SQL engine that runs entirely inside your Python process — no server, no configuration, no network overhead. Think of it as SQLite, but designed from the ground up for analytical queries instead of transactional workloads.

Installing and Getting Started

pip install duckdb

DuckDB's simplicity is genuinely its superpower. Here's how little code you need to start querying data:

import duckdb

# Run SQL directly — no setup, no configuration
result = duckdb.sql("SELECT 42 AS answer, 'hello' AS greeting")
result.show()

# Query a CSV file without loading it into memory first
result = duckdb.sql("""
    SELECT
        category,
        COUNT(*) AS count,
        AVG(price) AS avg_price,
        SUM(quantity) AS total_quantity
    FROM 'sales_data.csv'
    GROUP BY category
    ORDER BY total_quantity DESC
""")
print(result.df())  # Convert to pandas DataFrame

That's it. No pd.read_csv(), no schema definitions, no connection strings. DuckDB auto-detects the file format, infers column types, and runs your query with a columnar, vectorized execution engine optimized for analytical workloads.

Querying Files Directly

One of DuckDB's most compelling features (and one I find myself reaching for constantly) is its ability to query files directly — CSVs, Parquet files, JSON, and more — without any explicit loading step:

import duckdb

# Query a single Parquet file
result = duckdb.sql("""
    SELECT
        customer_segment,
        EXTRACT(YEAR FROM order_date) AS order_year,
        EXTRACT(MONTH FROM order_date) AS order_month,
        SUM(revenue) AS monthly_revenue,
        COUNT(DISTINCT customer_id) AS unique_customers
    FROM 'orders.parquet'
    WHERE order_date >= '2025-01-01'
    GROUP BY customer_segment, order_year, order_month
    ORDER BY order_year, order_month, monthly_revenue DESC
""")

# Query multiple Parquet files using glob patterns
result = duckdb.sql("""
    SELECT *
    FROM 'data/partitioned/year=*/month=*/*.parquet'
    WHERE revenue > 1000
""")

# Query a remote Parquet file from S3 (with httpfs extension)
duckdb.sql("INSTALL httpfs; LOAD httpfs;")
result = duckdb.sql("""
    SELECT COUNT(*), AVG(trip_distance)
    FROM 's3://nyc-tlc/trip data/yellow_tripdata_2024-01.parquet'
""")

DuckDB + Pandas: Seamless Integration

Perhaps DuckDB's killer feature for existing Python data scientists is how seamlessly it integrates with pandas DataFrames. You can query them directly with SQL — no conversion step required:

import pandas as pd
import duckdb

# Create a pandas DataFrame
orders_df = pd.DataFrame({
    "order_id": range(1, 10001),
    "customer_id": np.random.randint(1, 500, 10000),
    "product": np.random.choice(["A", "B", "C", "D"], 10000),
    "amount": np.random.uniform(10, 500, 10000).round(2),
    "order_date": pd.date_range("2024-01-01", periods=10000, freq="h")
})

# Query the pandas DataFrame directly with SQL!
result = duckdb.sql("""
    SELECT
        product,
        DATE_TRUNC('month', order_date) AS month,
        COUNT(*) AS order_count,
        SUM(amount) AS total_revenue,
        AVG(amount) AS avg_order_value
    FROM orders_df
    GROUP BY product, month
    ORDER BY month, total_revenue DESC
""").df()  # .df() converts back to pandas

print(result.head(10))

This is incredibly powerful for existing pandas workflows. You don't have to rewrite your entire pipeline — you can surgically replace just the slow parts (groupbys, joins, window functions) with DuckDB SQL queries while keeping everything else in pandas. It's the path of least resistance, honestly.

Window Functions: Where DuckDB Really Excels

Window functions are one area where SQL truly shines, and DuckDB makes them almost effortless:

import duckdb
import pandas as pd

# Assume we have a sales DataFrame
sales_df = pd.DataFrame({
    "salesperson": ["Alice", "Bob", "Alice", "Charlie", "Bob", "Alice"],
    "region": ["East", "East", "West", "West", "West", "East"],
    "quarter": ["Q1", "Q1", "Q1", "Q1", "Q1", "Q2"],
    "revenue": [50000, 45000, 62000, 38000, 55000, 71000]
})

# Complex window functions that would be verbose in pandas
result = duckdb.sql("""
    SELECT
        salesperson,
        region,
        quarter,
        revenue,
        SUM(revenue) OVER (PARTITION BY salesperson ORDER BY quarter) AS running_total,
        RANK() OVER (PARTITION BY quarter ORDER BY revenue DESC) AS revenue_rank,
        revenue - LAG(revenue) OVER (PARTITION BY salesperson ORDER BY quarter) AS revenue_change,
        revenue * 100.0 / SUM(revenue) OVER (PARTITION BY quarter) AS pct_of_quarter
    FROM sales_df
    ORDER BY quarter, revenue DESC
""").df()

print(result)

Writing the equivalent in pandas would require multiple groupby().transform() calls, rank() methods, and shift() operations. Far more verbose, and honestly, harder to read too.

Persistent Databases and Tables

While DuckDB works great as a transient query engine, it can also persist data to disk when you need it:

import duckdb

# Create a persistent database
con = duckdb.connect("my_analytics.duckdb")

# Create a table from a CSV file
con.sql("""
    CREATE TABLE IF NOT EXISTS sales AS
    SELECT * FROM 'raw_sales_data.csv'
""")

# Create views for common queries
con.sql("""
    CREATE OR REPLACE VIEW monthly_summary AS
    SELECT
        DATE_TRUNC('month', sale_date) AS month,
        category,
        SUM(amount) AS total_sales,
        COUNT(*) AS transaction_count
    FROM sales
    GROUP BY month, category
""")

# Query the view
result = con.sql("SELECT * FROM monthly_summary ORDER BY month DESC LIMIT 10")
result.show()

con.close()

Head-to-Head: Pandas vs. Polars vs. DuckDB

Let's compare all three libraries on the same tasks so you can see the real differences in syntax and approach. We'll work with a hypothetical dataset of 10 million rows of e-commerce transactions.

Task 1: Reading and Filtering Data

# === PANDAS ===
import pandas as pd

df = pd.read_csv("transactions.csv")
filtered = df[
    (df["amount"] > 100) &
    (df["category"] == "Electronics")
][["order_id", "customer_id", "amount", "order_date"]]


# === POLARS ===
import polars as pl

filtered = (
    pl.scan_csv("transactions.csv")
    .filter(
        (pl.col("amount") > 100) &
        (pl.col("category") == "Electronics")
    )
    .select("order_id", "customer_id", "amount", "order_date")
    .collect()
)


# === DUCKDB ===
import duckdb

filtered = duckdb.sql("""
    SELECT order_id, customer_id, amount, order_date
    FROM 'transactions.csv'
    WHERE amount > 100 AND category = 'Electronics'
""").pl()  # or .df() for pandas output

Key difference: Pandas reads the entire file into memory before filtering. Polars with scan_csv and DuckDB both push the filter down to the scan phase, reading less data and using less memory. That distinction matters a lot when your files are measured in gigabytes.

Task 2: GroupBy with Multiple Aggregations

# === PANDAS ===
result = (
    df.groupby(["category", "region"])
    .agg(
        total_revenue=("amount", "sum"),
        avg_order=("amount", "mean"),
        order_count=("order_id", "count"),
        unique_customers=("customer_id", "nunique")
    )
    .reset_index()
    .sort_values("total_revenue", ascending=False)
)


# === POLARS ===
result = (
    df_pl.lazy()
    .group_by("category", "region")
    .agg(
        pl.col("amount").sum().alias("total_revenue"),
        pl.col("amount").mean().alias("avg_order"),
        pl.len().alias("order_count"),
        pl.col("customer_id").n_unique().alias("unique_customers")
    )
    .sort("total_revenue", descending=True)
    .collect()
)


# === DUCKDB ===
result = duckdb.sql("""
    SELECT
        category, region,
        SUM(amount) AS total_revenue,
        AVG(amount) AS avg_order,
        COUNT(*) AS order_count,
        COUNT(DISTINCT customer_id) AS unique_customers
    FROM df
    GROUP BY category, region
    ORDER BY total_revenue DESC
""").df()

Task 3: Joining Two Datasets

# === PANDAS ===
result = (
    orders.merge(customers, on="customer_id", how="left")
    .merge(products, on="product_id", how="left")
)


# === POLARS ===
result = (
    orders.lazy()
    .join(customers.lazy(), on="customer_id", how="left")
    .join(products.lazy(), on="product_id", how="left")
    .collect()
)


# === DUCKDB ===
result = duckdb.sql("""
    SELECT o.*, c.name, c.tier, p.product_name, p.category
    FROM orders o
    LEFT JOIN customers c ON o.customer_id = c.customer_id
    LEFT JOIN products p ON o.product_id = p.product_id
""").df()

Performance Benchmarks: Real Numbers

Based on benchmarks from early 2026, here's how the three libraries stack up on a 10 million row dataset with typical analytical operations:

  • CSV Read: Polars is roughly 7.7x faster than pandas, DuckDB about 6x faster
  • GroupBy Aggregation: Polars is ~8.7x faster than pandas, DuckDB ~9.4x faster
  • Joins: Polars comes in at ~5x faster than pandas, DuckDB around 4x
  • Window Functions: DuckDB leads here, clocking in at ~10x faster than pandas equivalents
  • Peak Memory (large joins): Polars uses 30-60% less than pandas; DuckDB falls between the two but can spill to disk when memory gets tight

These numbers will obviously vary based on your specific data, hardware, and query complexity. The takeaway isn't that one library is universally faster — it's that both Polars and DuckDB consistently outperform pandas by a significant margin on analytical workloads.

The Hybrid Stack: Using All Three Together

So here's the real power move in 2026 — it isn't choosing one library. It's knowing when to use each. Here's a practical pattern for combining all three in a data pipeline:

import duckdb
import polars as pl
import pandas as pd

# Stage 1: DuckDB for heavy SQL-driven data prep
# Use DuckDB to filter, join, and aggregate across multiple large files
clean_data = duckdb.sql("""
    SELECT
        t.transaction_id,
        t.customer_id,
        t.amount,
        t.transaction_date,
        c.segment,
        c.region,
        p.category,
        p.subcategory
    FROM 'raw_transactions/*.parquet' t
    LEFT JOIN 'customers.parquet' c ON t.customer_id = c.customer_id
    LEFT JOIN 'products.parquet' p ON t.product_id = p.product_id
    WHERE t.transaction_date >= '2025-01-01'
      AND t.amount > 0
      AND c.is_active = true
""")

# Stage 2: Polars for fast, complex transformations
features_df = (
    clean_data.pl()  # Convert DuckDB result to Polars
    .lazy()
    .with_columns(
        pl.col("transaction_date").dt.weekday().alias("day_of_week"),
        pl.col("transaction_date").dt.month().alias("month"),
        pl.col("amount").log().alias("log_amount"),
    )
    .with_columns(
        pl.col("amount")
          .mean()
          .over("customer_id")
          .alias("customer_avg_spend"),
        pl.col("amount")
          .count()
          .over("customer_id")
          .alias("customer_transaction_count"),
        pl.col("amount")
          .rank("dense")
          .over("category")
          .alias("amount_rank_in_category"),
    )
    .collect()
)

# Stage 3: Pandas for ML integration and ecosystem compatibility
# scikit-learn, matplotlib, seaborn all expect pandas DataFrames
pandas_df = features_df.to_pandas()

from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier

X = pandas_df[["log_amount", "day_of_week", "month",
                "customer_avg_spend", "customer_transaction_count"]]
y = pandas_df["segment"].map({"Premium": 1, "Standard": 0})

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = GradientBoostingClassifier()
model.fit(X_train, y_train)
print(f"Accuracy: {model.score(X_test, y_test):.3f}")

This three-stage pattern — DuckDB for SQL-heavy data prep, Polars for fast transformations, pandas for ecosystem integration — is becoming the go-to approach for serious Python data practitioners in 2026. I've seen variations of this in production at multiple companies now.

Interoperability: Moving Data Between Libraries

One of the practical concerns with using multiple libraries is data conversion overhead. The good news? All three libraries share Apache Arrow as a common memory format, making conversions nearly zero-copy in many cases:

import polars as pl
import pandas as pd
import duckdb

# Polars → Pandas
polars_df = pl.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
pandas_df = polars_df.to_pandas()

# Pandas → Polars
polars_df = pl.from_pandas(pandas_df)

# DuckDB → Polars
result = duckdb.sql("SELECT 1 AS a, 'x' AS b").pl()

# DuckDB → Pandas
result = duckdb.sql("SELECT 1 AS a, 'x' AS b").df()

# Polars → DuckDB (query a Polars DataFrame directly)
polars_df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
result = duckdb.sql("SELECT SUM(a), AVG(b) FROM polars_df")

# DuckDB → Arrow → Polars (zero-copy path)
arrow_table = duckdb.sql("SELECT * FROM 'data.parquet'").arrow()
polars_df = pl.from_arrow(arrow_table)

The Arrow-based interchange means you can mix and match libraries freely without worrying about expensive serialization or deserialization steps. It just works.

Narwhals: Write Once, Run on Any DataFrame Library

As the ecosystem fragments across pandas, Polars, and other DataFrame libraries, a project called Narwhals has emerged to solve a real problem: how do library authors write code that works with all DataFrame backends?

Narwhals provides a lightweight compatibility layer with an API modeled on Polars. If you're writing a library or utility functions that should work with both pandas and Polars DataFrames, Narwhals is worth a look:

import narwhals as nw

@nw.narwhalify
def calculate_summary(df):
    return (
        df.group_by("category")
        .agg(
            nw.col("amount").sum().alias("total"),
            nw.col("amount").mean().alias("average"),
            nw.len().alias("count")
        )
        .sort("total", descending=True)
    )

# Works with pandas
import pandas as pd
pandas_df = pd.DataFrame({"category": ["A", "B", "A"], "amount": [10, 20, 30]})
result = calculate_summary(pandas_df)  # Returns pandas DataFrame

# Works with Polars
import polars as pl
polars_df = pl.DataFrame({"category": ["A", "B", "A"], "amount": [10, 20, 30]})
result = calculate_summary(polars_df)  # Returns Polars DataFrame

Narwhals is already used by major projects like Plotly, Altair, Bokeh, and Marimo, so it's a proven solution — not just a proof of concept.

Decision Guide: When to Use What

Here's a practical decision framework for choosing the right tool. (Bookmark this one — you'll come back to it.)

Use Pandas When:

  • Your dataset fits comfortably in memory (under ~1 GB or so)
  • You need to integrate with scikit-learn, matplotlib, seaborn, or statsmodels
  • You're doing exploratory data analysis in a Jupyter notebook
  • You need a feature that's only available in the pandas ecosystem
  • Your team is already proficient with pandas and the performance is acceptable

Use Polars When:

  • Performance matters — datasets from hundreds of megabytes to dozens of gigabytes
  • You want automatic parallelization without thinking about it
  • You're building data transformation pipelines that benefit from lazy evaluation
  • You prefer a more consistent, modern DataFrame API
  • Memory efficiency is important (running on constrained hardware, for instance)

Use DuckDB When:

  • You're comfortable with SQL and prefer it for analytical queries
  • You need to query files directly without loading them into memory
  • Your workload involves complex joins, window functions, or heavy aggregations
  • You want to query across multiple file formats (CSV, Parquet, JSON) in a single query
  • You need to work with data that exceeds available RAM — DuckDB can spill to disk

Use the Hybrid Stack When:

  • You have a multi-stage data pipeline with different requirements at each stage
  • You need production-grade performance but also ML ecosystem compatibility
  • Your data is too large for pandas alone but you still need pandas for the final analysis

Migration Tips: Moving from Pure Pandas

If you're currently running a pure pandas stack and want to start incorporating Polars or DuckDB, here are some practical tips for a smooth transition.

Start with the Bottlenecks

Don't rewrite everything at once. Profile your existing code, identify the slowest operations, and replace just those with Polars or DuckDB. The interoperability between all three libraries makes surgical replacements surprisingly easy.

DuckDB as a Drop-In Accelerator

The fastest way to speed up existing pandas code is to use DuckDB to accelerate specific queries while keeping everything else in pandas. It's almost embarrassingly easy:

import pandas as pd
import duckdb

# Your existing pandas workflow
df = pd.read_csv("data.csv")
df["date"] = pd.to_datetime(df["date"])

# Replace the slow groupby with DuckDB (just this one operation)
monthly_summary = duckdb.sql("""
    SELECT
        DATE_TRUNC('month', date) AS month,
        category,
        SUM(revenue) AS total_revenue,
        COUNT(DISTINCT customer_id) AS customers
    FROM df
    GROUP BY DATE_TRUNC('month', date), category
    HAVING SUM(revenue) > 10000
    ORDER BY month, total_revenue DESC
""").df()

# Continue with pandas as before
monthly_summary.to_excel("report.xlsx", index=False)

Polars for New Code

When writing new transformation logic, consider writing it in Polars from the start. The API is clean and well-documented, and the performance benefits compound as your data grows.

Test with Real Data Sizes

Performance differences between libraries are negligible on small datasets. Always benchmark with production-scale data to make sure the migration effort is actually justified.

Common Pitfalls and Gotchas

Before you dive in, here are some common mistakes I've seen (and made myself) that are worth avoiding:

Polars Pitfalls

  • Don't use Python loops with Polars. If you find yourself writing for row in df.iter_rows(), you're doing it wrong. Express the logic as Polars expressions instead.
  • Remember that Polars uses null, not NaN. Unlike pandas, Polars has a proper null type that works consistently across all data types. This trips up a lot of people coming from pandas.
  • Column names are case-sensitive and string-based. No positional indexing with integers for columns.
  • It's group_by with an underscore, not groupby like pandas. Muscle memory will fight you on this one.

DuckDB Pitfalls

  • Variable name collisions. DuckDB queries can reference Python variables, but if a variable has the same name as a table or file, things can get confusing. Use explicit aliases.
  • Connection management. The default duckdb.sql() uses a global in-memory connection. For concurrent access or persistent storage, create explicit connections with duckdb.connect().
  • Thread safety. A single DuckDB connection shouldn't be shared across threads without proper synchronization. I've seen this bite people in FastAPI apps especially.

Looking Ahead: What's Next for the Ecosystem

The Python data processing ecosystem continues to evolve at a rapid clip. Here's what to keep an eye on in 2026 and beyond:

  • Polars 2.0 is on the roadmap, featuring a completely redesigned streaming engine that combines morsel-driven parallelism with Rust's async state machines. GPU acceleration is also in active development.
  • DuckDB keeps expanding its extension ecosystem, with community-contributed extensions for spatial data, graph analytics, and more.
  • Narwhals is gaining traction as the standard interoperability layer, which means more libraries will "just work" with whatever DataFrame backend you prefer.
  • scikit-learn 1.8 now supports the Array API standard, so GPU-accelerated arrays from PyTorch and CuPy can be used directly — reducing the need for pandas as an intermediary in ML workflows.

The trend is clear: the future of Python data processing isn't a single library — it's a well-integrated ecosystem of specialized tools. Pandas provides ecosystem breadth, Polars delivers raw DataFrame performance, and DuckDB brings SQL analytics power. Understanding all three puts you in a great position to pick the right tool for each job.

Wrapping Up

The era of "just use pandas for everything" is over — and honestly, that's a good thing. Polars and DuckDB aren't here to replace pandas; they're expanding what's possible in the Python data ecosystem. Polars gives you a modern, type-safe DataFrame API with automatic parallelization and lazy evaluation. DuckDB gives you the full power of analytical SQL without any infrastructure overhead. And pandas remains the connective tissue that ties everything together with its unmatched ecosystem support.

The practical approach? Learn all three, understand their strengths, and build hybrid workflows that leverage each library where it shines. Start by identifying the bottlenecks in your current pandas code, experiment with DuckDB as a drop-in SQL accelerator, explore Polars for new transformation logic, and use Arrow-based interoperability to move data seamlessly between them.

The Python data stack in 2026 is faster, more capable, and more flexible than it's ever been. You just need to know which tools to reach for — and now you do.