Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Converting Between pandas and Polars in Python: Syntax and Pitfalls

Learn how to convert DataFrames between Polars and pandas in Python, including dtype mapping, index behavior, null handling, and performance tradeoffs.

~5 min readPython
#Polars#pandas#Data Conversion#DataFrames
Illustration of two data table icons with a bidirectional arrow between them, representing data conversion between Polars and pandas.

Converting a DataFrame between Polars and pandas is straightforward in principle, but the details matter. Index handling, null representation, and dtype inference differ between the two libraries, so a naive conversion can silently change your data. This article covers the core conversion calls and the pitfalls to watch for.

Why Convert Between Polars and pandas?

Polars and pandas serve different purposes in a Python data stack. pandas is mature, widely used, and integrates with many libraries. Polars uses a different execution model and often delivers better performance on large datasets, with a more explicit API. You may need to move data between the two in either direction: a team may standardize on Polars for new pipelines but still pass data to legacy pandas-based code, or you may want to use Polars for heavy transformations and then convert to pandas for libraries that expect pandas objects.

The conversion calls are simple, but the surrounding details matter. The way each library handles indexes, null values, and data types can cause subtle bugs if ignored. This article covers the conversion methods, what they do with your data, and where they can surprise you.

Converting a pandas DataFrame to Polars

The main API for this direction is pl.from_pandas. It accepts a pandas.DataFrame and returns a polars.DataFrame, copying the data into Polars' Arrow-based representation. By default the pandas index is not preserved. If the index contains information you need, move it into a regular column first with reset_index():

python
import polars as pl import pandas as pd pdf = pd.DataFrame({"id": [1, 2, 3], "name": ["a", "b", "c"]}) pdf = pdf.set_index("id") # The index is lost unless you reset it before conversion. pldf = pl.from_pandas(pdf.reset_index()) print(pldf)

The resulting Polars DataFrame has columns id and name, both regular columns. Without reset_index(), the Polars DataFrame would contain only the name column because Polars has no pandas-style index.

You can also call pl.DataFrame(pdf), which behaves like pl.from_pandas in most current versions. pl.from_pandas is the explicit conversion function and is the one to prefer in new code.

pl.from_pandas infers Polars dtypes from the pandas dtypes. For ordinary columns the mapping is predictable: pandas int64 becomes Polars Int64, float64 becomes Float64, and object columns become String when they contain only strings, or Object when the contents are mixed. Check the result when a column contains missing values or a mix of types.

Converting a Polars DataFrame to pandas

To convert a Polars DataFrame to pandas, call to_pandas on the Polars DataFrame. The method returns a pandas DataFrame with a default RangeIndex unless you specify otherwise.

python
pldf = pl.DataFrame({"x": [10, 20], "y": [1.5, 2.5]}) pdf = pldf.to_pandas() print(pdf)

to_pandas converts Polars data through Arrow into pandas. For ordinary non-null columns, Polars Int64 becomes pandas int64, Float64 becomes float64, and String becomes object with Python str values. Nulls create a special case: a Polars integer column containing nulls cannot be stored as plain int64 in classic pandas dtypes, so it is usually returned as float64 with NaN.

Recent Polars versions provide a use_pyarrow_extension=True option on to_pandas that returns Arrow-backed pandas dtypes, which can preserve nullable integer columns. Check your installed Polars and pandas versions before relying on that parameter.

The conversion copies data from Arrow format to pandas' internal block manager. For large DataFrames, this copy can be expensive in both time and memory. If a downstream function only accepts pandas objects, do the conversion as late as possible in the pipeline.

Handling Data Types and Nulls

The most common source of conversion bugs is the difference in missing-value representation. pandas uses NaN for missing float and object values and NaT for missing datetimes; integer columns cannot contain NaN and are often upcast to float64 when a value is missing. Polars uses null for all missing values and supports nullable integer types natively.

When converting from pandas to Polars, pandas NaN values are read as Polars null. For a float column this is direct. For an object column, the result depends on whether the remaining values share one type: a column of strings with missing values becomes String with null, while a genuinely mixed object column may become Object. If you need a specific dtype, cast the Polars column after conversion.

When converting from Polars to pandas, null values become NaN in classic pandas dtypes. A Polars integer column with nulls therefore cannot stay as plain int64; it is usually returned as float64. To keep a nullable integer representation, use to_pandas(..., use_pyarrow_extension=True) in a recent Polars/pandas combination that supports Arrow-backed pandas dtypes.

python
pldf = pl.DataFrame({"a": [1, None, 3]}) pdf = pldf.to_pandas() # a becomes float64 with NaN in classic pandas pdf_arrow = pldf.to_pandas(use_pyarrow_extension=True) # Arrow-backed dtype

The same issue appears in the other direction. A pandas column that looks like an integer column but contains missing values is actually float64; pl.from_pandas preserves that as Polars Float64. Cast to a nullable Polars integer when you need the integer semantics back:

python
pdf = pd.DataFrame({"a": [1, 2, None]}) # a is float64 pldf = pl.from_pandas(pdf).with_columns(pl.col("a").cast(pl.Int64))

Index and Row Labels

Pandas DataFrames can have a meaningful index: labels for alignment, time-series timestamps, or duplicate row labels. Polars has no separate index; row numbers are implicit. pl.from_pandas drops the index by default. to_pandas returns a default RangeIndex; the Polars row order becomes the pandas row order.

If you need to round-trip index values, reset the index before converting to Polars, then restore it after converting back:

python
pldf = pl.from_pandas(pdf.reset_index()) pdf_roundtrip = pldf.to_pandas().set_index("index")

If the original index had a name, reset_index() creates a column with that name; use that name in the final set_index() call.

Performance and Memory Considerations

Conversion is not free: both pl.from_pandas and to_pandas allocate a new DataFrame in the target library's memory layout. For large data, the copy can dominate runtime and temporarily double memory use. Keep data in one library for as much of the pipeline as possible. Perform heavy aggregations in Polars, for example, and convert only the final result to pandas for output.

If you are moving many DataFrames, Arrow can be an efficient interchange format. Polars reads Arrow with pl.from_arrow, and pandas can construct a DataFrame from a pyarrow.Table. Because Arrow is Polars' native interchange format, this path can avoid some per-cell Python overhead, though it still involves a copy when entering pandas.

Common Pitfalls and How to Avoid Them

  • Losing the index. Check whether the index carries information before converting. Reset it to a column when needed.
  • Unexpected object dtypes. pandas object columns can contain strings, numbers, or mixed values. After conversion, verify what Polars inferred, and cast object-ish columns to the dtype you actually need.
  • Assuming missing values always convert the same way. pandas NaN and None are different values in some contexts. On the usual conversion path, both become null in columns with an otherwise consistent type. For messy object columns, normalize missing values before conversion or check the resulting Polars dtype.
  • Assuming use_pyarrow_extension is always available. The parameter was added to to_pandas in recent Polars versions. Check your Polars and pandas versions before relying on it; the default path uses classic pandas dtypes and may upcast nullable integers to float64.

By being explicit about index handling, dtype mapping, and missing values, you can move data between Polars and pandas without losing information or introducing subtle bugs. Decide up front which DataFrame's semantics you need, and verify the resulting dtypes after conversion rather than assuming they were preserved.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.