Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python PyArrow: Read and Write Parquet Files

A practical guide to reading and writing Parquet files with PyArrow in Python, covering schema control, row group access, partitioning, compression, and memory considerations.

~5 min readPython
#pyarrow#parquet#data engineering#columnar storage
A visual representation of reading and writing Parquet files with PyArrow in Python, showing a columnar table and file I/O arrows.

PyArrow's pyarrow.parquet module provides a fast, native implementation of the Parquet format. It exposes both low-level file readers and writers and a higher-level Table API that integrates with pandas and other data tools. This article focuses on the core operations: reading a Parquet file into a PyArrow table, writing a table to Parquet, and the options that matter when you move beyond trivial examples.

Reading a Parquet File with PyArrow

The simplest way to read a Parquet file is pyarrow.parquet.read_table. It returns a pyarrow.Table object that holds the data in a columnar layout. Here is the minimal usage:

python
import pyarrow.parquet as pq table = pq.read_table("data.parquet") print(table.schema) print(table.num_rows)

The read_table function infers the schema from the file metadata. You can also read only a subset of columns to reduce memory usage and I/O:

python
table = pq.read_table("data.parquet", columns=["id", "name", "score"])

If you need a pandas DataFrame instead, use read_pandas and convert the resulting table:

python
df = pq.read_pandas("data.parquet").to_pandas()

For very large files that do not fit in memory, you can read row groups individually. A Parquet file is split into row groups, each independently compressed and readable. Use ParquetFile to inspect the structure and read row groups one at a time:

python
pf = pq.ParquetFile("large.parquet") for i in range(pf.num_row_groups): group_table = pf.read_row_group(i) process(group_table)

This pattern is essential when you want to process a file incrementally without loading the entire dataset.

Writing a Parquet File with PyArrow

Writing a table to Parquet is equally direct. You can create a pyarrow.Table from Python data or from a pandas DataFrame, then call write_table:

python
import pyarrow as pa import pyarrow.parquet as pq data = { "id": [1, 2, 3], "name": ["alice", "bob", "carol"], "score": [9.5, 8.0, 7.5] } table = pa.table(data) pq.write_table(table, "output.parquet")

The write_table function handles schema inference automatically. If you have a pandas DataFrame, you can convert it first:

python
import pandas as pd df = pd.DataFrame({"id": [1, 2], "value": [10, 20]}) table = pa.Table.from_pandas(df) pq.write_table(table, "from_pandas.parquet")

You can also write multiple tables into a single file using ParquetWriter when you need to stream data in chunks. This is useful when the full table does not fit in memory:

python
writer = pq.ParquetWriter("stream.parquet", schema=table.schema) for batch in table.to_batches(max_chunksize=1000): writer.write_batch(batch) writer.close()

Notice that ParquetWriter requires a schema upfront. You can derive it from the first batch or define it explicitly.

Controlling the Parquet Schema

When reading, the schema comes from the file. When writing, you often want to control the types explicitly to avoid surprises. PyArrow lets you define a schema and pass it to write_table:

python
schema = pa.schema([ ("id", pa.int64()), ("name", pa.string()), ("score", pa.float64()), ]) pq.write_table(table, "typed.parquet", schema=schema)

If the table has columns that do not match the schema, PyArrow will attempt to cast them. If casting fails, it raises an error. This is useful when you want to enforce a specific storage format, such as using int32 for small integers to save space.

When reading, you can also override the schema using the schema parameter in read_table. This is rarely needed, but it can help when a file has a schema that is not fully compatible with your downstream processing.

Partitioning and Row Group Size

Parquet supports Hive-style directory partitioning. To write a partitioned dataset with PyArrow, use pq.write_to_dataset:

python
pq.write_to_dataset(table, "partitioned/", partition_cols=["year", "month"])

This writes files under subdirectories like year=2024/month=01/. Partitioning is a common pattern for large datasets because it allows queries to skip entire directories based on partition filters.

Row group size controls the granularity of compression and reading. By default, PyArrow uses a row group size of about 1 million rows. You can adjust it with the row_group_size parameter:

python
pq.write_table(table, "output.parquet", row_group_size=50000)

Smaller row groups allow more selective reads but increase metadata overhead. Larger row groups reduce overhead but require reading more data when you only need a subset. Choose a size that matches your typical access pattern.

Compression and Encoding Options

Parquet supports several compression codecs. PyArrow defaults to snappy, which balances speed and compression ratio. You can change it with the compression parameter:

python
pq.write_table(table, "compressed.parquet", compression="gzip")

Common choices are snappy, gzip, brotli, lz4, and zstd, although availability can vary by PyArrow build. Each has different tradeoffs. snappy is fast but compresses less; gzip compresses more but is slower. If you need to minimize file size, zstd often gives a good balance. The compression_level parameter lets you tune the codec further, but not all codecs support it.

Encoding is another layer of optimization. PyArrow automatically chooses encodings like dictionary encoding for string columns when it reduces size. You can disable dictionary encoding with use_dictionary=False if you know it is not beneficial for your data. This is rarely necessary, but it can help when the cardinality is extremely high.

Performance Considerations When Reading and Writing Parquet

The main performance benefit of Parquet is its columnar layout. When you read only a few columns, PyArrow reads only those columns from disk, skipping the rest. This is why columns filtering is so effective. Writing is often dominated by the cost of compression and encoding; using a faster codec like lz4 can speed up writes at the expense of file size.

Memory usage is another concern. Reading a whole file into a Table loads all row groups into memory. For files larger than available RAM, use the row-group iteration pattern shown earlier. Writing with ParquetWriter in batches also keeps memory bounded.

One subtle point: by default, read_table loads all row groups into memory, while ParquetFile alone only reads metadata and read_row_group loads one row group. If you need to process a file that is several gigabytes, prefer the streaming approach.

Handling Large Files and Memory Usage

When working with datasets that exceed memory, you need to think about how PyArrow allocates memory. PyArrow uses its own memory pool, which can be configured. For most applications, the default is fine. However, when reading many files in a loop, you may want to call pa.gc_memory() to release unused memory back to the operating system:

python
import pyarrow as pa for file in files: table = pq.read_table(file) process(table) del table pa.gc_memory()

This is not a substitute for proper memory management, but it helps in long-running processes.

Another practical technique is to use memory mapping. PyArrow can read Parquet files using memory mapping via memory_map=True in read_table. This lets the OS manage page loads instead of copying the file into PyArrow's memory pool:

python
table = pq.read_table("large.parquet", memory_map=True)

Memory mapping is not always faster; it depends on the OS and the access pattern. For random access to a subset of columns, it can be very efficient.

Finally, consider using pq.read_schema to inspect a file without loading any data. This is useful for validating schemas before running a full read:

python
schema = pq.read_schema("data.parquet") print(schema)

This operation reads only the metadata, so it is fast even for huge files.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.