Python PyArrow: Read and Write Parquet Files
A practical guide to reading and writing Parquet files with PyArrow in Python, covering schema control, row group access, partitioning, compression, and memory considerations.

PyArrow's pyarrow.parquet module provides a fast, native implementation of the Parquet format. It exposes both low-level file readers and writers and a higher-level Table API that integrates with pandas and other data tools. This article focuses on the core operations: reading a Parquet file into a PyArrow table, writing a table to Parquet, and the options that matter when you move beyond trivial examples.
Reading a Parquet File with PyArrow
The simplest way to read a Parquet file is pyarrow.parquet.read_table. It returns a pyarrow.Table object that holds the data in a columnar layout. Here is the minimal usage:
import pyarrow.parquet as pq
table = pq.read_table("data.parquet")
print(table.schema)
print(table.num_rows)The read_table function infers the schema from the file metadata. You can also read only a subset of columns to reduce memory usage and I/O:
table = pq.read_table("data.parquet", columns=["id", "name", "score"])If you need a pandas DataFrame instead, use read_pandas and convert the resulting table:
df = pq.read_pandas("data.parquet").to_pandas()For very large files that do not fit in memory, you can read row groups individually. A Parquet file is split into row groups, each independently compressed and readable. Use ParquetFile to inspect the structure and read row groups one at a time:
pf = pq.ParquetFile("large.parquet")
for i in range(pf.num_row_groups):
group_table = pf.read_row_group(i)
process(group_table)This pattern is essential when you want to process a file incrementally without loading the entire dataset.
Writing a Parquet File with PyArrow
Writing a table to Parquet is equally direct. You can create a pyarrow.Table from Python data or from a pandas DataFrame, then call write_table:
import pyarrow as pa
import pyarrow.parquet as pq
data = {
"id": [1, 2, 3],
"name": ["alice", "bob", "carol"],
"score": [9.5, 8.0, 7.5]
}
table = pa.table(data)
pq.write_table(table, "output.parquet")The write_table function handles schema inference automatically. If you have a pandas DataFrame, you can convert it first:
import pandas as pd
df = pd.DataFrame({"id": [1, 2], "value": [10, 20]})
table = pa.Table.from_pandas(df)
pq.write_table(table, "from_pandas.parquet")You can also write multiple tables into a single file using ParquetWriter when you need to stream data in chunks. This is useful when the full table does not fit in memory:
writer = pq.ParquetWriter("stream.parquet", schema=table.schema)
for batch in table.to_batches(max_chunksize=1000):
writer.write_batch(batch)
writer.close()Notice that ParquetWriter requires a schema upfront. You can derive it from the first batch or define it explicitly.
Controlling the Parquet Schema
When reading, the schema comes from the file. When writing, you often want to control the types explicitly to avoid surprises. PyArrow lets you define a schema and pass it to write_table:
schema = pa.schema([
("id", pa.int64()),
("name", pa.string()),
("score", pa.float64()),
])
pq.write_table(table, "typed.parquet", schema=schema)If the table has columns that do not match the schema, PyArrow will attempt to cast them. If casting fails, it raises an error. This is useful when you want to enforce a specific storage format, such as using int32 for small integers to save space.
When reading, you can also override the schema using the schema parameter in read_table. This is rarely needed, but it can help when a file has a schema that is not fully compatible with your downstream processing.
Partitioning and Row Group Size
Parquet supports Hive-style directory partitioning. To write a partitioned dataset with PyArrow, use pq.write_to_dataset:
pq.write_to_dataset(table, "partitioned/", partition_cols=["year", "month"])This writes files under subdirectories like year=2024/month=01/. Partitioning is a common pattern for large datasets because it allows queries to skip entire directories based on partition filters.
Row group size controls the granularity of compression and reading. By default, PyArrow uses a row group size of about 1 million rows. You can adjust it with the row_group_size parameter:
pq.write_table(table, "output.parquet", row_group_size=50000)Smaller row groups allow more selective reads but increase metadata overhead. Larger row groups reduce overhead but require reading more data when you only need a subset. Choose a size that matches your typical access pattern.
Compression and Encoding Options
Parquet supports several compression codecs. PyArrow defaults to snappy, which balances speed and compression ratio. You can change it with the compression parameter:
pq.write_table(table, "compressed.parquet", compression="gzip")Common choices are snappy, gzip, brotli, lz4, and zstd, although availability can vary by PyArrow build. Each has different tradeoffs. snappy is fast but compresses less; gzip compresses more but is slower. If you need to minimize file size, zstd often gives a good balance. The compression_level parameter lets you tune the codec further, but not all codecs support it.
Encoding is another layer of optimization. PyArrow automatically chooses encodings like dictionary encoding for string columns when it reduces size. You can disable dictionary encoding with use_dictionary=False if you know it is not beneficial for your data. This is rarely necessary, but it can help when the cardinality is extremely high.
Performance Considerations When Reading and Writing Parquet
The main performance benefit of Parquet is its columnar layout. When you read only a few columns, PyArrow reads only those columns from disk, skipping the rest. This is why columns filtering is so effective. Writing is often dominated by the cost of compression and encoding; using a faster codec like lz4 can speed up writes at the expense of file size.
Memory usage is another concern. Reading a whole file into a Table loads all row groups into memory. For files larger than available RAM, use the row-group iteration pattern shown earlier. Writing with ParquetWriter in batches also keeps memory bounded.
One subtle point: by default, read_table loads all row groups into memory, while ParquetFile alone only reads metadata and read_row_group loads one row group. If you need to process a file that is several gigabytes, prefer the streaming approach.
Handling Large Files and Memory Usage
When working with datasets that exceed memory, you need to think about how PyArrow allocates memory. PyArrow uses its own memory pool, which can be configured. For most applications, the default is fine. However, when reading many files in a loop, you may want to call pa.gc_memory() to release unused memory back to the operating system:
import pyarrow as pa
for file in files:
table = pq.read_table(file)
process(table)
del table
pa.gc_memory()This is not a substitute for proper memory management, but it helps in long-running processes.
Another practical technique is to use memory mapping. PyArrow can read Parquet files using memory mapping via memory_map=True in read_table. This lets the OS manage page loads instead of copying the file into PyArrow's memory pool:
table = pq.read_table("large.parquet", memory_map=True)Memory mapping is not always faster; it depends on the OS and the access pattern. For random access to a subset of columns, it can be very efficient.
Finally, consider using pq.read_schema to inspect a file without loading any data. This is useful for validating schemas before running a full read:
schema = pq.read_schema("data.parquet")
print(schema)This operation reads only the metadata, so it is fast even for huge files.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.