Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Scrapy Export to CSV and JSON

Export Scrapy scraped data to CSV and JSON with feed exports. Learn field ordering, custom JSON serializers, JSON Lines vs array output, overwrite behavior, and debugging tips.

~5 min readPython
#Scrapy#Web Scraping#CSV Export#JSON Export
Illustration of a Scrapy spider exporting scraped data into CSV and JSON file formats.

When you run a Scrapy spider, the items it yields are only written somewhere if you configure an output. Scrapy's feed export system is the standard way to send scraped items to a file as they are scraped, with built-in support for CSV, JSON, JSON Lines, and other formats. For exporting to CSV and JSON, the two approaches are the -o command line flag for quick one-off runs and the FEEDS setting for repeatable, configurable exports.

Quick Output with the -o Flag

The simplest export is a single command line flag:

bash
scrapy crawl quotes -o quotes.csv

Scrapy infers the format from the file extension. The same works for JSON:

bash
scrapy crawl quotes -o quotes.json

If the extension is ambiguous or you want to force a format, use -t:

bash
scrapy crawl quotes -t csv -o quotes.data

By default, -o appends to an existing file if it already exists, which can cause duplicate rows when you rerun a spider. Use -O instead to overwrite the file, or remove the old file before starting.

Configuring Exports with FEEDS

For repeatable runs, put the export configuration in settings.py. The FEEDS setting maps output locations to format options:

python
FEEDS = { "output/quotes.csv": { "format": "csv", "encoding": "utf-8", "overwrite": True, }, "output/quotes.json": { "format": "json", "encoding": "utf-8", "overwrite": True, }, }

This writes both formats in a single crawl. The overwrite key controls whether an existing file is truncated or appended. Leaving it False (the default) appends, which is useful when you want to accumulate results across multiple runs but dangerous when you expect a fresh file each time.

You can also scope feeds per spider by defining custom_settings inside the spider class:

python
class QuotesSpider(scrapy.Spider): name = "quotes" custom_settings = { "FEEDS": { "quotes.csv": {"format": "csv", "overwrite": True}, } }

JSON Output: Array or JSON Lines

Scrapy's json feed format writes a single JSON array:

json
[ {"author": "Douglas Adams", "text": "The answer is 42"}, {"author": "Terry Pratchett", "text": "I don't know"} ]

If you set the format to jsonlines, the exporter writes one JSON object per line:

json
{"author": "Douglas Adams", "text": "The answer is 42"} {"author": "Terry Pratchett", "text": "I don't know"}

To produce JSON Lines, set format to jsonlines in FEEDS, or use -t jsonlines with -o.

Both formats stream items as they are scraped. The json exporter writes the opening bracket before the first item and the closing bracket after the last item, so it does not need to hold the whole result set in memory. JSON Lines is a better fit when you plan to append to the file or process records one at a time.

If you already have a JSON Lines file and need a JSON array, post-process it:

python
import json with open("quotes.jsonl", "r") as f: items = [json.loads(line) for line in f if line.strip()] with open("quotes_array.json", "w") as f: json.dump(items, f, indent=2)

This script reads the entire file into memory, so for very large files, stream the conversion line by line and write to the output as you go.

Controlling Field Order and Selection in CSV

CSV has no schema, so Scrapy decides which columns to write based on the first item it encounters. If your spider yields items with inconsistent fields, the CSV columns reflect whatever the first item contained, and later items with additional fields will silently drop those columns.

Fix this by declaring the field list in the feed configuration:

python
FEEDS = { "quotes.csv": { "format": "csv", "fields": ["author", "text", "tags"], "overwrite": True, }, }

The fields key fixes both the column order and the set of columns. Any item field not listed is omitted. This makes the CSV stable enough to load into a database or spreadsheet.

When items are Scrapy Item objects with defined fields, the field order in the item class also influences output, but the explicit fields list in FEEDS takes precedence and is the more reliable way to control it.

Serializing Non-String Values

CSV and JSON both need values to be serializable. Scrapy handles the common Python types, but you will hit edge cases with dates, enums, and custom objects.

For JSON, a datetime object raises a TypeError because the standard json module cannot serialize it. Pass a custom encoder to the JSON exporter through item_export_kwargs:

python
import json from datetime import datetime, date class DateTimeEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, (datetime, date)): return obj.isoformat() return super().default(obj) FEEDS = { "quotes.json": { "format": "json", "item_export_kwargs": {"encoder": DateTimeEncoder}, "overwrite": True, }, }

For CSV, every value is converted to a string by the writer, so dates become their str() representation unless you convert them before export. If you need ISO 8601 dates in the CSV, convert the value in the spider or in an item pipeline.

The cleanest place to normalize values is an item pipeline, because it runs before the feed exporter writes the item:

python
class DateNormalizerPipeline: def process_item(self, item, spider): if isinstance(item.get("published"), datetime): item["published"] = item["published"].isoformat() return item

Performance and Memory Behavior

The feed exporter writes items as they are yielded, so the memory footprint stays flat regardless of how many items the spider produces. The main cost is I/O: writing to local disk is fast, but writing to a remote location or a network filesystem can slow the crawl if the exporter cannot keep up with the spider's yield rate.

For large exports, consider these points:

  • The overwrite flag controls truncation, but appending to a large existing file adds new rows at the end without rewriting existing content.
  • If you convert JSON Lines into an array in a post-processing step, stream the conversion line by line instead of collecting every parsed item in a list first.
  • CSV with many columns and long text fields produces larger files than the equivalent JSON because every value is quoted and escaped according to CSV rules. This affects disk usage but not crawl speed in most cases.

The feed exporter does not compress files by default. If disk space is a concern, enable Scrapy's GzipPlugin through the postprocessing option, such as "postprocessing": ["scrapy.extensions.postprocessing.GzipPlugin"] in the feed's options, or write the feed to a temporary file and compress it after the crawl.

Failure Modes and Debugging

The most common failure is a TypeError during serialization, which stops the crawl. The traceback points to the item and the field that failed, so check whether the value is a datetime, Decimal, set, or a custom class before blaming the exporter.

Another frequent issue is empty output. If the spider yields no items, the feed exporter may create an empty file or no file at all, depending on the format and config. That is not an error; it means the spider's parsing rules matched nothing. Check the spider's logs for the number of items scraped to distinguish an empty result from a broken export.

When overwrite is False and the output file already exists, Scrapy appends to it. Rerunning a spider without cleaning the file produces duplicate rows. This is the single most common cause of "my CSV has double the data" reports, and it is a configuration issue, not a bug.

For CSV specifically, if the fields list is missing and items vary in shape, the output can have inconsistent columns. Always declare fields for CSV feeds that must be loaded by other tools. When you need both CSV and JSON from the same crawl, configure both in FEEDS so the exporter handles each format independently and you avoid writing custom conversion code between the two.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.