Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

How to Scrape Multiple Pages Concurrently with Python Playwright's Async API

Use Python Playwright's async API to scrape multiple pages concurrently. This article covers asyncio.gather, browser context isolation, concurrency limits, retries, and rate limiting.

~6 min readPython
#Playwright#Async#Web Scraping#Concurrency
Illustration of multiple browser pages being fetched concurrently by a Python Playwright async script

Playwright's async API lets you run several page loads concurrently in one Python process. This article shows how to structure a scraper that handles many URLs with asyncio and Playwright, including browser context isolation, error handling, and concurrency limits.

Why Use Playwright's Async API for Scraping Multiple Pages

Playwright's synchronous API blocks the Python thread while waiting for browser events, which makes sequential scraping slow when you have many URLs. The async API, built on asyncio, lets you interleave I/O operations such as page navigation, network requests, and DOM queries. Instead of waiting for one page to finish before starting the next, you can schedule multiple page loads and let the event loop manage them as they complete.

This is useful because the bottleneck is usually network latency and page rendering, not CPU work. By using async and await, you can keep a single browser instance busy across many pages. If your scraping logic is CPU-bound, such as heavy parsing, async alone will not solve that; the last section covers this distinction.

Setting Up the Async Playwright Environment

The async API is available in the playwright package. Install it with pip install playwright, then install the browser binaries with python -m playwright install. In your code, import async_playwright and use it as an async context manager.

python
from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) # scraping logic here await browser.close()

The async_playwright context manager starts and stops the Playwright driver. The browser launch is also async because it starts a browser process and waits for it to be ready. You can use chromium, firefox, or webkit; Chromium is the most common choice for scraping.

Scraping Multiple Pages Sequentially with Async

Before adding concurrency, start with a small async function that scrapes one page. It opens a page, navigates to a URL, extracts what you need, and closes the page.

python
async def scrape_page(browser, url): page = await browser.new_page() try: await page.goto(url, wait_until='domcontentloaded') title = await page.title() content = await page.content() return {'url': url, 'title': title, 'content': content} finally: await page.close()

The try/finally block ensures the page is closed even if navigation or extraction raises an exception.

If you call this function in a loop, each iteration waits for the previous page to finish before starting the next. That is sequential execution:

python
urls = ['https://example.com', 'https://example.org', 'https://example.net'] results = [] for url in urls: result = await scrape_page(browser, url) results.append(result)

This works, but the total time is roughly the sum of all page loads.

Scraping Multiple Pages Concurrently with asyncio.gather

The real benefit of the async API is running multiple scrape_page tasks concurrently. The standard way to do this is asyncio.gather, which schedules coroutines and waits for all of them to complete.

python
import asyncio async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) urls = ['https://example.com', 'https://example.org', 'https://example.net'] tasks = [scrape_page(browser, url) for url in urls] results = await asyncio.gather(*tasks) await browser.close() print(results)

When you call scrape_page(browser, url), it returns a coroutine object. asyncio.gather schedules these coroutines on the event loop. The event loop interleaves their execution: while one page waits for a network response, another page can start its navigation. In the best case, the total time moves from the sum of all page loads toward the time of the slowest page, assuming the server and your network can handle the parallel requests.

Each coroutine creates its own page with browser.new_page(). That convenience method creates a new browser context automatically, so these pages do not share cookies or localStorage. This is often what you want for scraping. If you need shared session state, create a context explicitly as shown next.

Managing Browser Contexts and Pages for Concurrency

Playwright separates browser state into contexts. A browser instance can have multiple contexts, and each context can have multiple pages. If you want pages in the same context to share cookies and localStorage, create a context and then make pages from it:

python
async def scrape_page_in_context(context, url): page = await context.new_page() try: await page.goto(url, wait_until='domcontentloaded') title = await page.title() content = await page.content() return {'url': url, 'title': title, 'content': content} finally: await page.close()

In main, create one context, schedule all tasks, and close the context afterward:

python
context = await browser.new_context() try: tasks = [scrape_page_in_context(context, url) for url in urls] results = await asyncio.gather(*tasks) finally: await context.close()

If you need isolation per URL, use separate contexts. browser.new_page() is one way to get that without manually tracking each context: when the page closes, its context closes too.

Creating a new context for each URL is more expensive than creating pages inside one context. If you do not need isolation, the shared-context approach has less overhead.

You should also limit how many pages are open at once. Opening hundreds of pages at the same time can overwhelm the browser and the target server. Use a semaphore to cap concurrency:

python
sem = asyncio.Semaphore(5) async def bounded_scrape(browser, url, sem): async with sem: return await scrape_page(browser, url)

This ensures that at most five tasks are active at any moment.

Handling Errors and Rate Limits in Concurrent Scraping

When you run many requests concurrently, errors become more likely. Network timeouts, HTTP 429 responses, and page crashes can all happen. If an exception escapes from a task, asyncio.gather will propagate it. Catch exceptions inside each task so one failure does not prevent you from collecting results from the other URLs.

python
async def safe_scrape(browser, url): try: return await scrape_page(browser, url) except Exception as exc: return {'url': url, 'error': str(exc)}

To retry failed pages, wrap the scraping logic in a loop with a delay.

python
async def scrape_with_retry(browser, url, retries=3): for attempt in range(retries): try: return await scrape_page(browser, url) except Exception: if attempt == retries - 1: raise await asyncio.sleep(2 ** attempt)

This applies an exponential backoff: 1 second, then 2, then 4.

Respecting rate limits is important. Even with a semaphore, you may send requests faster than a site allows. A common approach is to add a small random delay before navigation.

python
async def rate_limited_scrape(browser, url): await asyncio.sleep(random.uniform(0.5, 1.5)) return await scrape_page(browser, url)

The semaphore controls concurrency, while the delay spreads requests over time.

Performance Considerations and Resource Management

The main advantage of async scraping is reduced wall-clock time, but you need to balance concurrency with memory and CPU use. Each page consumes memory for the rendering engine. If you open too many pages at once, the browser process can become slow or crash.

Closing pages after each task helps free memory. The finally block in scrape_page does that. If you want to reduce setup overhead, you can reuse a small pool of Page objects instead of creating a new page for every URL. A Page should not be used for two concurrent navigations, so a pool works best when you assign one URL at a time. For most scrapers, creating a page per URL and closing it is simpler and avoids stale-state bugs.

The wait_until parameter in goto also matters. domcontentloaded is faster than load because it waits only for the HTML to be parsed, not for images and stylesheets. For scraping, you often only need the DOM, so domcontentloaded is a good default.

Finally, be aware of the target server's load. Even with a semaphore, you can still send many requests in a short time. Use a conservative concurrency limit and add delays to avoid being blocked. The exact limit depends on the site's tolerance and your network, so start low and adjust based on observed behavior.

A Complete Example: Scraping a List of URLs

Here is a full script that combines a semaphore, a per-URL delay, and error handling.

python
import asyncio import random from playwright.async_api import async_playwright async def scrape_page(browser, url): page = await browser.new_page() try: await page.goto(url, wait_until='domcontentloaded') title = await page.title() content = await page.content() return {'url': url, 'title': title, 'content': content} finally: await page.close() async def bounded_scrape(browser, url, sem): async with sem: await asyncio.sleep(random.uniform(0.5, 1.5)) try: return await scrape_page(browser, url) except Exception as exc: return {'url': url, 'error': str(exc)} async def main(): urls = [ 'https://example.com', 'https://example.org', 'https://example.net', 'https://example.edu', 'https://example.co', ] sem = asyncio.Semaphore(3) async with async_playwright() as p: browser = await p.chromium.launch(headless=True) tasks = [bounded_scrape(browser, url, sem) for url in urls] results = await asyncio.gather(*tasks) await browser.close() for result in results: print(result) if __name__ == '__main__': asyncio.run(main())

This script limits concurrency to three pages at a time, adds a random delay before each navigation, and catches exceptions per URL. The finally block ensures the page is closed even when an error occurs. Adjust the semaphore value and the delay range based on the target site's behavior.

The async API is not a silver bullet for every scraping scenario. If your processing is CPU-bound, you may need to combine it with multiprocessing. For most page-fetching tasks, though, async concurrency with Playwright reduces wall-clock time compared with sequential page loads while keeping the code readable and maintainable.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.