Python Boto3 Pagination and Error Handling

Working with paginated AWS responses in boto3 requires understanding how Paginator objects iterate and how botocore surfaces failures when an individual page request fails.

When you call a boto3 client method such as list_objects_v2, describe_instances, or scan, the AWS service returns a single page of results. Many services cap a page at 1,000 items, while others use different limits; for example, S3's list_objects_v2 can return at most 1,000 keys per page, and DynamoDB scan returns up to 1 MB per page. The response includes a continuation token when more data exists, but the client method does not follow it for you. A Paginator wraps the underlying API call, sends repeated requests with the continuation token, and yields each response as a page.

How Paginators Work in boto3

A paginator is created from a client with get_paginator, passing the name of the paginated operation. The resulting object exposes a paginate() method that accepts the same parameters as the underlying client call, plus an optional PaginationConfig dictionary.

import boto3

s3 = boto3.client("s3")
paginator = s3.get_paginator("list_objects_v2")

for page in paginator.paginate(Bucket="my-bucket"):
    for obj in page.get("Contents", []):
        print(obj["Key"])

The paginator keeps the continuation token internal. Each iteration of the outer loop is one HTTP request to S3, and the inner loop processes the objects in that page. If the bucket has 2,500 objects, the loop runs three times: two full pages of 1,000 and one page of 500.

Not every boto3 client method has a corresponding paginator. Paginators exist for operations that the AWS service model marks as paginated. You can check whether a client supports pagination for a given operation by calling client.can_paginate("operation_name"), or by calling get_paginator and catching botocore.exceptions.OperationNotPageableError.

Controlling Page Size and Result Limits with PaginationConfig

The paginate() method accepts a PaginationConfig dictionary with three commonly used keys.

for page in paginator.paginate(
    Bucket="my-bucket",
    PaginationConfig={"PageSize": 200, "MaxItems": 5000},
):
    for obj in page.get("Contents", []):
        print(obj["Key"])

PageSize does not change the service's hard limit; it only asks for a smaller page. A service that caps responses at 1,000 items will still return 1,000 if you request 2,000. MaxItems is useful when you need only a sample of results and want to avoid fetching the entire dataset.

StartingToken is returned in the paginate response under NextToken or a service-specific key. When you need to resume a paginated scan across separate invocations of your code, store that token and pass it back in PaginationConfig.

Filtering Results with search()

Paginator objects expose a search() method that applies a JMESPath expression to each page and yields only the matching elements. This avoids pulling every object into memory just to filter it in Python.

for obj in paginator.paginate(Bucket="my-bucket").search(
    "Contents[?Size > `1048576`]"
):
    print(obj["Key"], obj["Size"])

The expression selects objects from the Contents array where Size is greater than 1,048,576 bytes. The paginator still makes the same number of API calls, but the iterator yields only matching items. If the expression matches nothing, the loop simply does not run.

JMESPath is the same query language used by the AWS CLI's --query option. Keep expressions simple; complex projections can be harder to debug than a plain Python filter, especially when the response structure varies between pages.

Error Handling During Pagination

A paginated request can fail on any page. The paginator raises the same exceptions a direct client call would raise, most commonly botocore.exceptions.ClientError. When an exception is raised, the iterator stops; pages already yielded remain processed, but no further pages are fetched.

from botocore.exceptions import ClientError

try:
    for page in paginator.paginate(Bucket="my-bucket"):
        for obj in page.get("Contents", []):
            process(obj)
except ClientError as e:
    if e.response["Error"]["Code"] == "NoSuchBucket":
        print("Bucket does not exist")
    else:
        raise

ClientError carries the full service response in e.response, including Error, ResponseMetadata, and the HTTP status code. Match on e.response["Error"]["Code"] rather than parsing the message text; error codes are stable across services, while messages change.

A common mistake is to wrap only the first page in a try block. Because the paginator performs requests lazily, an error on the third page surfaces inside the loop, not at the paginate() call. The try block must wrap the entire iteration.

Retry Configuration and Throttling

AWS services throttle requests that exceed account-level or per-resource limits. Throttled calls return errors such as ThrottlingException, ProvisionedThroughputExceededException, or RequestLimitExceeded. botocore retries eligible requests when you configure retries on the client.

from botocore.config import Config

config = Config(retries={"max_attempts": 5, "mode": "standard"})
s3 = boto3.client("s3", config=config)

max_attempts includes the initial request, so a value of 5 means one original call plus up to four retries. The standard mode applies the SDK's standardized retry rules, which cover throttling and transient transport errors. The default retry mode has changed across botocore versions, so set mode explicitly if your code depends on it.

If a throttled service response occurs on every attempt and the retry limit is reached, the final ClientError for that response is the exception your code sees. Transport-level failures, such as connection or read timeouts, may surface as RetryError or another botocore exception. Catch the specific exceptions you expect and inspect e.response["Error"]["Code"] when available.

from botocore.exceptions import ClientError

try:
    for page in paginator.paginate(Bucket="my-bucket"):
        process_page(page)
except ClientError as e:
    if e.response["Error"]["Code"] == "NoSuchBucket":
        print("Bucket does not exist")
    else:
        raise

Retry behavior applies per request. A paginator making 50 requests can retry each one independently, so a slow or throttled service can extend the total runtime significantly. For long-running jobs, consider reducing PageSize to keep individual requests small, and set a reasonable max_attempts rather than relying on the default.

Memory and Runtime Considerations for Large Result Sets

The paginator yields pages as they arrive, but the way you consume them determines memory usage. Accumulating every object into a single list defeats the purpose of pagination:

all_keys = []
for page in paginator.paginate(Bucket="my-bucket"):
    all_keys.extend(obj["Key"] for obj in page.get("Contents", []))

This is fine for a few thousand objects but becomes wasteful for millions. Process each page as it arrives, or write results to a file or database incrementally. The search() method helps when you need only a subset, because it filters before your code sees the full page.

Each paginated request also has a fixed overhead: connection setup, authentication, and response parsing. Reducing PageSize increases the number of requests and the total latency. Increasing it reduces request count but can produce larger responses that take longer to parse. The right value depends on the service and the size of individual items; start with the service default and adjust only when you observe slow responses or throttling.

A Complete Example: Paginated S3 Listing with Error Handling

The following example combines pagination, retry configuration, and error handling into a single function that lists object keys from an S3 bucket and stops cleanly when the bucket does not exist or when a service error occurs.

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

def list_object_keys(bucket: str, prefix: str = "") -> list[str]:
    config = Config(retries={"max_attempts": 4, "mode": "standard"})
    s3 = boto3.client("s3", config=config)
    paginator = s3.get_paginator("list_objects_v2")

    keys = []
    try:
        for page in paginator.paginate(
            Bucket=bucket,
            Prefix=prefix,
            PaginationConfig={"PageSize": 500},
        ):
            for obj in page.get("Contents", []):
                keys.append(obj["Key"])
    except ClientError as e:
        if e.response["Error"]["Code"] == "NoSuchBucket":
            raise ValueError(f"Bucket {bucket} does not exist") from e
        raise

    return keys

The function returns a list, which is acceptable when the caller needs random access. For streaming use cases, replace the list with a generator that yields each key as it is read. A PageSize of 500 requests smaller pages; this can reduce per-response parse time, but it also increases the number of requests.

Note that list_objects_v2 returns Contents only when the bucket is not empty. Calling page["Contents"] directly would raise KeyError on an empty bucket, which is why the example uses page.get("Contents", []). This is a common edge case that only appears when a bucket happens to be empty.

The same pattern applies to other paginated AWS operations: describe_instances for EC2, scan for DynamoDB, list_tables, and most other services that expose a paginator. The paginator hides the continuation token, PaginationConfig controls the request shape, and botocore exceptions define how failures surface. Keep the try block around the entire loop, match on error codes rather than messages, and configure retries at the client level so throttled pages are retried before your code ever sees them.