Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Boto3 S3: Upload, Download, List, and Delete Files

Learn how to upload, download, list, and delete objects in AWS S3 using Python boto3, with code examples, error handling, and performance and security considerations.

~4 min readPython
#boto3#AWS S3#file operations#S3 upload
Illustration of Python boto3 managing files in AWS S3 bucket with upload, download, list, and delete operations.

Working with files in AWS S3 from Python usually means using boto3. The core operations—upload, download, list, and delete—are straightforward once you know the available methods and their parameters. This article shows how to perform those operations with boto3, including code examples, error handling, and production considerations.

Prerequisites and Setup

Before any S3 operation, install boto3 and configure AWS credentials:

bash
pip install boto3

Credentials can come from environment variables, the shared AWS configuration files, or an IAM role. For local development, put access keys in ~/.aws/credentials and set the region in ~/.aws/config:

ini
# ~/.aws/credentials [default] aws_access_key_id = YOUR_ACCESS_KEY aws_secret_access_key = YOUR_SECRET_KEY
ini
# ~/.aws/config [default] region = us-east-1

Boto3 offers two main interfaces: the low-level client and the high-level resource. The client mirrors the AWS API directly, while the resource provides a more Pythonic object model. Most file operations work well with either interface. The examples below use the client unless noted, because it exposes the full set of parameters and is often more direct for fine-grained control.

python
import boto3 s3_client = boto3.client('s3') s3_resource = boto3.resource('s3')

Uploading Files to S3

Uploading a file from your local filesystem to an S3 bucket is done with upload_file. This method handles the underlying HTTP request and, for larger files, automatically uses multipart upload.

python
s3_client.upload_file( Filename='local/path/to/file.txt', Bucket='my-bucket', Key='remote/path/file.txt' )

The Filename is the local path, Bucket is the bucket name, and Key is the destination object key. The key can include slashes to simulate folders, but S3 storage is flat—folders are just key prefixes.

If you already have a file-like object in memory, use upload_fileobj. This is useful when the data comes from a network stream or a BytesIO buffer.

python
import io buffer = io.BytesIO(b'data to upload') s3_client.upload_fileobj(buffer, 'my-bucket', 'data.txt')

For small payloads that are already in memory, put_object is simpler, but it does not support multipart upload and has a 5 GB limit.

python
s3_client.put_object(Bucket='my-bucket', Key='hello.txt', Body=b'Hello, S3!')

When uploading, you can set metadata, access control, and storage class through ExtraArgs. On buckets with ACLs enabled, you can make an object publicly readable with:

python
s3_client.upload_file( Filename='file.txt', Bucket='my-bucket', Key='file.txt', ExtraArgs={'ACL': 'public-read'} )

Downloading Files from S3

The counterpart to upload_file is download_file, which saves an object directly to a local file.

python
s3_client.download_file( Bucket='my-bucket', Key='remote/path/file.txt', Filename='local/path/file.txt' )

If you need the object content in memory, use get_object and read the Body stream.

python
response = s3_client.get_object(Bucket='my-bucket', Key='file.txt') content = response['Body'].read().decode('utf-8')

For streaming downloads, download_fileobj writes to a file-like object.

python
with open('output.bin', 'wb') as f: s3_client.download_fileobj(Bucket='my-bucket', Key='data.bin', Fileobj=f)

When downloading, you can use VersionId to retrieve a specific version if versioning is enabled on the bucket.

Listing Objects in an S3 Bucket

Listing objects requires pagination because S3 returns at most 1,000 keys per request. The list_objects_v2 method is the current API and supports a ContinuationToken for pagination.

python
def list_all_objects(bucket_name, prefix=''): paginator = s3_client.get_paginator('list_objects_v2') for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix): for obj in page.get('Contents', []): yield obj['Key']

The Prefix parameter filters results to keys that start with a given string. This is how you simulate listing a "folder"—you use the folder path as the prefix.

If you need only object names, you can iterate directly over the Contents list. Each object also includes metadata like Size, LastModified, and ETag.

Deleting Files from S3

To delete a single object, use delete_object.

python
s3_client.delete_object(Bucket='my-bucket', Key='file.txt')

For multiple objects, delete_objects accepts a list of keys and performs the deletion in a single request. This is more efficient than looping over delete_object calls.

python
objects_to_delete = [{'Key': 'file1.txt'}, {'Key': 'file2.txt'}] response = s3_client.delete_objects( Bucket='my-bucket', Delete={'Objects': objects_to_delete} )

The response includes any errors that occurred, so you should check Errors in the response for partial failures.

Deleting an object that does not exist does not raise an error; S3 returns a 204 No Content. This is important for idempotent cleanup scripts.

Error Handling and Retries

S3 operations can fail for many reasons: network issues, invalid credentials, missing buckets, or permission errors. Boto3 raises ClientError for most service-side failures. The exception's response includes an error code and message that you can inspect.

python
from botocore.exceptions import ClientError try: s3_client.get_object(Bucket='my-bucket', Key='nonexistent.txt') except ClientError as e: error_code = e.response['Error']['Code'] if error_code == 'NoSuchKey': print('Object not found') elif error_code == 'AccessDenied': print('Permission denied') else: print(f'Unexpected error: {error_code}')

Boto3 retries certain transient errors, such as throttling and network timeouts, using the SDK's configured retry behavior. You can adjust retry settings via the Config parameter when creating the client.

python
from botocore.config import Config config = Config(retries={'max_attempts': 5, 'mode': 'standard'}) s3_client = boto3.client('s3', config=config)

Performance and Security Considerations

For files above the configured threshold, upload_file uses multipart upload, which parallelizes the transfer. The default threshold is 8 MB, and you can tune it with multipart_threshold and max_concurrency in TransferConfig.

python
from boto3.s3.transfer import TransferConfig config = TransferConfig(multipart_threshold=16 * 1024 * 1024, max_concurrency=10) s3_client.upload_file( Filename='large.bin', Bucket='my-bucket', Key='large.bin', Config=config )

On the security side, always use IAM roles or least-privilege policies. For example, a policy that allows only s3:GetObject on a specific bucket prevents accidental deletion. For sensitive data, enable server-side encryption on the bucket or set the ServerSideEncryption parameter when uploading.

python
s3_client.put_object( Bucket='my-bucket', Key='secret.txt', Body=b'data', ServerSideEncryption='AES256' )

Choosing Between Client and Resource for File Operations

The resource interface can be more readable for simple tasks. For example, uploading with the resource:

python
bucket = s3_resource.Bucket('my-bucket') bucket.upload_file('local.txt', 'remote.txt')

The client is the lower-level interface and maps directly to the S3 API. It exposes explicit pagination, batch operations, and error details. The resource is convenient for common single-object operations, but it hides some of those details. In a script that combines many operations, using one interface consistently keeps the code predictable.

Both interfaces are valid. For most production scripts, the client is often the more straightforward choice because the API surface is visible. The resource is fine for prototypes or when you prefer object-oriented syntax.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.