Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python OpenAI File Upload: How to Upload Files with the API

Learn how to upload files to the OpenAI API with Python using files.create, including parameters, error handling, and use in fine-tuning and assistants.

~5 min readPython
#OpenAI API#File Upload#Python#Fine-tuning
Illustration of a Python script uploading a file to the OpenAI API, showing a file icon moving to a cloud with OpenAI branding.

Uploading files to the OpenAI API is a common task when you need to provide training data for fine-tuning, attach documents to an assistant, or use the Files API for other workflows. In Python, the openai package exposes client.files.create, which handles multipart uploads. This article covers the practical details: setup, parameters, error handling, and how to use the returned file ID in later API calls.

Prerequisites and Setup

Before you can upload a file, install the openai Python package and configure an API key. The package is available on PyPI and can be installed with pip:

bash
pip install openai

Use a Python version supported by the SDK; the exact minimum depends on the package release. After installation, create a client instance with your API key. Avoid hardcoding the key in source code; use an environment variable instead:

bash
export OPENAI_API_KEY='your-api-key'

Then in your Python code:

python
import os from openai import OpenAI client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

If the key is missing or invalid, the client raises an OpenAIError. For local development, you can use a .env file with the python-dotenv package, but the environment variable approach is the most portable.

Basic File Upload with files.create

The core method for uploading a file is client.files.create. It expects a file object opened in binary mode and a purpose string that tells the API how the file will be used. Here is a minimal example:

python
from openai import OpenAI client = OpenAI() # reads OPENAI_API_KEY from environment with open('training_data.jsonl', 'rb') as f: response = client.files.create( file=f, purpose='fine-tune' ) print(response.id)

The response is a FileObject with fields such as id, purpose, filename, bytes, and created_at. The id is the value you use later when creating a fine-tuning job or attaching the file to an assistant message.

If you are uploading a file for an assistant, change purpose to 'assistants'. The API accepts a limited set of purposes, and an invalid purpose returns a 400 error.

Understanding File Upload Parameters

files.create accepts several parameters beyond the mandatory file and purpose:

ParameterTypeDescription
filefileA file-like object opened in binary mode. Required.
purposestringThe intended use: 'fine-tune', 'assistants', or 'batch'. Required.
filenamestringOptional. Overrides the filename sent to the API. Useful when the local file name is not descriptive.

The filename parameter is also useful for in-memory objects. For example, you can build an in-memory JSONL object and set a custom name:

python
import io import json record = {'prompt': 'What is AI?', 'completion': 'Artificial Intelligence'} file_obj = io.BytesIO((json.dumps(record) + '\n').encode()) response = client.files.create( file=file_obj, purpose='fine-tune', filename='custom_name.jsonl' )

Note that the API enforces file size limits and supported file types. The exact limits change over time, so check the official OpenAI documentation for current values. The SDK does not validate these content constraints locally; invalid uploads are rejected by the server.

Handling Upload Errors and Retries

Network failures, rate limits, and invalid inputs can cause uploads to fail. The openai library raises exceptions that you can catch and handle explicitly. Common exception classes include:

  • openai.APIError: base class for API errors.
  • openai.APIConnectionError: network issues, such as a dropped connection.
  • openai.RateLimitError: exceeded allowed request rate.
  • openai.AuthenticationError: invalid API key.
  • openai.BadRequestError: invalid file or parameters.

A robust upload function should retry transient errors and surface permanent ones. Here is an example that retries connection errors and rate limits with exponential backoff:

python
import time from openai import OpenAI, APIConnectionError, RateLimitError client = OpenAI() def upload_file(file_path, purpose, max_retries=3): for attempt in range(max_retries): try: with open(file_path, 'rb') as f: return client.files.create(file=f, purpose=purpose) except (APIConnectionError, RateLimitError) as e: if attempt == max_retries - 1: raise wait_time = 2 ** attempt print(f'Upload failed, retrying in {wait_time}s: {e}') time.sleep(wait_time)

For permanent errors such as BadRequestError, fix the request before retrying; retrying the same invalid request will not help. Inspect the exception message to understand what failed—it often includes the field that failed validation.

Using Uploaded Files in Fine-Tuning and Assistants

The file ID returned by files.create is the key to using the file in other API calls. For fine-tuning, pass it to fine_tuning.jobs.create:

python
# response is the FileObject returned by files.create job = client.fine_tuning.jobs.create( training_file=response.id, model='gpt-3.5-turbo' )

For assistants, attach the file to a message using attachments. The exact syntax can vary by SDK/API version, but a common pattern is:

python
message = client.beta.threads.messages.create( thread_id=thread.id, role='user', content='Please analyze this document.', attachments=[{'file_id': response.id, 'tools': [{'type': 'file_search'}]}] )

In both cases, the file must be uploaded with the correct purpose. A file uploaded with purpose='fine-tune' cannot be attached to an assistant message, and vice versa. If you need to use the same data for both, upload it twice with different purposes.

The same pattern applies to Batch API jobs: upload with purpose='batch' and then pass the returned file ID when creating a batch job.

Performance and Operational Considerations

Uploading large files can take time and may use significant memory because the file is sent as a multipart upload. If you regularly hit size limits, preprocess or split files before uploading. The API has a maximum file size, and exceeding it results in a 400 error. Check the current limit in the docs and consider checking the size locally before uploading:

python
import os file_size = os.path.getsize('training_data.jsonl') if file_size > MAX_ALLOWED_SIZE: # check current limit from docs raise ValueError('File too large')

For many uploads, be mindful of rate limits. If you need to upload multiple files, use a semaphore or worker pool to limit concurrency and include retry logic. The SDK also supports automatic retries through a max_retries client parameter, but application-level handling gives you control over backoff.

Security and API Key Management

Never expose your API key in client-side code or commit it to version control. Use environment variables or a secrets manager. Before uploading sensitive data, review OpenAI's data usage and retention policies and your own compliance requirements.

Common Pitfalls and Edge Cases

One frequent mistake is opening the file in text mode instead of binary. The API expects a binary file object; passing a text-mode file can cause encoding errors or corrupted uploads. Always use 'rb'.

Another issue is using an invalid purpose string. The API is strict about allowed values. If you are unsure, check the documentation or the error message returned by the server.

Finally, file IDs can become invalid if you delete them or if API retention rules remove them. Keep a local copy of the file and a record of its ID. If an ID is no longer valid, re-upload the file before using it.

Handling these edge cases helps keep file uploads reliable in production. files.create is straightforward, but the surrounding details—parameter validation, error handling, and operational constraints—determine whether your integration is robust.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.