Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python OpenAI Image Input and Multimodal Requests

Use the OpenAI Python SDK to send images to vision-capable models: URL images, base64 local files, multiple images, response handling, and cost considerations.

~5 min readPython
#OpenAI API#Multimodal#Python SDK#Image Input
Illustration of a Python script sending an image and text to an OpenAI model, with a visual representation of multimodal input.

When you need a model to reason about an image, you send a multimodal request: a chat completion that includes both text and image content. In Python, the OpenAI SDK makes this straightforward, but the message structure differs from plain text requests. This article shows how to construct image input and multimodal requests with the official openai package, including URL images, local files, multiple images, and response handling.

Setting Up the OpenAI Python Client

Install the OpenAI SDK if you haven't already:

bash
pip install openai

Create a client with your API key. The SDK reads the OPENAI_API_KEY environment variable by default, so you can avoid hardcoding secrets:

python
from openai import OpenAI client = OpenAI() # reads OPENAI_API_KEY from environment

If you need to pass the key explicitly, use OpenAI(api_key="sk-..."), but prefer environment variables in production.

Constructing a Multimodal Message with an Image

In a chat completion, the messages list contains objects with role and content. For a user message that includes an image, content becomes a list of parts, each with a type. The two relevant types are text and image_url. The image_url part accepts a URL or a data URI.

Here is a minimal request using a publicly accessible image URL:

python
response = client.chat.completions.create( model="gpt-4o", messages=[ { "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, { "type": "image_url", "image_url": { "url": "https://example.com/cat.jpg" } } ] } ] )

The image_url object can also include a detail parameter (for example, "low", "high", or "auto") to control how the model processes the image. Lower detail reduces token cost and latency; high detail gives the model more visual information. When omitted, the default is auto, which chooses low or high detail based on the image size.

Sending the Request and Parsing the Response

The response object follows the standard chat completion structure. The model's text answer is in response.choices[0].message.content:

python
answer = response.choices[0].message.content print(answer)

Because the request is multimodal, the model may also return tool calls or other structured fields, but for typical image understanding tasks you only need the content string.

If you want to stream the response, pass stream=True and iterate over chunks, just as you would for text-only requests. The image is sent in the request; streaming only affects how the text response is delivered.

Working with Local Images and Base64 Encoding

To send an image stored on disk, you must encode it as base64 and embed it in a data URI. The OpenAI API accepts data URIs in the image_url.url field.

python
import base64 def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode("utf-8") base64_image = encode_image("local_image.jpg") data_uri = f"data:image/jpeg;base64,{base64_image}" response = client.chat.completions.create( model="gpt-4o", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, { "type": "image_url", "image_url": {"url": data_uri} } ] } ] )

Make sure the MIME type in the data URI matches the actual file format (for example, image/png or image/webp). If you send an incorrect MIME type, the API may reject the request or misinterpret the image.

Handling Multiple Images and Mixed Content

A single user message can contain multiple image parts and multiple text parts. This is useful for comparing images or providing context alongside several visuals.

python
content_parts = [ {"type": "text", "text": "Compare these two images."}, {"type": "image_url", "image_url": {"url": "https://example.com/first.png"}}, {"type": "image_url", "image_url": {"url": "https://example.com/second.png"}} ] response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": content_parts}] )

You can interleave text and image parts in any order. The order is preserved in the request, so keep the instruction close to the image it refers to, and test if the output seems unclear.

Model Compatibility and Choosing the Right Model

Not all OpenAI models accept image input. For example, the GPT-4o family and GPT-4 Turbo currently accept image input, while GPT-3.5 does not. Model availability and vision support can change, so verify the specific model in the official documentation.

For most applications, gpt-4o provides a good balance of reasoning ability, speed, and cost. gpt-4o-mini is cheaper and faster but may produce less accurate descriptions for complex images. If you need detailed visual analysis, prefer gpt-4o with detail: "high".

When you use a model that does not support images, the API returns an error. The error message typically states that the model does not support image input, so you can catch it and fall back to a text-only model if appropriate.

Error Handling and Common Failure Modes

Multimodal requests fail for several reasons. The most common are invalid image URLs, unsupported file formats, oversized images, and authentication issues. The SDK raises openai.APIError subclasses, so you can handle them uniformly:

python
from openai import APIError, AuthenticationError, BadRequestError try: response = client.chat.completions.create( model="gpt-4o", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}}, ], } ], ) except BadRequestError as e: print(f"Invalid request: {e}") except AuthenticationError as e: print(f"Check your API key: {e}") except APIError as e: print(f"API error: {e}")

A common mistake is sending a base64 string without the data:image/...;base64, prefix. The API interprets the URL literally and tries to fetch it, which fails. Always include the proper data URI.

Image size also matters. The API has a maximum size limit (currently around 20 MB per image), and very large images may be rejected or require downscaling. For production, consider resizing images before encoding to reduce payload size and token cost.

Performance, Cost, and Token Considerations

Image inputs are converted into tokens for processing and billing. The token cost depends on the image's dimensions and the detail setting. A low detail image costs fewer tokens than a high detail image. This directly affects both latency and cost per request.

For example, a low-detail image might be represented by a fixed number of tokens, while a high-detail image is split into tiles, each costing additional tokens. The exact numbers are documented by OpenAI and can change, so check the pricing page before building a large-scale application.

If you only need a simple classification or a short caption, use detail: "low" to reduce cost. If the task requires reading small text or identifying fine details, use detail: "high". The auto setting picks low or high detail based on the image size, which is a reasonable default.

Latency also tends to increase with image complexity. A high-detail request can take longer to process because the model must analyze more visual information. In user-facing applications, you can show a loading state or use streaming to improve perceived performance.

Finally, consider caching image analysis results if you repeatedly send the same image. This avoids redundant API calls and reduces cost. For dynamic images, there is no benefit, but for static assets, a simple cache keyed by image hash can save significant resources.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.