Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python Sentence Transformers Batch Encoding and GPU

Learn how to batch encode texts with sentence-transformers on a GPU, control batch size, avoid out-of-memory errors, and choose the right output format for embeddings.

~5 min readPython
#sentence-transformers#GPU#batch encoding#embeddings
Illustration of multiple sentence embeddings being generated in parallel on a GPU chip

When you need embeddings for thousands of sentences, calling model.encode() on each sentence individually is wasteful. The sentence-transformers library is designed to encode lists of texts in one call, and on a GPU that call can process many sentences in parallel. This article explains how batch encoding works with sentence-transformers on a GPU, how to control batch size, and what to watch out for when scaling up.

Why Batch Encoding Matters for GPU Inference

A transformer model processes token sequences, normally with padding inside a batch so the inputs form a rectangular tensor. When you call encode() with a single sentence, the GPU executes the forward pass for that one sequence. With a batch, the GPU processes multiple sequences in one forward pass, sharing the model weights and spreading fixed overhead across more inputs. The result is usually higher throughput: more sentences per second, and often lower total latency for a large corpus.

GPU hardware is designed for parallel computation. Once a model is loaded, a GPU can often process many sequences in parallel, so batching is a practical way to use the device efficiently. The exact speedup varies with GPU capacity, model size, and sequence length, but a loop of single-sentence calls generally leaves much of that capacity unused.

Basic Batch Encoding with sentence-transformers

The SentenceTransformer class provides an encode() method that accepts a list of strings. Here is the simplest form:

python
from sentence_transformers import SentenceTransformer model = SentenceTransformer('all-MiniLM-L6-v2') sentences = [ 'The cat sits on the mat.', 'A dog runs through the park.', 'Machine learning models require data.' ] embeddings = model.encode(sentences)

The embeddings variable is a NumPy array of shape (3, 384) for this model. Internally, the library tokenizes all the sentences, pads them to the same length, and runs the transformer on the whole batch. In common releases of the library, the default batch size is 32; you can change it with the batch_size argument.

Controlling Batch Size and Device

The encode() method accepts a batch_size parameter. You can specify the device explicitly, for example device='cuda' to force GPU usage. If PyTorch detects a GPU, the model is usually placed on it automatically, but being explicit can make the behavior clearer:

python
embeddings = model.encode(sentences, batch_size=64, device='cuda')

When you pass a batch size larger than the number of sentences, the library processes all the sentences in one batch. When the batch size is smaller, it splits the list into chunks and processes each chunk sequentially. The show_progress_bar parameter can help you monitor progress for large corpora:

python
embeddings = model.encode(sentences, batch_size=128, show_progress_bar=True)

GPU Memory and Choosing the Right Batch Size

GPU memory is finite. Each sentence in a batch consumes memory for token embeddings, attention masks, and intermediate activations during the forward pass. The exact amount depends on the model size and the maximum sequence length. If the batch size is too large, you will get a CUDA out-of-memory error. The error typically looks like:

text
RuntimeError: CUDA out of memory.

To avoid this, find the largest batch size that fits on your GPU. A practical approach is to start with a small value, say 16, and double it until you hit an error, then back off. The optimal batch size also depends on the sequence length of your sentences. Longer sentences consume more memory because the standard transformer attention matrix grows quadratically with sequence length.

The table below is illustrative, not a benchmark.

Batch sizeMemory usageThroughput
16LowModerate
32ModerateGood
64HighBetter
128Very highDepends on GPU and sequence length

The key is to monitor memory usage and choose the largest batch that does not cause an out-of-memory error.

Normalization and Output Formats

The encode() method has several useful options. By default, it returns a NumPy array of float32 vectors. If you need PyTorch tensors, set convert_to_tensor=True. If you need normalized vectors for cosine similarity, set normalize_embeddings=True. This is often useful for retrieval tasks:

python
embeddings = model.encode(sentences, normalize_embeddings=True, convert_to_tensor=True)

When the model is on GPU, normalization happens on the GPU and is inexpensive. It is also better to normalize the whole batch at once rather than doing it manually in a loop.

Common Failure Modes and How to Handle Them

One common mistake is mixing CPU and GPU tensors. If the model is on GPU and you pass a list of strings, the library handles the transfer automatically. However, if you try to move inputs manually, you can run into device mismatch errors. Let the library manage device placement unless you have a specific reason not to.

Another issue is using a batch size that is too large for the GPU. The library does not automatically reduce the batch size; it raises an error instead. You can catch that error and retry with a smaller batch:

python
try: embeddings = model.encode(sentences, batch_size=256) except RuntimeError as e: if 'out of memory' in str(e): embeddings = model.encode(sentences, batch_size=64) else: raise

This is a simple fallback, but in production you should determine a safe batch size in advance based on your GPU and typical sequence length.

When to Use Batch Encoding vs. Streaming or Incremental Encoding

For a one-off script that encodes a few thousand sentences, a single encode() call with a reasonable batch size is sufficient. For very large corpora that do not fit in memory, you may need to process the data in chunks. The library does not include a general-purpose streaming API for encode(); you can read your data in chunks and call encode() on each chunk, reusing the same model. This avoids loading all the sentences into RAM at once.

Batch encoding is also useful when you need to compare many sentences against each other, such as in clustering or semantic search. The embeddings are produced in one call, and you can then compute similarity matrices with NumPy or PyTorch operations.

If you are encoding one sentence at a time, for example in an online inference service, you cannot batch within a single encode() call. If the service receives many requests, consider collecting them briefly and encoding them together; otherwise, a single-sentence call is the normal path. Batching usually gives the largest benefit when you have many texts to process at once.

A final consideration is the interaction with torch.no_grad(). The encode() method already disables gradient computation internally, so you do not need to wrap it yourself. This saves memory and speeds up inference.

On multiple GPUs, encode() does not automatically split a batch across devices. You can select a specific GPU with device='cuda:0', but for most applications a single GPU with a well-chosen batch size is sufficient. If you need more throughput, use a dedicated inference service or shard the data across separate processes.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.