Python Sentence Transformers Batch Encoding and GPU
Learn how to batch encode texts with sentence-transformers on a GPU, control batch size, avoid out-of-memory errors, and choose the right output format for embeddings.

When you need embeddings for thousands of sentences, calling model.encode() on each sentence individually is wasteful. The sentence-transformers library is designed to encode lists of texts in one call, and on a GPU that call can process many sentences in parallel. This article explains how batch encoding works with sentence-transformers on a GPU, how to control batch size, and what to watch out for when scaling up.
Why Batch Encoding Matters for GPU Inference
A transformer model processes token sequences, normally with padding inside a batch so the inputs form a rectangular tensor. When you call encode() with a single sentence, the GPU executes the forward pass for that one sequence. With a batch, the GPU processes multiple sequences in one forward pass, sharing the model weights and spreading fixed overhead across more inputs. The result is usually higher throughput: more sentences per second, and often lower total latency for a large corpus.
GPU hardware is designed for parallel computation. Once a model is loaded, a GPU can often process many sequences in parallel, so batching is a practical way to use the device efficiently. The exact speedup varies with GPU capacity, model size, and sequence length, but a loop of single-sentence calls generally leaves much of that capacity unused.
Basic Batch Encoding with sentence-transformers
The SentenceTransformer class provides an encode() method that accepts a list of strings. Here is the simplest form:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = [
'The cat sits on the mat.',
'A dog runs through the park.',
'Machine learning models require data.'
]
embeddings = model.encode(sentences)The embeddings variable is a NumPy array of shape (3, 384) for this model. Internally, the library tokenizes all the sentences, pads them to the same length, and runs the transformer on the whole batch. In common releases of the library, the default batch size is 32; you can change it with the batch_size argument.
Controlling Batch Size and Device
The encode() method accepts a batch_size parameter. You can specify the device explicitly, for example device='cuda' to force GPU usage. If PyTorch detects a GPU, the model is usually placed on it automatically, but being explicit can make the behavior clearer:
embeddings = model.encode(sentences, batch_size=64, device='cuda')When you pass a batch size larger than the number of sentences, the library processes all the sentences in one batch. When the batch size is smaller, it splits the list into chunks and processes each chunk sequentially. The show_progress_bar parameter can help you monitor progress for large corpora:
embeddings = model.encode(sentences, batch_size=128, show_progress_bar=True)GPU Memory and Choosing the Right Batch Size
GPU memory is finite. Each sentence in a batch consumes memory for token embeddings, attention masks, and intermediate activations during the forward pass. The exact amount depends on the model size and the maximum sequence length. If the batch size is too large, you will get a CUDA out-of-memory error. The error typically looks like:
RuntimeError: CUDA out of memory.To avoid this, find the largest batch size that fits on your GPU. A practical approach is to start with a small value, say 16, and double it until you hit an error, then back off. The optimal batch size also depends on the sequence length of your sentences. Longer sentences consume more memory because the standard transformer attention matrix grows quadratically with sequence length.
The table below is illustrative, not a benchmark.
| Batch size | Memory usage | Throughput |
|---|---|---|
| 16 | Low | Moderate |
| 32 | Moderate | Good |
| 64 | High | Better |
| 128 | Very high | Depends on GPU and sequence length |
The key is to monitor memory usage and choose the largest batch that does not cause an out-of-memory error.
Normalization and Output Formats
The encode() method has several useful options. By default, it returns a NumPy array of float32 vectors. If you need PyTorch tensors, set convert_to_tensor=True. If you need normalized vectors for cosine similarity, set normalize_embeddings=True. This is often useful for retrieval tasks:
embeddings = model.encode(sentences, normalize_embeddings=True, convert_to_tensor=True)When the model is on GPU, normalization happens on the GPU and is inexpensive. It is also better to normalize the whole batch at once rather than doing it manually in a loop.
Common Failure Modes and How to Handle Them
One common mistake is mixing CPU and GPU tensors. If the model is on GPU and you pass a list of strings, the library handles the transfer automatically. However, if you try to move inputs manually, you can run into device mismatch errors. Let the library manage device placement unless you have a specific reason not to.
Another issue is using a batch size that is too large for the GPU. The library does not automatically reduce the batch size; it raises an error instead. You can catch that error and retry with a smaller batch:
try:
embeddings = model.encode(sentences, batch_size=256)
except RuntimeError as e:
if 'out of memory' in str(e):
embeddings = model.encode(sentences, batch_size=64)
else:
raiseThis is a simple fallback, but in production you should determine a safe batch size in advance based on your GPU and typical sequence length.
When to Use Batch Encoding vs. Streaming or Incremental Encoding
For a one-off script that encodes a few thousand sentences, a single encode() call with a reasonable batch size is sufficient. For very large corpora that do not fit in memory, you may need to process the data in chunks. The library does not include a general-purpose streaming API for encode(); you can read your data in chunks and call encode() on each chunk, reusing the same model. This avoids loading all the sentences into RAM at once.
Batch encoding is also useful when you need to compare many sentences against each other, such as in clustering or semantic search. The embeddings are produced in one call, and you can then compute similarity matrices with NumPy or PyTorch operations.
If you are encoding one sentence at a time, for example in an online inference service, you cannot batch within a single encode() call. If the service receives many requests, consider collecting them briefly and encoding them together; otherwise, a single-sentence call is the normal path. Batching usually gives the largest benefit when you have many texts to process at once.
A final consideration is the interaction with torch.no_grad(). The encode() method already disables gradient computation internally, so you do not need to wrap it yourself. This saves memory and speeds up inference.
On multiple GPUs, encode() does not automatically split a batch across devices. You can select a specific GPU with device='cuda:0', but for most applications a single GPU with a well-chosen batch size is sufficient. If you need more throughput, use a dedicated inference service or shard the data across separate processes.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.