Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

How to Save and Load PyTorch state_dict and Checkpoints

Learn how to save and load PyTorch models with state_dict and full checkpoints, including optimizer state, epoch, and safe device handling.

~5 min readPython
#PyTorch#state_dict#checkpoints#model persistence
Illustration of a PyTorch model state_dict being saved and loaded as a checkpoint file

Working with PyTorch, saving and loading model state is a routine task in almost every training pipeline. There are two related but distinct operations to understand: saving the model's parameter tensors as a state_dict, and saving a full checkpoint that also includes optimizer state, epoch, and other training metadata. Knowing when to use each is essential for reproducible experiments and resumable training.

What a state_dict Contains

Every PyTorch nn.Module has a state_dict() method that returns a Python mapping from each learnable parameter and persistent buffer to its current tensor value. For a convolutional network, this includes convolution weights and biases, batch norm running statistics, and any other registered parameters or persistent buffers. The state_dict is an OrderedDict, so it can be inspected, modified, and saved directly.

python
import torch import torch.nn as nn model = nn.Linear(10, 5) print(model.state_dict().keys()) # odict_keys(['weight', 'bias'])

For parameters, the keys are the same names produced by named_parameters(); persistent buffers are included in the mapping as well. For nested modules, the keys are prefixed with the module path, such as features.0.weight. This naming is what allows the same model architecture to load a state_dict later.

Saving and Loading a state_dict

The simplest way to persist a model is to save only the state_dict. This is sufficient when you need to reuse the model for inference or transfer learning, and when you do not need to resume training from the exact point where it stopped.

python
# Save torch.save(model.state_dict(), 'model_weights.pth') # Load model = nn.Linear(10, 5) # must match the original architecture model.load_state_dict(torch.load('model_weights.pth')) model.eval()

torch.save uses Python's pickle-based serialization to write the mapping, and torch.load deserializes it. The file extension is arbitrary; .pt, .pth, and .bin are all common. When loading, you must instantiate the model first and then call load_state_dict on it. The model architecture must produce matching parameter names and shapes; otherwise the default strict load fails.

Saving and Loading a Full Checkpoint

A full checkpoint goes beyond the model parameters. It also saves the optimizer's state_dict, the current epoch, learning rate scheduler state, and any other training variables you need to resume training exactly where you left off. This is the standard approach for long-running training jobs where interruptions are expected.

python
# During training torch.save({ 'epoch': epoch, 'model_state_dict': model.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'loss': loss, }, 'checkpoint.pt') # To resume def load_checkpoint(path, model, optimizer): checkpoint = torch.load(path) model.load_state_dict(checkpoint['model_state_dict']) optimizer.load_state_dict(checkpoint['optimizer_state_dict']) start_epoch = checkpoint['epoch'] + 1 return model, optimizer, start_epoch

The checkpoint dictionary is a regular Python dict, so you can add any metadata you need, such as validation metrics, random number generator states, or configuration arguments. This makes the checkpoint self-contained and allows you to reproduce the exact training state. After resuming, remember to call model.train() if you are continuing training.

Handling Device Placement When Loading

PyTorch tensors are device-specific. When a model is trained on a GPU, its state_dict tensors live in CUDA memory, and saving them records that device information. If you later load the checkpoint on a machine without a GPU, or on a machine where the original device is unavailable or not the one you want, you must tell PyTorch where to load the tensors. The torch.load function accepts a map_location argument to handle this.

python
# Load on CPU from a GPU-trained checkpoint checkpoint = torch.load('checkpoint.pt', map_location='cpu') model.load_state_dict(checkpoint['model_state_dict'])

Without map_location, PyTorch attempts to restore CUDA tensors to the CUDA device recorded in the file. That can fail when the device is unavailable and can place tensors on the wrong device in multi-GPU settings. For mixed-device environments, a common pattern is to load with map_location='cpu' first, then move the model to the target device after loading. This avoids partial-load errors and keeps the code portable.

Dealing with Missing or Unexpected Keys

When loading a state_dict into a model, PyTorch validates that the key names and tensor shapes match by default. If the model architecture differs, the error lists missing and unexpected keys; if a shared key has a different shape, loading raises a size mismatch. The load_state_dict method has a strict parameter, set to True by default.

python
# Non-strict loading: allow missing or unexpected keys model.load_state_dict(torch.load('weights.pth'), strict=False)

strict=False is useful when you want to load only a subset of layers. It does not ignore shape mismatches for keys that are present, and it can hide missing or unexpected keys. A safer approach is to load with strict=True and explicitly handle the issues reported in the error, or filter the state_dict before loading.

For transfer learning, you often need to load only the backbone weights. You can filter the state_dict before loading:

python
pretrained_dict = torch.load('backbone.pth') model_dict = model.state_dict() # Filter out missing keys and keys whose shapes do not match pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict and v.shape == model_dict[k].shape} model_dict.update(pretrained_dict) model.load_state_dict(model_dict)

This pattern lets you reuse pretrained layers while keeping incompatible layers randomly initialized. Here backbone.pth is expected to contain a plain state_dict; if it is a full checkpoint, unpack it first with pretrained_dict = checkpoint['model_state_dict'].

Checkpointing Strategies for Long Training Runs

Saving checkpoints at regular intervals is critical for long training runs. A common practice is to save every N epochs, and also keep the best model based on a validation metric. This protects against interruptions and lets you revert to a known-good state.

python
best_loss = float('inf') for epoch in range(num_epochs): train_one_epoch() val_loss = validate() # Save latest checkpoint torch.save({ 'epoch': epoch, 'model_state_dict': model.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'val_loss': val_loss, }, 'checkpoint_last.pt') # Save best model separately if val_loss < best_loss: best_loss = val_loss torch.save(model.state_dict(), 'best_model.pth')

Storing the best model as a plain state_dict is convenient for inference because you do not need the optimizer or epoch. The full checkpoint is for resuming training. Separating the two keeps the inference artifact small and avoids loading unnecessary metadata.

When disk space is a concern, you can overwrite the latest checkpoint and keep only the best model. Some training frameworks also use cyclic checkpoints, saving every K epochs and deleting older ones. The tradeoff is between recovery granularity and storage cost.

Another consideration is the file format. torch.save uses pickle-based serialization, which is not secure against maliciously crafted files. Only load checkpoints from trusted sources. For large models, you can use torch.save with _use_new_zipfile_serialization=True (the default in recent versions) to produce a zip archive that is more portable and easier to inspect.

Finally, when saving checkpoints in distributed training, ensure that only one process writes to disk to avoid file corruption. The common pattern is to save only on the main process, or to use a temporary file and rename it atomically after writing.

By understanding the difference between state_dict and full checkpoints, and by following these loading and saving patterns, you can build training pipelines that are robust, reproducible, and easy to resume.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.