Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python PyTorch Autograd: requires_grad and backward

Understand how PyTorch's autograd engine uses requires_grad and backward() to compute gradients, including graph construction, detach(), no_grad(), and common pitfalls.

~6 min readPython
#PyTorch#autograd#requires_grad#backward
Diagram of a PyTorch computational graph showing gradient flow backward from a loss node to parameter nodes.

PyTorch's autograd engine computes gradients by recording operations that produce tensors with requires_grad=True. When you call backward() on a tensor, it propagates gradients back through that recorded graph to the leaf tensors that require them. This article explains how requires_grad and backward() work together, how to control gradient tracking, and where common mistakes occur.

How Autograd Builds a Computational Graph

PyTorch's autograd records operations on tensors that have requires_grad=True. Each operation creates a node in a directed acyclic graph (DAG) that connects the input tensors to the output. The graph stores the function used to compute the output, so that during backward() the gradient can be propagated back through the chain rule.

Consider a simple expression:

python
import torch x = torch.tensor([2.0], requires_grad=True) y = x * 3 z = y.sum()

Here, x is a leaf tensor because it was created directly by the user. y is a non-leaf tensor produced by the multiplication operation. z is the result of a sum. The autograd graph links z to y and y to x. When you call z.backward(), PyTorch walks this graph backward, computing gradients for the leaf tensors that require them.

Tensors with requires_grad=True are connected to the backward graph, while a tensor with requires_grad=False is treated as a fixed input: it does not receive a gradient, and operations that produce no tensor requiring a gradient are not recorded.

Setting requires_grad on Tensors

The requires_grad flag controls whether autograd tracks operations on a tensor. By default, tensors created with torch.tensor() have requires_grad=False. You can enable it at creation time:

python
x = torch.tensor([1.0, 2.0], requires_grad=True)

Or you can set it later using the .requires_grad_() method (note the trailing underscore, which indicates an in-place operation):

python
x = torch.ones(3) x.requires_grad_(True)

When you set requires_grad=True on a tensor, autograd starts recording operations that produce outputs depending on that tensor. This is necessary for training, but it also increases memory usage because the graph must be stored until backward() is called.

For tensors that are used only for inference or as fixed inputs, leaving requires_grad=False avoids unnecessary graph construction. You can also disable gradient tracking temporarily with torch.no_grad() or permanently with detach().

Calling backward() to Compute Gradients

The backward() method propagates gradients from the current tensor to the leaf tensors that require them. It is typically called on a scalar loss value:

python
loss = (y - target).pow(2).sum() loss.backward()

After calling backward(), the .grad attribute of each leaf tensor with requires_grad=True is populated with the gradient. For example:

python
x = torch.tensor(2.0, requires_grad=True) y = x ** 2 y.backward() print(x.grad) # tensor(4.0)

The gradient is accumulated into .grad. If you call backward() again in a later step without clearing the gradient, the new gradient is added to the existing value. This is intentional for gradient accumulation across mini-batches, but it can lead to unexpected results if you forget to zero the gradients before each training step.

For a non-scalar tensor with more than one element, you must provide a gradient argument that matches the shape of the tensor. This argument represents the gradient of the final loss with respect to that tensor. For example:

python
x = torch.tensor([1.0, 2.0], requires_grad=True) y = x * 2 # y is a vector, so we need to pass a gradient vector y.backward(torch.tensor([1.0, 1.0])) print(x.grad) # tensor([2.0, 2.0])

In practice, you rarely call backward() on a non-scalar tensor directly; instead, you reduce the output to a scalar loss first.

Stopping Gradient Tracking: detach() and no_grad()

There are two common ways to prevent autograd from tracking operations: detach() and torch.no_grad(). They serve different purposes.

tensor.detach() returns a new tensor that shares the same data but has requires_grad=False and is not connected to the current graph. This is useful when you need to use a tensor's value for something that should not affect gradient computation, such as logging or computing a metric:

python
x = torch.tensor([1.0], requires_grad=True) y = x * 2 y_detached = y.detach() # no gradient connection

torch.no_grad() is a context manager that disables gradient tracking for all operations inside the block. It is commonly used during evaluation or inference:

python
with torch.no_grad(): predictions = model(x_val)

Inside the block, tensors created from operations have requires_grad=False, and existing tensors with requires_grad=True do not have their operations recorded. This saves memory and computation because no graph is built.

A related context manager is torch.inference_mode(), which is intended for inference-only code and can be faster than no_grad() because it skips additional autograd bookkeeping. It is also more restrictive, so use no_grad() when you need flexibility and inference_mode() when you are certain no backward pass is needed.

Common Pitfalls with requires_grad and backward

Several subtle issues arise when working with autograd. One is in-place operations on tensors that require gradients. For example, using x.add_(1) on a leaf tensor can corrupt the graph because the original value is overwritten, and autograd may raise an error or produce incorrect gradients. In general, avoid in-place operations on tensors that are part of the graph.

Another pitfall is forgetting to zero gradients before a training step. Gradients accumulate by default, so if you call backward() in a loop without zeroing .grad, the gradients from previous steps are added to the new ones, leading to unstable training.

Leaf tensors are also a common source of confusion. Only leaf tensors get their .grad populated by default. Non-leaf tensors do not store gradients unless you explicitly use retain_grad(). If you need the gradient of an intermediate tensor, you must set retain_grad() on it, or use register_hook().

Finally, calling backward() on a tensor with more than one element and without a gradient argument raises a runtime error. Always reduce the output to a scalar loss before calling backward().

Performance and Memory Implications of Autograd

Building the computational graph has a real cost. Every recorded operation allocates memory for the graph node and stores intermediate values needed for the backward pass. This can significantly increase memory usage compared to inference-only code.

To minimize overhead, disable gradient tracking whenever you do not need gradients. For example, when evaluating a model on a validation set, wrap the forward pass in torch.no_grad(). This prevents the graph from being built and reduces memory consumption.

Another consideration is the retain_graph parameter of backward(). By default, the graph is freed after a backward pass to save memory. If you need to call backward() multiple times on the same graph—for example, when several loss terms are computed from shared activations—pass retain_graph=True. This keeps the graph alive but increases memory usage. Higher-order gradients are a separate case: use create_graph=True when you want the gradient computation itself to be recorded.

For large models, the graph can become the dominant memory consumer. Techniques like gradient checkpointing trade computation for memory by recomputing intermediate activations during the backward pass. However, for most use cases, simply using no_grad() during inference and clearing gradients each step is sufficient.

Practical Example: Training a Linear Regression Model

To see requires_grad and backward() in action, consider a minimal linear regression training loop. The model has one parameter w and one bias b. We create them with requires_grad=True so that autograd tracks operations on them.

python
import torch # Training data x = torch.linspace(0, 10, 100) y_true = 3 * x + 2 + torch.randn(100) * 0.5 # Model parameters w = torch.randn(1, requires_grad=True) b = torch.zeros(1, requires_grad=True) learning_rate = 0.01 for epoch in range(100): # Forward pass y_pred = w * x + b loss = (y_pred - y_true).pow(2).mean() # Backward pass loss.backward() # Update parameters (no autograd tracking) with torch.no_grad(): w -= learning_rate * w.grad b -= learning_rate * b.grad # Zero gradients w.grad.zero_() b.grad.zero_() print(f"Learned w: {w.item():.3f}, b: {b.item():.3f}")

In this loop, loss.backward() computes the gradients of w and b. The parameter update is done inside torch.no_grad() to avoid building a graph for the update operation itself. After updating, we zero the gradients so the next backward pass does not accumulate.

This example demonstrates the core pattern used in many PyTorch training loops: define parameters with requires_grad=True, compute a loss, call backward(), update parameters without gradient tracking, and zero gradients before the next iteration. When debugging autograd, check which tensors are leaves, whether requires_grad is set as intended, and whether gradients have been zeroed before the next step.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.