Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Creating Processes in Python with subprocess

Learn how to create and manage processes in Python with the subprocess module: run commands, capture output, handle errors, and manage long-running processes.

~4 min readPython
#subprocess#process management#system commands#Popen
A Python process spawning a child process with input and output streams, illustrating subprocess creation.

To create and manage a process in Python, the standard library provides the subprocess module. It is the recommended way to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This article walks through the core APIs—subprocess.run and subprocess.Popen—and shows how to handle output, errors, timeouts, and resource cleanup in real-world scripts. The examples assume Python 3.7 or later.

The subprocess Module as the Primary API

Python's subprocess module replaces older functions like os.system and os.spawn. It gives you full control over how a child process is created, how its streams are connected, and how it terminates. The two most commonly used entry points are subprocess.run and subprocess.Popen. run is a high-level wrapper that waits for the process to complete and returns a CompletedProcess object. Popen is a lower-level constructor that gives you a handle to the running process, allowing you to interact with it while it executes.

Running a Command and Capturing Output

To run a command and capture its standard output and error, use subprocess.run with capture_output=True. This redirects both stdout and stderr to pipes. The text=True argument makes the output strings instead of bytes. Here's a minimal example:

python
import subprocess result = subprocess.run(['ls', '-l'], capture_output=True, text=True) print(result.stdout) print(result.stderr) print(result.returncode)

After the call, result.stdout and result.stderr contain the captured output, and result.returncode holds the exit status. If the command fails, the return code is non-zero, but run does not raise an exception unless you pass check=True.

Passing Arguments Safely

One common mistake is passing a command as a single string and relying on shell=True. That can lead to injection vulnerabilities when the string includes user input, and it makes quoting unpredictable. Instead, pass a list of arguments. With a list, subprocess does not need to invoke a shell, so command arguments are not interpreted by the shell. For example:

python
# Preferred: list of arguments subprocess.run(['grep', 'pattern', 'file.txt']) # Avoid unless you need shell features, and never with untrusted input subprocess.run('grep pattern file.txt', shell=True)

The list form is preferred unless you explicitly need shell features like globbing or environment variable expansion. If you must use shell=True, never pass user input directly into the string.

Controlling Process Input and Output Streams

When you need to send data to a process's stdin or manage a process while it is running, use Popen. The communicate() method sends input and waits for the process to finish, returning a tuple of (stdout, stderr). Here's an example that feeds a line to a child process:

python
import subprocess proc = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True) stdout, stderr = proc.communicate(input='hello\n') print(stdout)

Popen is also useful when you need to start a process and continue doing other work while it runs. You can poll proc.poll() to check if it has exited, or read from proc.stdout line by line. If you read incrementally from one pipe, remember to consume or redirect the other pipes as well; otherwise a full pipe buffer can block the child.

Handling Errors and Exit Codes

To make a failed command raise an exception, pass check=True to run. This raises subprocess.CalledProcessError if the return code is non-zero. The exception object contains the command, return code, and captured output:

python
import subprocess try: subprocess.run(['false'], check=True, capture_output=True, text=True) except subprocess.CalledProcessError as e: print(f'Command failed with exit code {e.returncode}')

If you need to handle different failure modes, check the returncode manually and branch accordingly. For many scripts, check=True is the simplest way to fail fast.

Managing Long-Running Processes and Timeouts

Long-running processes can hang your script. The run() function and the Popen.communicate() method both accept a timeout argument. If the process does not finish within the given seconds, a TimeoutExpired exception is raised. With run(), the child process is killed and waited for before the exception is re-raised; with Popen.communicate(), you usually need to kill or terminate the process yourself in the except block.

python
import subprocess try: subprocess.run(['sleep', '10'], timeout=2) except subprocess.TimeoutExpired: print('Process timed out')

If you use Popen directly, call proc.kill() or proc.terminate() in the except block to stop the child when a timeout expires. Always handle timeouts in production code so you do not leave a child process running.

Performance and Resource Considerations

Creating a process is expensive compared to a function call. Each subprocess spawns a separate OS process, which involves memory allocation, file descriptor setup, and scheduler overhead. If you need to run many short commands, consider batching them into a single script or using a persistent worker process.

Also, explicitly wait for Popen processes. If stdout or stderr is a pipe, use communicate() rather than wait() alone, so the child does not block on a full pipe. Call wait() or communicate(), or use Popen as a context manager, so the child is reaped and file descriptors are released. Otherwise, finished child processes can remain as zombies until the parent process exits.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.