Creating Processes in Python with subprocess
Learn how to create and manage processes in Python with the subprocess module: run commands, capture output, handle errors, and manage long-running processes.

To create and manage a process in Python, the standard library provides the subprocess module. It is the recommended way to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This article walks through the core APIs—subprocess.run and subprocess.Popen—and shows how to handle output, errors, timeouts, and resource cleanup in real-world scripts. The examples assume Python 3.7 or later.
The subprocess Module as the Primary API
Python's subprocess module replaces older functions like os.system and os.spawn. It gives you full control over how a child process is created, how its streams are connected, and how it terminates. The two most commonly used entry points are subprocess.run and subprocess.Popen. run is a high-level wrapper that waits for the process to complete and returns a CompletedProcess object. Popen is a lower-level constructor that gives you a handle to the running process, allowing you to interact with it while it executes.
Running a Command and Capturing Output
To run a command and capture its standard output and error, use subprocess.run with capture_output=True. This redirects both stdout and stderr to pipes. The text=True argument makes the output strings instead of bytes. Here's a minimal example:
import subprocess
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)
print(result.returncode)After the call, result.stdout and result.stderr contain the captured output, and result.returncode holds the exit status. If the command fails, the return code is non-zero, but run does not raise an exception unless you pass check=True.
Passing Arguments Safely
One common mistake is passing a command as a single string and relying on shell=True. That can lead to injection vulnerabilities when the string includes user input, and it makes quoting unpredictable. Instead, pass a list of arguments. With a list, subprocess does not need to invoke a shell, so command arguments are not interpreted by the shell. For example:
# Preferred: list of arguments
subprocess.run(['grep', 'pattern', 'file.txt'])
# Avoid unless you need shell features, and never with untrusted input
subprocess.run('grep pattern file.txt', shell=True)The list form is preferred unless you explicitly need shell features like globbing or environment variable expansion. If you must use shell=True, never pass user input directly into the string.
Controlling Process Input and Output Streams
When you need to send data to a process's stdin or manage a process while it is running, use Popen. The communicate() method sends input and waits for the process to finish, returning a tuple of (stdout, stderr). Here's an example that feeds a line to a child process:
import subprocess
proc = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
stdout, stderr = proc.communicate(input='hello\n')
print(stdout)Popen is also useful when you need to start a process and continue doing other work while it runs. You can poll proc.poll() to check if it has exited, or read from proc.stdout line by line. If you read incrementally from one pipe, remember to consume or redirect the other pipes as well; otherwise a full pipe buffer can block the child.
Handling Errors and Exit Codes
To make a failed command raise an exception, pass check=True to run. This raises subprocess.CalledProcessError if the return code is non-zero. The exception object contains the command, return code, and captured output:
import subprocess
try:
subprocess.run(['false'], check=True, capture_output=True, text=True)
except subprocess.CalledProcessError as e:
print(f'Command failed with exit code {e.returncode}')If you need to handle different failure modes, check the returncode manually and branch accordingly. For many scripts, check=True is the simplest way to fail fast.
Managing Long-Running Processes and Timeouts
Long-running processes can hang your script. The run() function and the Popen.communicate() method both accept a timeout argument. If the process does not finish within the given seconds, a TimeoutExpired exception is raised. With run(), the child process is killed and waited for before the exception is re-raised; with Popen.communicate(), you usually need to kill or terminate the process yourself in the except block.
import subprocess
try:
subprocess.run(['sleep', '10'], timeout=2)
except subprocess.TimeoutExpired:
print('Process timed out')If you use Popen directly, call proc.kill() or proc.terminate() in the except block to stop the child when a timeout expires. Always handle timeouts in production code so you do not leave a child process running.
Performance and Resource Considerations
Creating a process is expensive compared to a function call. Each subprocess spawns a separate OS process, which involves memory allocation, file descriptor setup, and scheduler overhead. If you need to run many short commands, consider batching them into a single script or using a persistent worker process.
Also, explicitly wait for Popen processes. If stdout or stderr is a pipe, use communicate() rather than wait() alone, so the child does not block on a full pipe. Call wait() or communicate(), or use Popen as a context manager, so the child is reaped and file descriptors are released. Otherwise, finished child processes can remain as zombies until the parent process exits.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.