Python Threads: Concurrency and Synchronization
Understand how to create and manage Python threads, synchronize access to shared data, and work effectively around the GIL.

When you need to run multiple tasks concurrently in Python, the threading module provides a straightforward way to create and manage threads. This article focuses on the practical aspects of the Python thread model: creating and starting threads, synchronizing access to shared state, and understanding how the Global Interpreter Lock (GIL) shapes their behavior.
Creating and Starting Threads
The Thread class is the core abstraction. You can create a thread by passing a callable target and then calling start():
import threading
import time
def worker(name):
print(f"Worker {name} starting")
time.sleep(2)
print(f"Worker {name} finished")
t1 = threading.Thread(target=worker, args=("A",))
t2 = threading.Thread(target=worker, args=("B",))
t1.start()
t2.start()
t1.join()
t2.join()The start() method begins execution in a new thread, and join() blocks until the thread completes. Without join(), the main thread has no way to know when the workers have completed, which can cause problems if later code depends on their results. Daemon threads, in particular, are cut off when the interpreter exits.
You can also subclass Thread and override run(), but passing a target function is usually simpler and keeps the logic separate.
Thread Lifecycle and Daemon Threads
A thread runs until its target function returns or raises an unhandled exception. By default, threads are non-daemon, meaning the program will wait for them to finish before exiting. Daemon threads are abruptly terminated when the main program exits, which is useful for background tasks like monitoring or periodic cleanup.
daemon = threading.Thread(target=monitor, daemon=True)
daemon.start()A daemon thread does not block program exit. This can be dangerous if it is holding resources or writing data. Use daemon threads only when you can tolerate losing their work on shutdown.
Synchronization: Locks, RLock, and Semaphore
When multiple threads read and write the same data, you need synchronization to prevent race conditions. The simplest mechanism is a Lock:
lock = threading.Lock()
counter = 0
def increment():
global counter
for _ in range(1000):
with lock:
counter += 1The with lock: block acquires and releases the lock automatically. A Lock is non-reentrant: if the same thread tries to acquire it again without releasing, it will deadlock. For reentrant behavior, use RLock, which allows the same thread to acquire it multiple times.
A Semaphore limits how many threads can enter a critical section at once. It is useful for controlling access to a fixed pool of resources, such as a limited number of database connections.
Thread-Safe Data Structures and Queues
Python's queue.Queue is designed for thread-safe communication. It handles the locking internally, so you can safely pass work between threads:
import queue
work_queue = queue.Queue()
def producer():
for i in range(10):
work_queue.put(i)
work_queue.put(None) # sentinel to stop the consumer
def consumer():
while True:
item = work_queue.get()
if item is None:
work_queue.task_done()
break
process(item)
work_queue.task_done()The producer sends a None sentinel to signal the end of input. In real code, replace process(item) with the actual work the consumer should do. The get() method blocks until an item is available, and task_done() signals that processing of a queue entry is complete. Using a queue avoids the need for explicit locks when passing data between threads.
The GIL and Its Effect on Thread Performance
In CPython, the Global Interpreter Lock (GIL) is a mutex that protects the interpreter's internal state. It ensures only one thread executes Python bytecode at a time. As a result, CPU-bound Python code does not gain speed from threads in CPython, because the GIL prevents parallel execution across cores. I/O-bound tasks, however, benefit significantly because the GIL is released during blocking I/O operations, allowing other threads to run.
If your workload is CPU-heavy, consider using multiprocessing instead. It starts separate processes, each with its own Python interpreter, so a shared GIL does not limit CPU-bound parallelism. For I/O-bound work, threads are often simpler and use less memory than processes.
When to Use Threads vs Processes
| Criterion | Threads | Processes |
|---|---|---|
| Memory overhead | Low (shared address space) | High (separate memory) |
| Data sharing | Shared by default | Requires IPC |
| GIL impact | Limited for CPU-bound | None (separate interpreters) |
| Startup cost | Low | Higher |
| Best fit | I/O-bound tasks | CPU-bound parallel tasks |
Use threads when your tasks spend most of their time waiting on network, disk, or user input. Use processes when you need to utilize multiple CPU cores for computation.
Common Pitfalls and Debugging Threads
One frequent issue is relying on worker completion without calling join(); if the main thread exits while daemon threads are still running, their remaining work may be lost. Another is sharing mutable state without a lock, causing unpredictable behavior. When debugging, use threading.current_thread().name to identify which thread is executing a log line.
Deadlocks often occur when two threads each hold a lock and then wait for the other's lock. To avoid them, acquire locks in a consistent order and use timeouts with lock.acquire(timeout=...) when possible.
Managing Thread Lifecycles in Production
In production systems, creating a new thread for every task is inefficient. Use a thread pool to reuse a fixed number of threads. The concurrent.futures.ThreadPoolExecutor provides a high-level interface:
from concurrent.futures import ThreadPoolExecutor
def task(n):
return n * 2
with ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(task, range(10)))This handles thread creation, reuse, and shutdown automatically. It also provides a way to collect results and exceptions, making it easier to manage the lifecycle of concurrent work.
Threads are a powerful tool for concurrent I/O, but they require careful synchronization and an understanding of the GIL's limitations. By choosing the right abstraction—whether raw threads, queues, or a thread pool—you can keep your code both correct and maintainable.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.