Python Pandas: Handling Missing Values with fillna, dropna, and replace
Learn how to handle missing values in pandas using dropna(), fillna(), and replace(), with syntax examples and practical guidance on choosing the right approach.

When working with real-world datasets in pandas, missing values appear as NaN, None, or even empty strings. The pandas library provides three primary methods for handling them: dropna(), fillna(), and replace(). Each serves a different purpose, and knowing when to use each is essential for clean data analysis. This article explains the syntax, behavior, and practical usage of these methods, with examples to help you choose the right approach.
How Missing Values Appear in pandas
Before deciding how to handle missing data, you need to detect it. In pandas, missing values are typically represented by numpy.nan (a float), None (Python's null), or the newer pd.NA for nullable integer and boolean dtypes. The isna() method returns a boolean mask indicating which cells are missing:
import pandas as pd
import numpy as np
df = pd.DataFrame({
'A': [1, 2, np.nan],
'B': ['x', None, 'z']
})
print(df.isna())Output:
A B
0 False False
1 False True
2 True FalseUnderstanding the representation of missing values matters because fillna() and dropna() treat all of them as missing by default, while replace() can target specific values.
Removing Rows or Columns with dropna()
dropna() removes rows or columns that contain missing values. The most common use is to drop rows with any missing value:
df_clean = df.dropna()This returns a new DataFrame with only rows that have no NaN or None. You can control the axis: axis=0 (default) drops rows, axis=1 drops columns. The how parameter accepts 'any' (default) or 'all'. Use how='all' to drop rows where every value is missing:
df.dropna(how='all')The thresh parameter sets a minimum number of non-missing values required to keep the row/column. For example, thresh=2 keeps rows with at least two non-null values:
df.dropna(thresh=2)To restrict the check to specific columns, use subset:
df.dropna(subset=['B'])This drops rows where column B is missing, ignoring A. dropna() is useful when you can afford to lose rows, such as when the missing data is not recoverable and the dataset is large enough.
Filling Missing Values with fillna()
fillna() replaces missing values with a constant, a computed value, or a method-based fill. The simplest form is a scalar:
df.fillna(0)This replaces every missing value with 0. You can pass a dictionary to fill different columns with different values:
df.fillna({'A': -1, 'B': 'unknown'})For time series or ordered data, you can propagate non-null values forward or backward:
df.ffill() # forward fill
# or
df.bfill() # backward fillIn current pandas, the method argument to fillna() is deprecated; the dedicated DataFrame.ffill() and DataFrame.bfill() methods are preferred. The limit parameter controls how many consecutive missing values to fill:
df.ffill(limit=1)fillna() is the go-to when you want to preserve the dataset size and need to impute missing values with a reasonable estimate, such as the column mean:
df['A'].fillna(df['A'].mean())Note that fillna() returns a new object by default. To modify the original DataFrame, set inplace=True, but this is generally discouraged in modern pandas code due to chaining issues (see the pitfalls section).
Replacing Values with replace()
While fillna() is specifically for missing values, replace() can substitute any value, including missing ones. This is useful when you want to treat certain sentinel values as missing or replace missing values with a different placeholder. For example, to replace NaN with a string:
df.replace(np.nan, 'missing')You can also replace multiple values at once:
df.replace({np.nan: 0, 'x': 'y'})The replace() method is more flexible than fillna() because it works on both missing and non-missing values. However, for missing values specifically, fillna() is often clearer and more direct. Use replace() when you need to map arbitrary values, such as converting 'N/A' or 'NULL' strings to np.nan before further processing:
df.replace(['N/A', 'NULL'], np.nan)This two-step pattern—replace sentinel strings with NaN, then use fillna() or dropna()—is common in data cleaning pipelines.
Choosing Between dropna, fillna, and replace
The choice depends on your data and the goal of your analysis. Use dropna() when:
- The rows with missing values are few and can be discarded without biasing the results.
- You need a complete-case analysis for statistical modeling.
- The missing values are concentrated in a column that is not essential.
Use fillna() when:
- You cannot afford to lose data, especially in small datasets.
- You have a sensible imputation strategy (mean, median, forward fill, or a constant).
- The missingness is random and imputation will not introduce significant bias.
Use replace() when:
- You need to convert non-standard missing markers (like
'N/A'or'NULL') intoNaNor another value. - You want to replace both missing and non-missing values in one operation.
- You are cleaning data before applying
dropna()orfillna().
In practice, these methods are often used together. A typical pipeline might first use replace() to standardize missing markers, then fillna() to impute, or dropna() to remove rows that remain incomplete.
Performance and Memory Considerations
All three methods return new objects by default. This means that each operation copies the data, which can be memory-intensive for large datasets. The inplace=True parameter can modify the original object, but it is often discouraged in modern pandas code and can lead to subtle bugs when used with chained indexing. The recommended approach is to assign the result to a new variable or the same variable:
df = df.dropna() # instead of df.dropna(inplace=True)When working with large data, consider the memory footprint of creating intermediate copies. For example, df.fillna(0) creates a full copy, but you can reduce memory by operating on specific columns:
df['A'] = df['A'].fillna(0)This still creates a new Series but avoids copying the entire DataFrame. Additionally, dropna() can be expensive if you have many columns because it checks each element. Using subset to limit the check to relevant columns reduces the work.
Common Pitfalls and Edge Cases
A frequent mistake is using inplace=True in a chained expression. For instance:
df[df['A'].notna()].fillna(0, inplace=True) # wrongThis may raise a SettingWithCopyWarning or silently fail because the chained selection returns a copy. Always assign the result to a variable instead.
Another edge case is mixing types. Filling a numeric column with a string changes the column to object dtype, which can break later operations. If you need that, convert the column to an explicit dtype afterwards.
When using replace() with a list, the replacement is element-wise, not list-wise. For example, df.replace([1, 2], [10, 20]) replaces 1 with 10 and 2 with 20. If you need to replace an entire list as a single value, use a dictionary.
Finally, remember that dropna() and fillna() treat None and np.nan as missing, but they do not treat empty strings as missing by default. If your dataset uses empty strings as placeholders, convert them first with replace('', np.nan).
Handling missing values correctly is a core part of data cleaning. By understanding the distinct roles of dropna(), fillna(), and replace(), you can choose the right tool for each situation and avoid common pitfalls that lead to incorrect analysis.
Try matching browser tools
All toolsSmall utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.
Related Python guides
All Python guidesPython iter(): How the Built-in Function Works
Explains Python's iter() built-in: how it creates iterators, the two-argument sentinel form, custom iterator classes, and common edge cases.
Python Pika: Queues, Exchanges, Routing Keys, and Acknowledgements
Learn how to use Python's Pika client with RabbitMQ: declare queues and exchanges, publish with routing keys, consume messages, and handle acknowledgements reliably.
Python Kafka: kafka-python vs Confluent Client
python kafka python vs confluent kafka: Compare kafka-python and Confluent Kafka Python clients: API differences, performance, feature coverage, and when to choose eac...
Python Kafka Offsets, Partitions, and Manual Commits
Learn how Kafka offsets and partitions work and how to manually commit offsets in Python consumers with kafka-python and confluent-kafka. Covers sync vs async commits, rebalances, and seeking.
More in Python
All Python guidesThe guides filed next to this one in Python, in library order.