Handling Python sklearn Missing Values with SimpleImputer

Most scikit-learn estimators assume the input feature matrix is complete. A NaN in a feature column will often raise a ValueError during fit, and many estimators do not have native missing-value support. Before training, missing values must be resolved with an imputation step. SimpleImputer is the standard sklearn tool for this task, and it fits naturally into a preprocessing pipeline. Knowing how to use SimpleImputer means knowing not just the API, but also when each strategy is appropriate and how to combine imputation with the rest of your preprocessing flow.

Why Missing Values Break sklearn Models

Most scikit-learn estimators assume a complete numeric input matrix. A NaN in the feature matrix will typically cause fit to raise an error such as ValueError: Input contains NaN. A small number of estimators have native support for missing values, but for the majority you need to remove or replace missing values before the data reaches the estimator.

Imputation replaces missing entries with a computed value rather than dropping entire rows. This is preferable when the missingness is informative or when dropping rows would remove too much training data. SimpleImputer performs this replacement column by column, learning a statistic from the observed values and applying it to the gaps.

SimpleImputer Basics

The default behavior of SimpleImputer is to fill each column with its mean, but the strategy is configurable. A minimal usage looks like this:

import numpy as np
from sklearn.impute import SimpleImputer

X = np.array([[1.0, 2.0], [np.nan, 3.0], [4.0, np.nan]])
imputer = SimpleImputer(strategy='mean')
X_imputed = imputer.fit_transform(X)

The imputer learns the mean of each non-missing column during fit, then applies that value to the missing entries during transform. The result is a complete array that can be passed directly to an estimator. The fit_transform method combines both steps for convenience, but when you are building a pipeline, the imputer should be fitted only on training data.

Choosing an Imputation Strategy

The strategy parameter controls how missing values are replaced:

For numeric data, median is often safer than mean when the feature has extreme values, since the median is not pulled by outliers. For categorical data, most_frequent preserves the original value space, while constant lets you inject a domain default such as unknown. The constant strategy lets you supply a fill_value for the replacement; if you leave it as None, the imputer uses 0 for numeric data and missing_value for object data.

Handling Categorical and Numeric Features Together

Real datasets rarely contain only one data type. A common pattern is to use ColumnTransformer to apply different imputation strategies to different column groups.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer

numeric_cols = ['age', 'income']
categorical_cols = ['city', 'occupation']

preprocessor = ColumnTransformer(
    transformers=[
        ('num', SimpleImputer(strategy='median'), numeric_cols),
        ('cat', SimpleImputer(strategy='most_frequent'), categorical_cols),
    ]
)

The ColumnTransformer applies the numeric imputer only to the listed numeric columns and the categorical imputer only to the categorical columns. This prevents a categorical column from receiving a mean value, which would corrupt the feature space. If categorical columns also need one-hot encoding, put the imputer and encoder together in a small Pipeline and use that Pipeline as the transformer for those columns.

Using SimpleImputer Inside a Pipeline

Imputation should stay in the same Pipeline as the rest of the preprocessing and the model so that imputation statistics are learned only from training data. When you mix numeric and categorical columns, apply scaling to numeric columns only and use one-hot encoding for categorical columns.

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_cols = ['age', 'income']
categorical_cols = ['city', 'occupation']

preprocessor = ColumnTransformer(
    transformers=[
        (
            'num',
            Pipeline(
                steps=[
                    ('imputer', SimpleImputer(strategy='median')),
                    ('scaler', StandardScaler()),
                ]
            ),
            numeric_cols,
        ),
        (
            'cat',
            Pipeline(
                steps=[
                    ('imputer', SimpleImputer(strategy='most_frequent')),
                    ('encoder', OneHotEncoder(handle_unknown='ignore')),
                ]
            ),
            categorical_cols,
        ),
    ]
)

pipeline = Pipeline(
    steps=[
        ('preprocess', preprocessor),
        ('model', RandomForestClassifier()),
    ]
)

When the pipeline is fitted, each imputer computes its statistics from the training data. During prediction, those learned statistics are applied to new data. In cross-validation, the pipeline refits from scratch in each fold, so every validation split is transformed using statistics learned only from the corresponding training split. This avoids data leakage. If you imputed the entire dataset before splitting into train and test sets, the test fold would indirectly influence the imputation values used for training, which inflates validation scores.

Common Pitfalls with SimpleImputer

NaN in Categorical Columns

If a categorical column contains NaN and you apply strategy='mean', the imputer will fail because the mean of a string column is undefined. Always match the strategy to the column type. For object-dtype columns, use most_frequent or constant.

All-Missing Columns

When a column is entirely missing, strategy='mean' and strategy='median' cannot compute a statistic. The imputer raises an error during fit. In this case, strategy='constant' with an explicit fill_value is the only option that works. You may also want to drop the column entirely if it carries no information.

Missing Values in the Target

SimpleImputer only handles feature matrices. Missing values in the target vector y must be handled separately, usually by dropping those rows before training. Imputing the target is rarely appropriate because it introduces artificial labels that the model cannot learn from reliably.

Sparse Matrices

If your data is stored as a scipy sparse matrix, check how the imputer and the rest of your preprocessing handle sparse input. Some strategies or estimators require dense input, and missing values are not always represented the same way in sparse matrices. Converting a large sparse matrix to dense form can consume significant memory, so evaluate whether that cost is justified.

Performance and Maintainability Considerations

Imputation is computed once during fit and applied as a simple replacement during transform, so the runtime cost is generally small relative to model training. The main performance concern is data format: if you need to densify a large sparse matrix, the memory cost can dominate the preprocessing budget.

From a maintainability perspective, keeping imputation inside the pipeline means the preprocessing logic travels with the model. When the model is serialized with joblib or pickle, the imputer statistics are saved alongside it, so predictions on new data use the same transformation that was applied during training. This is especially important in production systems where the model is retrained on a schedule and the imputation logic must stay consistent across versions.

One operational detail worth noting: SimpleImputer stores the learned statistics in the statistics_ attribute after fitting. Inspecting this attribute is a useful way to verify that the imputer learned what you expect, especially when debugging a pipeline that behaves unexpectedly on new data. For example, after fitting a median imputer, imputer.statistics_ should contain the median of each column. If the values look wrong, the column selection in your ColumnTransformer is likely misconfigured.