Variance Inflation Factor (VIF)

When you build a regression model, it’s tempting to throw in every useful-looking feature. The catch is that some features overlap so much that the model can’t clearly tell which one deserves the credit—this is where VIF comes in.

What VIF measures

The Variance Inflation Factor (VIF) is a diagnostic for multicollinearity in linear regression. It quantifies how much the uncertainty (variance) of a coefficient estimate is “inflated” because a predictor is strongly linearly related to other predictors. For predictor X_j, VIF is:

VIF_j = 1 / (1 - R_j^2)

where R_j² comes from regressing X_j on all the other predictors. If X_j is well-explained by the others, R_j² is high, and VIF grows large.

How to interpret it (practically)

VIF = 1: no linear overlap with other predictors.
Higher VIF: coefficient becomes unstable; standard errors grow; p-values can look worse even if the feature is genuinely useful.
Common rules of thumb: > 5 (concerning) or > 10 (serious), but context matters.

Everyday example

Predicting house prices with both “square footage” and “number of bedrooms” often creates multicollinearity: bigger homes usually have more bedrooms. VIF helps reveal that the model may struggle to separate their individual effects, even if predictions stay decent.

Why it matters in AI/ML workflows

VIF is especially important when you care about interpretability (which features matter and by how much). High VIF can make linear models, GLMs, and even linear parts of pipelines (e.g., after one-hot encoding) hard to trust. Typical fixes include dropping redundant features, combining them, or using regularization (ridge/elastic net), which stabilizes coefficients.

Variance Inflation Factor (VIF) quantifies how much the variance of a regression coefficient is increased due to multicollinearity among predictors. For predictor j, VIF = 1/(1 − R²j), where R²j comes from regressing that predictor on all others. High VIFs indicate unstable, hard-to-interpret coefficients and inflated standard errors, affecting inference and feature importance in ML pipelines. Example: in linear regression, VIF > 5–10 often flags redundant features.

Imagine you’re trying to figure out how much each ingredient affects the taste of a soup, but two ingredients are almost always added together. It becomes hard to tell which one is really responsible for the flavor change.

That’s what Variance Inflation Factor (VIF) checks in a regression model. When two or more input features are strongly related to each other, the model struggles to separate their individual effects. VIF measures how much this overlap “inflates” the uncertainty (the wobbliness) of a feature’s estimated impact. A low VIF suggests the feature is fairly independent; a high VIF warns that the feature’s effect may be unreliable.