Distance-Weighted k-NN
Imagine asking your neighbors for advice: the person next door probably matters more than someone three blocks away. Distance-weighted k-NN bakes that common sense into k-nearest neighbors by letting closer training examples “vote” more strongly than farther ones.
How it works
Standard k-NN finds the k closest training points to a new input using a distance like Euclidean distance. In the distance-weighted version, each neighbor gets a weight that shrinks with distance—commonly 1 / (d + ε) or 1 / d², where d is the distance and ε prevents division by zero. Predictions then use these weights:
- Classification: sum the weights per class and pick the class with the largest total weight.
- Regression: take a weighted average of neighbors’ target values.
Why it matters
Plain k-NN treats the 1st-closest and k-th-closest neighbors equally, which can blur decisions near boundaries or in unevenly spaced data. Distance weighting reduces the influence of “barely-neighbors,” often improving accuracy when the local neighborhood is informative. It also makes the choice of k a bit less brittle, because far points contribute less anyway. The flip side: if your features aren’t scaled, the distance (and therefore the weights) can be misleading—so standardization is especially important.
Practical examples and where you’ll see it
- Credit risk: closer historical borrowers (similar income, debt, history) influence the decision more than marginally similar ones.
- House prices: nearby comparable homes dominate the estimate; farther comps matter less.
- In scikit-learn:
from sklearn.neighbors import KNeighborsClassifier model = KNeighborsClassifier(n_neighbors=15, weights="distance")
Distance-Weighted k-NN is a variant of k-nearest neighbors (k-NN) that combines the labels of the k closest training points using weights that decrease with distance (e.g., inverse-distance), so nearer neighbors influence the prediction more than farther ones. It matters because it reduces errors when neighbor distances vary, improving robustness to local density differences and helping prevent distant points from dominating classification votes or regression averages.
Imagine you’re asking a few neighbors which restaurant to try. If one neighbor lives next door and another lives across town, you’d probably trust the closer neighbor more. Distance-Weighted k-NN works the same way in AI.
Regular k-NN predicts something (like “spam” vs “not spam,” or a house price) by looking at the k nearest past examples and letting them “vote.” In Distance-Weighted k-NN, the closest examples get a louder vote than the farther ones. This often helps when nearby examples are more relevant and distant ones might be misleading.