Notes
🔍

Distance Metric

When a model says two customers are “similar,” it needs a concrete way to measure that similarity. A distance metric is the rule that turns “how different are these two rows of features?” into a number the model can compare.

What a distance metric is

A distance metric is a function that takes two feature vectors (for example, two applicants described by income, age, and debt ratio) and returns a non-negative value where smaller means “closer.” In the strict mathematical sense, a metric satisfies: distance is 0 only for identical points, it’s symmetric, and it obeys the triangle inequality. In practice, people also use distance-like measures (e.g., cosine distance) because they behave sensibly for the task.

Why it matters for k-NN

In k-nearest neighbors, the distance metric defines what “nearest” means, which directly changes predictions. Common choices include:

  • Euclidean distance (straight-line): natural for continuous, similarly scaled features.
  • Manhattan distance (city-block): can be more robust when many features add small effects.
  • Cosine distance: focuses on direction, popular for text features (TF-IDF) where magnitude is less meaningful.
  • Hamming distance: for binary/categorical indicators (e.g., one-hot encoded fields).

Practical pitfalls and examples

Distance is only as good as the feature space. If “annual income” ranges in the tens of thousands while “number of late payments” ranges 0–5, Euclidean distance will be dominated by income unless you scale features (e.g., standardization). In spam detection with TF-IDF vectors, cosine distance often beats Euclidean because it compares word usage patterns rather than email length. In scikit-learn’s KNeighborsClassifier, you’ll see this as the metric parameter.

A distance metric is a function that assigns a nonnegative number to a pair of feature vectors to quantify how far apart they are, typically satisfying identity, symmetry, and the triangle inequality (e.g., Euclidean distance, Manhattan distance). It matters because supervised methods that rely on similarity—especially k-nearest neighbors (k-NN)—use it to define neighborhoods; a poor metric distorts “closeness” and directly degrades predictions.

Think of a map app choosing the “closest” coffee shop. It needs a clear rule for what “closest” means: straight-line distance, walking time, or driving time. A distance metric is that kind of rule, but for data points instead of places.

In supervised learning, each example (like an email or a patient record) can be described by a list of measurements. A distance metric turns “how different are these two examples?” into a single number. Methods like k-nearest neighbors use that number to find the most similar past cases and make a prediction based on them.