Notes
🔍

Cosine Similarity

When you represent an example as a long list of numbers (a feature vector), you need a way to say whether two examples “point in the same direction.” Cosine similarity does exactly that, and it’s especially handy when the overall size of the numbers matters less than their pattern.

What cosine similarity measures

Cosine similarity compares two vectors by the cosine of the angle between them. If two vectors point in the same direction, the angle is small and the similarity is close to 1. If they’re orthogonal (unrelated directions), it’s near 0. If they point in opposite directions, it becomes negative. The formula is:

cos_sim(x, y) = (x · y) / (||x|| * ||y||)

This “dot product divided by lengths” normalization means cosine similarity ignores magnitude. Doubling every feature in x doesn’t change its similarity to y, because the direction stays the same.

Why it matters in supervised learning (especially k-NN)

In k-Nearest Neighbors, the “nearest” points depend entirely on your metric. Cosine similarity is a strong choice when:

  • Features are high-dimensional and sparse (common in text).
  • You care about relative proportions rather than absolute counts.
  • Different examples have very different total magnitudes (e.g., short vs. long documents).

If you used Euclidean distance on raw word counts for spam detection, longer emails can look “farther” just because they contain more words; cosine similarity reduces that length bias.

Practical examples and tools

  • Spam detection: compare TF-IDF vectors of emails; neighbors with similar vocabulary patterns vote on spam/ham.
  • Customer churn: compare normalized behavior profiles (fractions of actions) rather than total activity volume.

In scikit-learn you’ll see this via sklearn.metrics.pairwise.cosine_similarity or by setting k-NN to use the cosine distance (often implemented as 1 - cosine_similarity).

Cosine similarity measures how aligned two feature vectors are by computing the cosine of the angle between them, typically as the normalized dot product, yielding values from −1 to 1 (or 0 to 1 for nonnegative features). It captures similarity in direction rather than magnitude, making it useful when vector length is not meaningful (e.g., text TF-IDF vectors). In supervised learning, it directly affects neighbor selection and predictions in distance-based models like k-NN.

Imagine you and a friend both like music, but you listen to 10 times more songs. If we only compare totals, you look “more musical.” Cosine Similarity instead compares the pattern of your tastes: do you like the same kinds of artists, in similar proportions? It measures how much two lists of numbers point in the same direction, not how big they are.

In AI, data often looks like long checklists (words in an email, items in a shopping cart, features of a patient). Cosine Similarity helps decide which examples are most alike, which is useful for things like finding similar documents, spotting spam, or choosing the “nearest neighbors” for a prediction.