Support Vectors
When an SVM draws a boundary between classes, it doesn’t “listen” equally to every training point. A small set of especially informative points end up doing most of the work—those are the support vectors.
What support vectors are
In a Support Vector Machine (SVM), the model chooses a separating hyperplane that maximizes the margin—the gap between the boundary and the closest points from each class. The training examples that lie closest to that boundary (or even on the wrong side when the data isn’t perfectly separable) are the support vectors. They “support” the solution because the position of the optimal hyperplane is determined by them; points far from the boundary don’t affect it.
Why they control the decision boundary
Mathematically, SVM training assigns a weight (a Lagrange multiplier) to each training point. Only points with non-zero weight become support vectors, and the decision function is built from them. With a kernel (like RBF), the model still depends only on support vectors—just measured through kernel similarities rather than raw coordinates.
Practical intuition and examples
- Spam detection: borderline emails (almost-spam vs. almost-ham) become support vectors; obvious spam contributes little.
- Credit scoring: applicants near the approve/deny threshold drive the boundary; clearly safe or risky cases matter less.
- SVR regression: in Support Vector Regression, support vectors are points outside the epsilon-insensitive tube (or on its edge) that shape the fitted function.
This matters because prediction time and model size scale with the number of support vectors. In scikit-learn’s SVC, you can inspect them via support_ and support_vectors_; many support vectors can mean a slower, more memory-hungry model.
Support vectors are the training examples that lie on or closest to an SVM’s decision boundary (or, in SVR, on the edges of the epsilon-insensitive tube) and therefore have nonzero influence on the learned solution. They effectively define the margin and the separating hyperplane through their associated Lagrange multipliers. This matters because an SVM’s predictions and generalization hinge on these points; removing or perturbing them can change the boundary, while most other samples have no effect.
Imagine you’re trying to draw a line on the floor to separate cats from dogs based on their size and weight. Most animals are clearly on one side or the other, but a few are right near where the line should go. Those “borderline” examples are the ones that really matter for placing the line.
In a Support Vector Machine, those key borderline training examples are called support vectors. They’re the data points closest to the dividing boundary, and they “support” (anchor) where that boundary ends up. Change a support vector, and the boundary can shift; change far-away points, and it often won’t.