Notes
🔍

Cost Function

Training a supervised model is basically a game of “make the predictions look more like the labels.” A cost function is the scorekeeper: it turns “how wrong are we?” into a single number the training algorithm can try to push down.

What a cost function is doing

A cost function (closely related to a loss function) measures the mismatch between a model’s predictions and the true targets. For one example, you compute a per-example loss; then you aggregate across the dataset (commonly an average) to get the cost. Training becomes an optimization problem: find model parameters that minimize this cost. Many cost functions are designed to be smooth enough that gradients exist, enabling efficient updates with gradient descent and its variants.

Common choices in supervised learning

  • Mean Squared Error (MSE) for regression (e.g., house price prediction): large errors get penalized heavily.
  • Mean Absolute Error (MAE) for regression when you want robustness to outliers.
  • Cross-entropy loss for classification (e.g., spam detection): strongly penalizes confident wrong predictions and aligns with probabilistic outputs.
  • Regularized cost adds a penalty like L2 (weight decay) to discourage overly complex models and reduce overfitting.

Why it matters in training and model behavior

The cost function is the model’s “objective,” so it directly shapes what the model learns. If you pick MSE for a problem with extreme outliers (like fraud amounts), the model can get pulled toward rare huge errors; MAE or Huber loss can behave better. If classes are imbalanced (like churn), you may weight the cost so mistakes on the minority class matter more. In libraries, you’ll see this as `loss` in scikit-learn’s `SGDClassifier` or `objective`/`eval_metric` in XGBoost.

A cost function (or loss function) maps a model’s predictions and the true labels to a single numeric penalty that quantifies error on a dataset, typically aggregated over training examples. It defines the objective minimized during training (e.g., mean squared error for regression, cross-entropy for classification). It matters because the choice of cost function determines what “good” performance means, drives gradient-based optimization, and directly shapes convergence and generalization.

Think of training an AI model like practicing free throws in basketball. You take a shot, see how far you missed, and that “miss amount” tells you what to fix next time. A cost function is that “miss amount,” but for predictions. It’s a single score that says how bad the model’s answers are compared to the correct answers in the training data.

For example, in house-price prediction, it measures how far off the predicted prices are from real sale prices. In spam filtering, it penalizes marking real emails as spam (and vice versa). Training tries to make this score as small as possible.