Notes
🔍

Likelihood Function

The likelihood function is a way to ask a very practical question: “Given the data I observed, which model settings make that data look most plausible?” It’s the engine that connects what you see in the world to the parameters you’re trying to learn.

What it is (and what it is not)
A likelihood function takes a statistical model and treats the parameters as the “input,” while the observed data are held fixed. If your model says the probability of data D given parameters θ is P(D | θ), then the likelihood is that same expression, just viewed as a function of θ: L(θ) = P(D | θ). It is not a probability distribution over θ by itself (it doesn’t have to sum/integrate to 1 over parameters).

Intuition with a simple example
Imagine exam scores are modeled as Normal with unknown mean μ and known spread. After seeing your class’s scores, different values of μ make the observed scores more or less plausible. The likelihood is like a “scorecard” over possible μ values: higher likelihood means “this μ would more naturally produce what we saw.”

Why it matters in Bayesian inference
Bayesian learning updates beliefs using: posterior ∝ likelihood × prior. The prior encodes what you believed before data; the likelihood encodes what the data say. Without the likelihood, data can’t meaningfully pull your beliefs toward parameter values that explain observations.

Where it shows up in ML practice

  • Maximum Likelihood Estimation (MLE): choose parameters that maximize likelihood.
  • Negative log-likelihood becomes common loss functions (e.g., cross-entropy for classification).
  • Probabilistic models in libraries like PyTorch/TF (and PyMC/Stan) compute likelihoods to fit models and quantify uncertainty.

The Likelihood Function treats observed data as fixed and expresses how plausible different parameter values are under a statistical model, typically written L(θ)=p(data|θ). It underpins maximum likelihood estimation and, in Bayesian inference, combines with a prior to form the posterior via Bayes’ rule. Example: for coin flips with k heads in n trials, L(p)=p^k(1−p)^(n−k) ranks candidate biases p.

Imagine you’re a detective trying to figure out which suspect best explains the clues. For each suspect, you ask: “If this person did it, how likely is it that I’d see these exact clues?” That “how well does this explanation fit what I saw?” idea is the likelihood function.

In statistics and AI, the “suspects” are different settings of a model (often called parameters), and the “clues” are your observed data. The likelihood function doesn’t tell you how probable the parameters are by themselves; it tells you how strongly each possible parameter setting would produce the data you actually observed.