Foundations

Every statistical idea rests on a handful of basic concepts about data and the things we measure. This section establishes that shared vocabulary — the core entities of a dataset, then the ways data is classified by type and by how it is organised.

Statistics begins with a Population, the entire group we want to understand, and a Sample, the manageable subset we actually observe. A numerical fact about the whole population is a Parameter, while the corresponding quantity computed from a sample is a Statistic — the sample-based estimate of the unknown parameter. Anything that can differ between individuals is a Variable, also called an Attribute.

The data itself is organised around records and fields. A single record is a Data Point, equivalently an Observation, and the full collection of them is a Dataset. Each measured property of a record is a Feature, and in supervised settings the value we aim to predict is the Label.

Types of Data

Data comes in kinds that dictate which analyses are valid. The broadest split is between Qualitative Data, which describes qualities or categories, and Quantitative Data, which records numerical amounts. Quantitative data is further either Discrete Data, taking separate countable values, or Continuous Data, taking any value in a range. Qualitative data appears as Categorical Data, which divides into Nominal Data, whose categories have no inherent order, and Ordinal Data, whose categories follow a meaningful ranking.

Types of Datasets

Datasets are also classified by their structure and how they arrive. Structured Data fits neatly into tables, whereas Unstructured Data — text, images, audio — does not. Several types are defined by their relationship to time: Time Series Data tracks one subject across many time points, Cross-Sectional Data captures many subjects at a single moment, and Panel Data combines both by following many subjects over time. By delivery, Streaming Data flows in continuously and must be processed on the fly, while Batch Data is collected and processed in fixed chunks.

Descriptive Statistics

Descriptive statistics summarise data so its main features can be grasped at a glance, before any modelling begins. The starting point is Exploratory Data Analysis (EDA), the open-ended practice of examining a dataset to understand its structure, spot patterns, and catch problems. This section organises the descriptive toolkit into measures of a distribution's centre, of its spread, of how variables relate, and the plots that reveal a distribution's full shape.

Measures of Central Tendency

These measures answer "what is a typical value?" The Mean is the arithmetic average, the Median is the middle value when data is sorted and resists outliers, and the Mode is the most frequent value. Variations suit special cases: the Weighted Mean lets some values count more than others, the Geometric Mean suits multiplicative quantities like growth rates, and the Harmonic Mean suits rates and ratios.

Measures of Dispersion

Dispersion measures describe how spread out the data is. The Variance averages squared deviations from the mean, and its square root, the Standard Deviation, restores the original units. Simpler summaries include the Range, the gap between largest and smallest, and the Interquartile Range (IQR), the spread of the middle half. The Coefficient of Variation expresses spread relative to the mean for comparing differently scaled data. Two further measures describe a distribution's shape: Skewness captures asymmetry, and Kurtosis captures how heavy its tails are.

Covariance and Correlation

These measures describe how two variables move together. Covariance indicates the direction of their joint variation, and the Covariance Matrix collects it for many variables at once. Because covariance depends on scale, the Correlation Coefficient standardises it to a value between minus one and one, with the Correlation Matrix tabulating all pairwise correlations. The common form is Pearson Correlation, which measures linear association, while Spearman Rank Correlation and Kendall Tau measure monotonic association using ranks and so resist outliers. Extending the idea to multivariate distance, Mahalanobis Distance measures how far a point lies from a distribution's centre while accounting for correlations.

Data Distribution and Visualisation

Finally, several tools reveal a distribution's full shape. A Quantile marks a value below which a given fraction of data falls, and a Percentile is the same idea on a hundred-point scale. The Empirical Distribution Function traces the observed cumulative distribution directly from the data, and a Q-Q Plot (Quantile-Quantile Plot) compares it against a theoretical one to check fit. For seeing the density, a Density Plot draws a smooth curve, often via Kernel Density Estimation (KDE), while a Histogram bins values into bars. A Box Plot compactly shows the median, quartiles, and outliers, and Error Bars display uncertainty around plotted estimates.

Probability Theory

Probability theory is the mathematics of uncertainty, and it underpins nearly all of statistics and machine learning. This section builds the ideas from the ground up — starting with the basic language of chance, then the rules for combining probabilities, and finally the notion of a random variable that turns outcomes into numbers we can reason about.

It all starts with an Experiment, any process with an uncertain result. Each possible result is an Outcome, and the set of all of them is the Sample Space. An Event is a collection of outcomes we care about, and Probability assigns each event a number between zero and one measuring how likely it is. Two events are Independent Events when one occurring tells us nothing about the other, and Mutually Exclusive Events when they cannot both happen at once.

Probability Rules

A few rules let us compute the probability of complex events from simpler ones. The whole edifice rests on the Axioms of Probability, the basic consistency requirements every probability must obey. From a joint situation, the Marginal Probability isolates the chance of a single event ignoring the others, while Conditional Probability measures the chance of one event given that another has occurred. The Joint Probability captures the chance of two events happening together.

Two results tie these together. The Law of Total Probability builds an event's overall probability by summing across a set of mutually exclusive cases, and Bayes Theorem reverses a conditional probability, letting us update beliefs in light of evidence — the cornerstone of much of modern inference.

Random Variables

To do quantitative work we attach numbers to outcomes. A Random Variable is a function that does exactly that, turning chance results into numerical values. Its average value over many trials is its Expectation, and its spread around that average is the Variance (of Random Variable). A useful guarantee, Chebyshev's Inequality, bounds how much a random variable can stray from its mean using only the variance, regardless of the underlying distribution.

Probability Distributions

A probability distribution describes how likely each possible value of a random variable is. This section introduces the general idea and the functions used to describe distributions, then surveys the common families split into discrete and continuous types.

At its core, a Probability Distribution is the full specification of how probability is spread across outcomes. For discrete values it is given by a Probability Mass Function (PMF), which assigns a probability to each individual value. For continuous values it is given by a Probability Density Function (PDF), where probability corresponds to area under the curve. Either way, the Cumulative Distribution Function (CDF) accumulates probability up to a given value, answering "how likely is an outcome at most this large?"

Discrete Distributions

Discrete distributions describe counts and yes/no outcomes. The Bernoulli Distribution models a single trial with two outcomes, such as one coin flip. Repeating such trials and counting successes gives the Binomial Distribution. When events occur independently at a steady average rate over time or space, the number of them follows the Poisson Distribution.

Continuous Distributions

Continuous distributions describe measurements that can take any value in a range. The most important is the Normal Distribution, the familiar bell curve that arises naturally whenever many small effects add up. The Uniform Distribution spreads probability evenly across an interval, representing complete indifference between values. The Exponential Distribution models the waiting time until an event occurs, the continuous companion to the Poisson process.

Sampling and Experimental Design

Good conclusions depend on good data, and good data depends on how it was collected. This section covers the foundations of gathering representative samples and designing experiments that support sound causal claims — starting with the basic idea of sampling, then the techniques for drawing samples, and finally the principles of experimental design.

The core activity is Sampling, selecting a subset of a population to study because measuring everyone is impractical. Two dangers haunt it. Sampling Error is the natural variability that arises because a sample is only part of the whole, and it shrinks as samples grow. More insidious is Sampling Bias, a systematic skew from a flawed selection process that no amount of extra data will fix.

Sampling Techniques

Several methods aim to make a sample representative. Random Sampling gives every member an equal chance of selection, the gold standard for avoiding bias. Systematic Sampling picks every nth member from an ordered list, simpler in practice but vulnerable to hidden patterns. Stratified Sampling divides the population into meaningful subgroups and samples each in proportion, ensuring all are represented, while Cluster Sampling instead samples whole naturally occurring groups, trading some precision for practicality.

Experimental Design

To establish cause rather than mere correlation, experiments compare outcomes under different conditions. A Control Group receives no intervention and provides a baseline, against which a Treatment Group that does receive the intervention is compared. Randomization assigns participants to groups by chance, balancing out unknown differences so the groups are comparable. This guards against a Confounding Variable — a hidden factor that influences both the treatment and the outcome and can fake a causal link. A common practical application of these ideas is A/B Testing, which compares two versions of something by randomly showing each to different users.

Inferential Statistics

Inferential statistics uses a sample to draw conclusions about a larger population, quantifying how much confidence those conclusions deserve. This section starts with the theoretical results that make inference possible, then covers estimating unknown quantities, testing hypotheses, distribution-free methods, and the Bayesian approach to updating beliefs.

The bridge from sample to population is the Sampling Distribution, which describes how a statistic varies from sample to sample. Two theorems give it shape: the Central Limit Theorem says that averages tend toward a normal distribution as samples grow, regardless of the original shape, and the Law of Large Numbers says sample averages converge to the true value with more data. Together they let us judge Statistical Significance — whether an observed effect is more than chance.

Estimation and Confidence Intervals

Estimation infers unknown population values from a sample. Point Estimation produces a single best guess, often found by Maximum Likelihood Estimation (MLE), which picks the value making the observed data most probable. Because a single number hides uncertainty, Interval Estimation reports a range instead — a Confidence Interval — whose reliability is set by the chosen Confidence Level. The half-width of that interval is the Margin of Error.

Hypothesis Testing

Hypothesis testing weighs evidence for a claim. It pits a Null Hypothesis of "no effect" against an Alternative Hypothesis, then computes a p-value measuring how surprising the data would be if the null were true. Two mistakes are possible: a Type I Error falsely rejects a true null, and a Type II Error fails to detect a real effect; a test's ability to avoid the latter is its Statistical Power. The right test depends on the data: a t-test compares means in small samples, a Z-test does so when the population spread is known, a Chi-Square Test examines categorical associations, ANOVA (Analysis of Variance) compares means across several groups, and a Goodness-of-Fit Test checks whether data matches an expected distribution.

Non-Parametric Methods

When data violates the assumptions of standard tests, non-parametric methods step in, relying on ranks rather than specific distributions. The Wilcoxon Signed-Rank Test compares paired samples, the Mann-Whitney U Test compares two independent groups, and the Kruskal-Wallis Test extends that to several groups. The Kolmogorov-Smirnov Test compares whole distributions to check whether two samples come from the same one.

Bayesian Statistics

The Bayesian approach treats probability as degree of belief and updates it with evidence. It begins with a Prior Probability encoding what is believed before seeing data, combines it with the Likelihood Function describing how probable the data is under each hypothesis, and produces a Posterior Probability — the revised belief. From it, the Posterior Predictive Distribution forecasts future observations while carrying that remaining uncertainty forward.

Statistical Modeling

Statistical modelling fits mathematical relationships to data so we can explain and predict. This section introduces the workhorse models and the assumptions they rely on, then the principles for choosing a model that generalises and the diagnostics for checking that it fits.

The foundational model is Linear Regression, which fits a straight-line relationship between predictors and a continuous outcome. Its sibling Logistic Regression adapts the approach to predict probabilities for categorical outcomes. Both belong to the broader framework of Generalized Linear Models (GLM), which extends linear modelling to many kinds of response. Every such model rests on Model Assumptions that must hold for its conclusions to be trustworthy, and Residual Analysis — studying the leftover errors — is the main way to check them. Two common violations are Multicollinearity, where predictors are too correlated to separate their effects, and Heteroscedasticity, where the error spread changes across the range of predictions.

Model Selection and Generalization

A good model performs well on new data, not just the data it was fitted to. The central tension is the Bias-Variance Tradeoff: too simple a model suffers Underfitting, missing real structure, while too complex a model suffers Overfitting, memorising noise. The difficulty grows with the Curse of Dimensionality, as data becomes sparse when features are many. Regularization counters overfitting by penalising excessive complexity, nudging the model toward simpler, more general solutions.

Model Assessment

Diagnostics quantify how well a fitted model performs and whether its assumptions hold. The Mean Squared Error (MSE) measures average squared prediction error, and R-Squared (Coefficient of Determination) reports the fraction of variance the model explains. Several checks examine the residuals: Residual Plots reveal patterns a good model should not leave behind, Normality of Residuals tests a key assumption of inference, and Homoscedasticity confirms the error spread is constant. Targeted tests round this out: the Variance Inflation Factor (VIF) detects multicollinearity, and the Durbin-Watson Test checks for correlation between successive residuals.

Time Series Analysis

A time series is data recorded in sequence over time, where the order carries information that ordinary methods ignore. Time series analysis describes such data, makes it amenable to modelling, and forecasts its future. This section opens with the fundamental patterns and correlations, then moves through decomposition, the stationarity that models require, the models themselves, and practical forecasting methods.

Three patterns recur in time series: a Trend is a long-term drift up or down, Seasonality is regular repetition over a fixed period, and Stationarity is the convenient property of statistical behaviour that does not change over time. Correlations within a series are central tools: Autocorrelation measures how a series relates to its own past values, Partial Autocorrelation isolates the direct relationship at a given lag after removing intermediate ones, and Cross-Correlation measures how one series relates to another over time.

Decomposition and Components

Decomposition splits a series into interpretable parts: a Trend Component, a Seasonal Component, a longer-wave Cyclical Component, and the leftover random Irregular Component. How these combine defines the model: Additive Decomposition sums them when the seasonal swing is roughly constant, while Multiplicative Decomposition multiplies them when the swing grows with the level. The procedures that perform the split include Classical Decomposition and the more robust, flexible STL Decomposition.

Stationarity and Transformations

Most time series models assume stationarity, so it must be defined and achieved. Weak Stationarity requires only a constant mean, variance, and autocorrelation structure, whereas Strict Stationarity demands the entire distribution be unchanging. A series may be Trend Stationary, becoming stationary once a trend is removed, or Difference Stationary, becoming stationary after differencing. The Order of Integration counts how many differences are needed, and a Unit Root Process is the non-stationary case that differencing cures. The Augmented Dickey-Fuller (ADF) Test checks formally for such a root. Common remedies are Differencing to remove trends, and variance-stabilising transforms like the Log Transformation and the more general Box-Cox Transformation.

Time Series Models

A family of models captures temporal dependence. The Autoregressive (AR) Model predicts a value from its own past values, and the Moving Average (MA) Model predicts it from past forecast errors; combining them gives the ARMA Model. Adding differencing for non-stationary data yields the ARIMA Model, extended to seasonal data as the Seasonal ARIMA (SARIMA) Model and to include external predictors as the ARIMAX Model. For several interacting series, the Vector Autoregression (VAR) Model models them jointly. More general frameworks include the State Space Model and the Hidden Markov Model (HMM), which posits hidden states driving the observations. To test whether one series helps predict another, Granger Causality provides a formal check.

Forecasting Methods

Forecasting turns these models into predictions. The simplest baseline is Naive Forecasting, which just repeats the last value. Smoothing methods weight recent data more heavily: Simple Exponential Smoothing (SES) for level-only series, Holt's Linear Trend Method when a trend is present, and the Holt-Winters Method when both trend and seasonality appear. Model-based forecasting via ARIMA Forecasting projects an ARIMA model forward. Producing predictions several steps ahead is Multi-step Forecasting, evaluated reliably with a Rolling Forecast Origin that repeatedly retrains as time advances, and combining several forecasts gives Ensemble Forecasting for greater robustness.