Problem Types

Unsupervised learning covers a family of problems united by one feature: there are no labels. Instead of being told the right answer, the algorithm must discover structure in the data on its own. This section maps out the main kinds of problems that fall under that umbrella, giving a vocabulary for the rest of the field before any specific method is introduced.

The most familiar task is Clustering, which groups similar examples together so that natural categories emerge without anyone defining them in advance. A second major task is Dimensionality Reduction, which compresses data with many features into a compact form that keeps the essential information while discarding redundancy. Closely related, Manifold Learning assumes that high-dimensional data really lies on a much simpler curved surface and tries to uncover that hidden low-dimensional shape.

Other problems concern the probability distribution behind the data. Density Estimation models how likely different values are, building a picture of where data concentrates. Generative Modeling goes a step further and learns to produce brand-new samples that resemble the training data. Underpinning many of these is Latent Variable Modeling, which explains observations through hidden factors that were never directly measured, and the broader goal of Representation Learning, which seeks features that make data easier for downstream tasks to use.

Finally, two task types focus on specific patterns. Anomaly Detection identifies rare examples that differ markedly from the norm, and Association Rule Learning uncovers items that tend to occur together, such as products frequently bought in the same basket. Together these problem types frame everything that follows.

Clustering Algorithms

Clustering groups data so that similar points fall together and dissimilar points stay apart — all without labels. There is no single right way to do it, and the major algorithm families embody different ideas of what a "cluster" even is: a region around a centre, a nested hierarchy, a dense neighbourhood, a probability distribution, or a connected piece of a graph. This section walks through those families in turn.

Partitional Clustering

Partitional methods divide data into a fixed number of groups, usually organised around centres. This is the idea of Centroid-Based Clustering, where each cluster is represented by a central point. The defining algorithm is K-Means, which alternates between assigning points to the nearest centre and recomputing those centres. Its quality depends heavily on initialisation, which K-Means++ Initialization improves by spreading the starting centres apart. The objective it minimises is Inertia, the total squared distance of points to their centre, and the Elbow Method uses how inertia falls with more clusters to suggest a good count. Variants address weaknesses: K-Medoids uses actual data points as centres for robustness to outliers, and Fuzzy C-Means lets points belong partially to several clusters. Each cluster's representative point is its Cluster Centroid.

Hierarchical Clustering

Hierarchical methods build a tree of nested groups rather than a single flat partition. Agglomerative Clustering starts with every point alone and repeatedly merges the closest groups, while Divisive Clustering works top-down by splitting. The result is visualised as a Dendrogram, a tree showing how and at what distance clusters join. How distance between groups is measured is the Linkage Criterion, and the common choices each behave differently: Single Linkage uses the nearest pair, Complete Linkage the farthest, Average Linkage the mean distance, and Ward's Linkage the increase in variance from merging. The faithfulness of the tree to the original distances is measured by Cophenetic Correlation, and for large datasets BIRCH summarises data into a compact structure to cluster efficiently.

Density-Based Clustering

Density-based methods define clusters as dense regions separated by sparse ones, which lets them find arbitrarily shaped clusters and ignore noise. The classic is DBSCAN, which classifies each point by its local density: a Core Point has many neighbours, a Border Point lies at the edge of a dense region, and a Noise Point belongs to no cluster. "Many neighbours" is made precise by the Epsilon Neighborhood, the radius searched around a point, and the Minimum Points (MinPts) threshold needed within it. Clusters grow through Density-Reachability, chaining together points connected by dense neighbourhoods. Extensions improve on this: OPTICS handles clusters of varying density by ordering points rather than fixing one radius, HDBSCAN builds a hierarchy to extract stable clusters automatically, and Mean Shift finds clusters by climbing toward peaks of the data density.

Distribution-Based Clustering

Distribution-based methods assume the data was generated by a blend of probability distributions. This is the Mixture Model view, and its most common form is the Gaussian Mixture Model (GMM), which models data as a weighted sum of Gaussian bells. Each bell is a Component Distribution, and fitting them is done with the Expectation-Maximization (EM) Algorithm, which alternates between guessing cluster memberships and updating the distributions. Because memberships are probabilities, this gives a Soft Assignment in which points can belong partly to several clusters. Choosing how many components to use can be guided by the Bayesian Information Criterion (BIC), which rewards fit while penalising complexity.

Spectral And Graph-Based Clustering

Graph-based methods treat data points as nodes connected by similarity, then cut the graph into clusters. Spectral Clustering does this using the eigenvectors of a matrix derived from the connections, excelling on non-convex shapes. It starts from a Similarity Graph linking nearby points, encoded numerically as an Affinity Matrix. From it the Graph Laplacian is formed, whose spectrum reveals the cluster structure, and a good split corresponds to a Normalized Cut that balances separating groups against keeping each group internally connected. A related message-passing method, Affinity Propagation, finds clusters by exchanging preferences between points to elect representative exemplars.

Dimensionality Reduction

High-dimensional data is hard to visualise, store, and learn from, and much of it is redundant. Dimensionality reduction compresses many features into a few while preserving what matters. There are two broad approaches: linear methods that project data onto a flat subspace, and manifold methods that uncover curved low-dimensional structure. This section covers both.

Linear Dimensionality Reduction

Linear methods find new axes that are weighted combinations of the originals. The cornerstone is Principal Component Analysis (PCA), which finds the directions of greatest variance and projects onto them. Those directions are the Principal Components, and the Explained Variance Ratio reports how much information each one retains. A Scree Plot charts these values to help decide how many components to keep. Underlying PCA is the Singular Value Decomposition (SVD), a matrix factorisation that exposes the principal directions, with Truncated SVD keeping only the top few for efficiency, especially on sparse data.

Other linear methods optimise different criteria. Independent Component Analysis (ICA) separates a signal into statistically independent sources rather than merely uncorrelated ones. Non-Negative Matrix Factorization (NMF) decomposes data into parts that only add, never subtract, giving interpretable components. Factor Analysis models observed features as arising from a few hidden causes, each such cause being a Latent Factor.

Manifold Learning

Manifold methods assume data lies on a curved surface embedded in the high-dimensional space, and try to flatten it out. The motivation is the Curse of Dimensionality, the way distances and intuition break down as dimensions grow, together with the Manifold Hypothesis that real data occupies far fewer effective dimensions than it appears to. That true count is the Intrinsic Dimensionality.

Several techniques recover this hidden structure, often for visualisation. t-Distributed Stochastic Neighbor Embedding (t-SNE) places similar points close together in two or three dimensions, with a setting called Perplexity controlling how much local versus global structure it preserves. Uniform Manifold Approximation and Projection (UMAP) achieves similar visual results faster while better retaining global layout. Classic approaches include Locally Linear Embedding (LLE), which preserves each point's local linear relationships, and Isomap, which preserves Geodesic Distance — distance measured along the manifold's surface rather than straight through space. Multidimensional Scaling (MDS) arranges points to preserve pairwise distances, Kernel PCA extends PCA to nonlinear structure via a kernel, and Laplacian Eigenmaps uses a graph of neighbours to find a structure-preserving low-dimensional embedding.

Density Estimation

Density estimation asks a deceptively simple question: given some data, how likely is each possible value? Answering it builds a model of where data concentrates and where it thins out, which underlies generation, anomaly detection, and much of statistics. This section introduces the idea and the two broad strategies for carrying it out.

The object we want to estimate is the Probability Density Function (PDF), a curve describing the relative likelihood of each value. There are two philosophies for fitting one. Parametric Density Estimation assumes the data follows a known family of distributions and fits only its few parameters, most often by Maximum Likelihood Estimation (MLE), which chooses the parameters making the observed data most probable. This is efficient when the assumed form is right and misleading when it is wrong.

The alternative makes no such assumption. Non-Parametric Density Estimation lets the data shape the curve directly. The leading method is Kernel Density Estimation (KDE), which places a small smooth bump at every data point and adds them up; how wide those bumps are is controlled by Bandwidth Selection, the key tuning choice that trades smoothness against detail. Simpler still is Histogram Density Estimation, which counts how many points fall into each bin, while Nearest-Neighbor Density Estimation infers density from how close a point's neighbours are — dense regions have neighbours nearby, sparse regions far away.

Anomaly Detection

Anomaly detection finds the data points that do not belong — the rare, surprising, or suspicious cases hiding among the ordinary ones. It matters for fraud, fault monitoring, and data cleaning, and because anomalies are by definition scarce, it is usually approached without labels. This section introduces the vocabulary of anomalies and then the methods used to score and flag them.

The basic object is an Outlier, a point that deviates markedly from the rest. When the goal is specifically to spot genuinely new patterns not seen in training, the task is called Novelty Detection. Anomalies come in kinds: a Point Anomaly is a single value that is odd on its own, whereas a Contextual Anomaly is only unusual given its context, such as a temperature normal in summer but extreme in winter. Methods typically assign each point an Anomaly Score reflecting how unusual it is, and the assumed Contamination Rate — the expected fraction of anomalies — sets the threshold for flagging them.

Several simple statistical methods catch outliers in one dimension. The Z-Score Method flags values far from the mean in units of standard deviation, and the Interquartile Range (IQR) Method flags values lying well outside the middle half of the data. These are fast and interpretable but limited to relatively simple distributions.

Richer methods handle complex, high-dimensional data. The Local Outlier Factor (LOF) compares a point's local density to that of its neighbours, catching anomalies that are unusual only relative to their surroundings. The Isolation Forest isolates points by random splitting, exploiting the fact that anomalies are easier to separate and so isolated in fewer steps. The One-Class SVM (OC-SVM) learns a boundary around the normal data and treats anything outside it as anomalous. Finally, Autoencoder-Based Anomaly Detection trains a neural network to reconstruct normal data and flags points it reconstructs poorly, since the model has never learned to represent them.

Association Rule Learning

Association rule learning discovers items that tend to appear together. Its classic application gives the field its flavour: analysing shopping data to find that customers who buy one product often buy another. This section introduces the core task, the measures that decide which patterns are meaningful, and the algorithms that find them efficiently.

The motivating problem is Market Basket Analysis, the study of which products co-occur in transactions. Its building block is the Frequent Itemset — a group of items that appears together often enough to be worth noting.

To judge whether a co-occurrence is genuinely interesting rather than coincidental, several measures are used. Support reports how frequently an itemset appears in all transactions, and Confidence measures how often one item follows another when the rule's condition holds. Because common items can inflate confidence, Lift corrects for baseline popularity to show whether two items truly attract each other, and Conviction gauges how strongly a rule's conclusion depends on its premise.

Finding frequent itemsets in large data needs efficient algorithms. The Apriori Algorithm prunes the search by noting that any superset of an infrequent set is also infrequent, while the FP-Growth Algorithm compresses the data into a tree to avoid repeated scanning, and the Eclat Algorithm uses a vertical data layout for fast intersection counting. Extending the idea to ordered events, Sequential Pattern Mining discovers patterns that unfold over time, such as purchases made in a typical sequence.

Representation Learning

Representation learning seeks better ways to encode data — compact, meaningful features that capture what matters and make downstream tasks easier. Much of modern unsupervised learning lives here, especially the neural models that learn to compress and generate data. This section moves from the basic idea of an encoding through autoencoders to probabilistic and generative models.

The central concept is the Embedding, a learned vector that places each example in a meaningful space where distance reflects similarity. The space those vectors live in is the Latent Space, a compressed coordinate system capturing the data's underlying factors.

The workhorse model for learning such codes is the Autoencoder, a network that compresses input to a small code and reconstructs it, trained to minimise a Reconstruction Loss that measures how faithfully the output matches the input. Several variants shape what the code captures. An Undercomplete Autoencoder forces compression by making the code smaller than the input, a Sparse Autoencoder instead constrains most code units to stay inactive, and a Denoising Autoencoder learns robust features by reconstructing clean data from a corrupted version.

A probabilistic turn makes these models generative. The Variational Autoencoder (VAE) learns a smooth, structured latent space by treating codes as distributions, optimising a bound on the data likelihood called the Evidence Lower Bound (ELBO). Encouraging each latent dimension to control a single independent factor yields a Disentangled Representation, and the Beta-VAE (β-VAE) tunes a weight in the objective to strengthen exactly that property.

A different generative approach pits two networks against each other. The Generative Adversarial Network (GAN) trains a generator to fool a discriminator into accepting its samples as real, producing strikingly realistic data — though training can suffer from Mode Collapse, where the generator produces only a narrow variety of outputs instead of the full diversity of the data.

Model Development

Building an unsupervised model involves more than choosing an algorithm. The data must be prepared, a notion of similarity must be chosen, and the whole pipeline must scale to large datasets. This section gathers those practical foundations: preprocessing the data, measuring distance between examples, and keeping computation efficient.

Data Preprocessing

Raw data is rarely ready to use. Missing Value Imputation fills in gaps so records are complete, and Outlier Treatment handles extreme values that could distort results. Because most unsupervised methods rely on distances, putting features on a comparable scale is critical: Feature Scaling is the general practice, achieved through Standardization, which rescales features to zero mean and unit variance, or Normalization, which squeezes values into a fixed range. Categorical fields are made numeric with One-Hot Encoding, and Feature Engineering crafts more informative inputs from the raw ones. Whitening goes further by removing correlations between features so each contributes independently. Before clustering at all, Cluster Tendency Assessment checks whether the data even contains meaningful groups.

Distance And Similarity Measures

Unsupervised learning lives or dies by how it measures resemblance. A Distance Metric is the general notion of how far apart two points are. The most familiar is Euclidean Distance, straight-line distance, while Manhattan Distance sums coordinate-wise differences. For direction rather than magnitude, Cosine Similarity compares the angle between vectors, and for sets, Jaccard Similarity measures overlap. Mahalanobis Distance accounts for feature correlations and scale, and Correlation Distance compares the shape of patterns rather than their level. For sequences, Dynamic Time Warping (DTW) aligns series that vary in speed.

Comparing probability distributions calls for its own measures. Bregman Divergence is a general family of such measures, of which the best known is Kullback-Leibler (KL) Divergence, quantifying how one distribution differs from another. Wasserstein Distance measures the cost of transforming one distribution into another, and Kernel Similarity implicitly compares points in a richer transformed space.

Scalability And Efficiency

Real datasets are large, so methods must scale. Approximate Nearest Neighbor Search finds close points quickly by accepting tiny inaccuracies, often using Locality-Sensitive Hashing (LSH), which hashes similar points into the same buckets. Mini-Batch Processing updates models from small chunks of data at a time, and Online Clustering processes a continuous stream without storing it all. Random Projections reduce dimensionality cheaply by projecting onto random directions while approximately preserving distances.

Evaluation

Without labels, judging an unsupervised model is subtle: there is often no single right answer to compare against. Evaluation therefore splits into measures that rely only on the data's own structure, measures that compare results to a known reference, and measures specific to dimensionality reduction. This section covers all three.

Internal Clustering Evaluation

Internal measures assess a clustering using only the data itself, with no external truth — this is Internal Validation. They generally reward clusters that are compact and well separated. The Silhouette Score compares how close each point is to its own cluster versus the nearest other one. The Davies-Bouldin Index measures the average similarity between each cluster and its most similar neighbour, with lower being better, while the Calinski-Harabasz Index takes the ratio of between-cluster to within-cluster spread. The Dunn Index contrasts the smallest gap between clusters with the largest cluster width, and the Gap Statistic compares the clustering's compactness to what random data would produce, helping choose the number of clusters.

External Clustering Evaluation

External measures compare a clustering to known ground-truth labels — External Validation. The Rand Index counts the fraction of point pairs that are grouped consistently between the two labellings, and the Adjusted Rand Index (ARI) corrects it for agreement expected by chance. Information-theoretic measures include Normalized Mutual Information (NMI), capturing how much knowing one labelling tells you about the other, and the Fowlkes-Mallows Index (FMI), a balance of pair-based precision and recall. Cluster Purity reports how dominated each cluster is by a single true class. Two complementary properties, Homogeneity — each cluster containing only one class — and Completeness — each class confined to one cluster — are combined into the V-Measure, their harmonic mean.

Dimensionality Reduction Evaluation

Reduction methods are judged on how faithfully the low-dimensional version preserves the original. Reconstruction Error measures how well the compressed data can rebuild the original. Two neighbourhood-based measures guard against distortion: Trustworthiness penalises points that appear close in the projection but were far apart originally, while Continuity penalises the opposite — neighbours in the original that get pushed apart. Ultimately, Downstream Task Evaluation tests the reduced representation by how well it serves a later task, the most practical verdict of all.