AI Foundations

Unsupervised Learning

Unsupervised learning is a branch of machine learning that finds structure in unlabeled data without any target labels, using tasks such as clustering, dimensionality reduction, and density estimation to discover patterns the data already contains.

What Unsupervised Learning Means

Unsupervised learning is the family of machine learning methods that operate on data with no target labels. Where supervised learning is given input-output pairs and asked to predict the output for new inputs, unsupervised learning receives only the inputs and is asked to describe their structure: which points group together, which directions carry the most variation, and what probability distribution the data appears to follow. There is no correct answer supplied in advance, so the algorithm must rely entirely on patterns present in the data itself.

Because no ground-truth label exists, the goals of unsupervised learning are descriptive rather than predictive. A clustering algorithm partitions points into groups; a dimensionality-reduction method projects high-dimensional points into a lower-dimensional space that preserves important relationships; a density estimator builds a model of how likely each region of the input space is. These outputs are then used for exploration, visualization, anomaly detection, feature construction, or as a preprocessing step that feeds a later supervised model.

Classic application areas include customer segmentation, where shoppers are grouped by behavior without predefined categories; gene-expression analysis, where samples are clustered to reveal subtypes; and recommendation pipelines, where compressed representations of users and items are learned before any rating prediction. In each case the value comes from organizing raw, unlabeled observations into a structure that a human or a downstream system can act on.

Operates on inputs only, with no target labels or human-provided answers.
Optimizes for descriptive structure: groupings, compressed coordinates, or a density model.
Common task families are clustering, dimensionality reduction, density estimation, and association rule mining.
Frequently used as exploratory analysis or as a feature-extraction stage feeding a later supervised model.

Clustering: Grouping Points Without Labels

Clustering assigns each data point to a group so that points in the same group are more similar to each other than to points in other groups. The most widely taught method is k-means, which partitions n observations into k clusters by alternating two steps: assign each point to the nearest cluster center, then recompute each center as the mean of the points assigned to it. The standard procedure is Lloyd's algorithm, and the number of clusters k must be chosen in advance.

Different clustering methods make different assumptions and suit different data shapes. DBSCAN (Ester, Kriegel, Sander, and Xu, 1996) groups points by density and can discover clusters of arbitrary shape while labeling sparse points as noise, without requiring k to be set. Hierarchical (agglomerative) clustering builds a tree of nested groupings by repeatedly merging the closest clusters, producing a dendrogram rather than a single flat partition. Gaussian mixture models treat the data as generated by a weighted sum of several Gaussian distributions and assign each point a soft probability of membership in each component.

The k-means objective is the within-cluster sum of squares, the total squared Euclidean distance from each point to the center of its assigned cluster. Minimizing this quantity is what drives the assignment and update steps, though the algorithm converges only to a local minimum and is sensitive to the initial center placement, which is why initializations such as k-means++ are commonly used.

arg min_{S} Σ_{i=1}^{k} Σ_{x ∈ S_i} ‖x − μ_i‖²

The k-means objective: choose the partition S = {S_1, ..., S_k} that minimizes the total squared distance from each point x to its cluster mean μ_i (the within-cluster sum of squares).

python

from sklearn.cluster import KMeans
import numpy as np

# X is unlabeled: only inputs, no target column
X = np.array([[1, 2], [1, 4], [1, 0],
              [10, 2], [10, 4], [10, 0]])

kmeans = KMeans(n_clusters=2, random_state=0, n_init="auto")
kmeans.fit(X)

print(kmeans.labels_)        # cluster id per point
print(kmeans.inertia_)       # within-cluster sum of squares
print(kmeans.predict([[0, 0], [12, 3]]))  # assign new points

Fitting k-means on unlabeled data with scikit-learn. The inertia_ attribute reports the within-cluster sum of squares the algorithm minimized.

k-means partitions data into k clusters by minimizing within-cluster squared distance; k is fixed in advance.
DBSCAN clusters by density, finds arbitrary shapes, and marks low-density points as noise.
Hierarchical clustering produces a dendrogram of nested groupings via repeated merging.
Gaussian mixture models assign soft, probabilistic cluster memberships under a generative assumption.

Dimensionality Reduction and Density Estimation

Dimensionality reduction compresses high-dimensional data into fewer coordinates while preserving as much useful structure as possible. Principal component analysis (PCA) is the linear baseline: it finds the orthogonal directions, the principal components, along which the data varies the most, and projects points onto the top few of them. Equivalently, PCA chooses the directions that maximize the variance of the projected data, which makes it a compact summary of the dominant axes of variation.

Nonlinear methods reveal structure that a linear projection cannot. t-SNE, introduced in 'Visualizing Data using t-SNE' by van der Maaten and Hinton (2008), maps high-dimensional points to two or three dimensions so that nearby points stay nearby, which makes it a popular tool for visualizing clusters in embeddings. UMAP (McInnes, Healy, and Melville, 2018) pursues a similar goal with faster runtime and a stronger tendency to preserve global structure. Both are mainly used for visualization rather than as exact distance-preserving transforms.

Density estimation builds a model of the probability distribution the data was drawn from, which supports sampling, anomaly detection, and likelihood scoring. Association rule mining is a related unsupervised task from transaction data: it discovers co-occurrence patterns such as items frequently bought together, producing rules of the form 'if a basket contains A and B, it often also contains C.'

w_1 = arg max_{‖w‖ = 1} Var(X w) = arg max_{‖w‖ = 1} (wᵀ Σ w)

PCA's first principal component w_1 is the unit direction that maximizes the variance of the projected data, where Σ is the covariance matrix of X; later components maximize remaining variance while staying orthogonal.

PCA is linear: it projects onto the orthogonal directions of maximum variance (the principal components).
t-SNE (van der Maaten & Hinton, 2008) is a nonlinear method that preserves local neighborhoods for visualization.
UMAP offers faster, often more globally faithful low-dimensional embeddings than t-SNE.
Density estimation models the data distribution; association rule mining finds co-occurrence patterns.

Versus Supervised, Reinforcement, and Self-Supervised Learning

The three classical paradigms differ in what feedback the learner receives. Supervised learning is trained on labeled examples and learns to map inputs to known outputs. Reinforcement learning has no labeled examples either, but it receives a scalar reward signal from an environment and learns a policy that maximizes cumulative reward over time. Unsupervised learning sits apart from both: it gets neither labels nor rewards, only the raw inputs, and must extract structure from them alone.

Self-supervised learning is a fourth framing that sits between supervised and unsupervised. It uses labels that are derived automatically from the data rather than annotated by humans. In next-token prediction, the label for each position is simply the next token already present in the text; in masked-language-modeling, tokens are hidden and the model is trained to reconstruct them. No human labeling is involved, yet the training objective is a supervised-style prediction task whose targets come for free from the data.

This distinction matters for how modern large language models are described. Their pretraining is self-supervised: GPT-style models are trained on next-token prediction and BERT-style models on masked-language-modeling, both using labels extracted from raw text. Older writing often called this 'unsupervised pretraining' because no manual annotation is required, and the term still appears, but the now-common and more precise framing is that LLM pretraining is self-supervised rather than purely unsupervised.

Supervised learning uses human-provided labels; reinforcement learning uses a reward signal; unsupervised learning uses neither.
Self-supervised learning derives its labels automatically from the data, e.g. the next token or a masked token.
Next-token prediction and masked-language-modeling are self-supervised objectives that pretrain modern LLMs.
LLM pretraining is best described as self-supervised, though older texts often label it unsupervised pretraining.

Evaluating Models Without Ground-Truth Labels

Evaluation is the hardest part of unsupervised learning, because the absence of labels removes the obvious yardstick used in supervised settings. Without a correct answer to compare against, quality must be judged by internal properties of the result or by performance on a separate downstream task. This makes model selection, such as choosing the number of clusters, an open question with no single definitive answer.

Internal clustering metrics score a grouping using only the data and the assignments. The silhouette coefficient, introduced by Rousseeuw in 1987, compares each point's average distance to others in its own cluster with its average distance to the nearest other cluster, yielding a value from -1 to 1 where higher means better-separated clusters. Inertia, the within-cluster sum of squares that k-means minimizes, measures compactness and always decreases as more clusters are added, so it is read with the elbow heuristic rather than maximized directly.

Because internal metrics can be misleading, learned representations are often evaluated extrinsically through downstream-task probing: the unsupervised output is frozen and fed into a small supervised model on a labeled task, and that model's accuracy serves as a proxy for representation quality. This approach ties an unsupervised model's worth to its usefulness elsewhere, which is frequently the most honest available signal.

No ground-truth labels means quality is judged by internal structure or downstream usefulness, not direct accuracy.
The silhouette coefficient (Rousseeuw, 1987) scores cluster cohesion versus separation on a -1 to 1 scale.
Inertia measures cluster compactness but decreases with more clusters, so it is read via the elbow heuristic.
Downstream-task probing freezes the unsupervised output and measures its accuracy on a labeled task.

Where Unsupervised Learning Fits in Practice

In production systems unsupervised learning rarely works in isolation; it is usually one stage in a larger pipeline. Embeddings produced by a model are routinely clustered to group similar documents, deduplicated, or reduced with PCA or UMAP for visualization and inspection. Anomaly detection systems fit a density or clustering model to normal behavior and flag points that fall outside it, which is valuable precisely because anomalies are rare and seldom labeled.

Unsupervised techniques also support retrieval and memory systems. Semantic search relies on vector representations whose geometry, the relative positions of points, is exactly the kind of structure unsupervised methods analyze; clustering those vectors can organize a knowledge base, and dimensionality reduction can compress it. The structure-finding mindset of unsupervised learning underpins how dense representations are understood and managed.

The boundary with self-supervised learning continues to blur in modern systems. The representations that power retrieval, clustering, and search are themselves often produced by self-supervised pretraining, then consumed by genuinely unsupervised steps such as k-means or density estimation. Understanding which stage supplies labels, automatically or not, clarifies how a pipeline actually learns and what each component can and cannot guarantee.

Embeddings are commonly clustered, deduplicated, or reduced for organization and visualization.
Anomaly detection fits a model of normal data and flags outliers, suited to rare unlabeled events.
Semantic search and vector stores depend on the geometric structure unsupervised methods analyze.
Modern pipelines mix self-supervised representation learning with genuinely unsupervised downstream steps.

Key takeaways

Unsupervised learning finds structure in unlabeled data using inputs alone, with no target labels and no reward signal.
Its main task families are clustering (k-means, DBSCAN, hierarchical, Gaussian mixture models), dimensionality reduction (PCA, t-SNE, UMAP), density estimation, and association rule mining.
k-means minimizes the within-cluster sum of squares; PCA finds the orthogonal directions of maximum projected variance.
Self-supervised learning derives labels automatically from the data, and modern LLM pretraining (next-token prediction, masked-language-modeling) is best described as self-supervised rather than purely unsupervised.
Evaluation is hard without labels and relies on internal metrics like the silhouette coefficient and inertia, or on downstream-task probing.
In practice unsupervised methods are usually a stage in a pipeline, clustering or compressing embeddings for search, organization, and anomaly detection.

Frequently asked questions

Supervised learning is trained on labeled examples, input-output pairs, and learns to predict the output for new inputs. Unsupervised learning receives only inputs with no labels and must discover structure such as clusters or compressed coordinates on its own. The presence or absence of target labels is the defining distinction.

It is best described as self-supervised. Objectives like next-token prediction and masked-language-modeling use labels taken automatically from the raw text, the next token or a masked token, so no human annotation is needed. Older writing often called this unsupervised pretraining because no manual labeling is involved, but the more precise modern term is self-supervised.

The major task families are clustering, which groups similar points; dimensionality reduction, which compresses data into fewer coordinates; density estimation, which models the data distribution; and association rule mining, which finds co-occurrence patterns. Clustering methods include k-means, DBSCAN, hierarchical clustering, and Gaussian mixture models, while PCA, t-SNE, and UMAP are common for dimensionality reduction.

Because there is no ground truth, quality is judged by internal metrics or downstream usefulness. Internal clustering measures include the silhouette coefficient, which compares within-cluster and between-cluster distances, and inertia, which measures compactness. Alternatively, downstream-task probing freezes the unsupervised output, feeds it to a small supervised model on a labeled task, and uses that accuracy as a proxy.

k-means minimizes the within-cluster sum of squares, the total squared Euclidean distance from each point to the mean of its assigned cluster. It does this by alternating between assigning points to the nearest center and recomputing each center as the mean of its members, which is Lloyd's algorithm. The method converges only to a local minimum, so initialization such as k-means++ matters.

PCA is unsupervised. It uses only the input features and finds the orthogonal directions, the principal components, along which the data varies the most, with no labels involved. Projecting onto the top components reduces dimensionality while keeping the directions of greatest variance.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free