Training & Alignment

Mechanistic Interpretability

Mechanistic interpretability is the study of reverse-engineering the internal computations of neural networks into human-understandable algorithms. It aims to identify the features and circuits a model uses, so its behavior can be predicted, audited, and made safer.

What is Mechanistic Interpretability?

Mechanistic interpretability is the field of reverse-engineering the internal computations of neural networks into human-understandable terms. Rather than treating a model as a black box that maps inputs to outputs, it tries to explain how specific weights and activations implement the algorithms that produce a behavior.

The goal is concrete: identify the internal features a model represents and the circuits, or connected groups of components, that combine those features to perform a task. If researchers can read a model's internal reasoning the way one reads source code, they can predict failures, detect deception or bias, and intervene before deployment. This makes mechanistic interpretability a core tool for AI safety and alignment.

Goal: explain model behavior in terms of internal features and circuits.
Contrast: black-box methods explain outputs; mechanistic work explains computation.
Motivation: safety, auditing, and trust through understanding.

Features and circuits

A feature is a direction in a model's internal activation space that corresponds to a human-meaningful concept, for example a specific person, a programming bug, or the idea of sycophancy. A circuit is a set of features and connections that together carry out a computation, such as moving information about a subject from earlier to later token positions.

Early circuit-level work focused on small or vision models, where individual neurons sometimes mapped cleanly to concepts. In large language models the picture is harder because neurons are polysemantic: a single neuron responds to many unrelated concepts, which obscures what the network is actually computing.

Feature: an activation direction tied to a human concept.
Circuit: features and connections that implement a function.
Polysemanticity: one neuron firing for many unrelated concepts.

Superposition and sparse autoencoders

Polysemanticity is explained by superposition, the hypothesis that models pack more features than they have neurons by representing them as overlapping directions rather than dedicated units. Superposition lets a network store many concepts efficiently, but it makes individual neurons hard to interpret.

Sparse autoencoders (SAEs) are the leading technique for pulling features back out of superposition. An SAE is trained to reconstruct a layer's activations using a much larger but sparse set of learned features, so that only a few activate at once. Each learned feature tends to be monosemantic, meaning it corresponds to one concept. Anthropic's 2024 work Scaling Monosemanticity applied SAEs to Claude 3 Sonnet and extracted millions of interpretable features, including the well-known Golden Gate Bridge feature that activated across text in multiple languages and even relevant images.

Superposition: features stored as overlapping directions, not single neurons.
Sparse autoencoders decompose activations into sparse, mostly monosemantic features.
Scaling Monosemanticity recovered millions of features from Claude 3 Sonnet.

Why it matters for safety

Mechanistic interpretability supports safety in two ways. First, it enables auditing: if a deceptive or harmful capability has an identifiable internal signature, it can be detected even when the model's outputs look benign. Second, the same features can be used for steering, where amplifying or suppressing a feature changes behavior, which both tests causal claims and offers a lever for control.

The field is still maturing. Extracting features at scale is computationally expensive, no single method captures every behavior, and verifying that a proposed circuit is the true mechanism remains difficult. Even so, it is one of the most direct approaches to understanding what large models are actually doing internally rather than inferring it from outputs alone.

Auditing: detect hidden capabilities from internal signatures.
Steering: amplify or suppress features to test and control behavior.
Open challenges: cost, coverage, and verifying that a circuit is the real mechanism.

Key takeaways

Mechanistic interpretability reverse-engineers neural network internals into human-understandable features and circuits.
Neurons in large models are polysemantic, explained by superposition, where features are stored as overlapping directions.
Sparse autoencoders decompose activations into sparse, mostly monosemantic features.
Anthropic's Scaling Monosemanticity extracted millions of interpretable features from Claude 3 Sonnet.
The field underpins AI safety by enabling auditing, behavior steering, and detection of hidden capabilities.

Frequently asked questions

Mechanistic interpretability is the study of reverse-engineering a neural network's internal computations into human-understandable algorithms. It identifies the features a model represents and the circuits that combine them, so behavior can be explained rather than just observed.

Much explainable AI explains outputs, for example which inputs influenced a prediction. Mechanistic interpretability goes deeper, aiming to explain the internal computation itself in terms of features and circuits, closer to reading a model's source code.

Superposition is the hypothesis that models represent more features than they have neurons by encoding them as overlapping directions in activation space. It explains why individual neurons are polysemantic and respond to many unrelated concepts.

Sparse autoencoders decompose a layer's activations into a large, sparse set of learned features, only a few of which activate at once. These features tend to be monosemantic, making them far easier to interpret than raw polysemantic neurons.

It allows auditing of model internals to detect hidden or harmful capabilities and supports steering, where adjusting a feature changes behavior. Understanding what a model computes internally helps predict and prevent failures before deployment.

Put the idea into practice

MemX is an AI memory app built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free