Models & Evaluation

BERT

BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only Transformer model from Google, introduced in 2018, that learns deeply bidirectional contextual representations of text by jointly conditioning on both left and right context. It is pretrained with masked language modeling and then fine-tuned for understanding tasks like classification, question answering, and named entity recognition.

What BERT Is

BERT stands for Bidirectional Encoder Representations from Transformers. It was introduced by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova at Google in the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (arXiv:1810.04805). BERT is built only from the encoder stack of the original Transformer architecture, which makes it a model for understanding text rather than generating it.

The central idea is deep bidirectionality. Earlier language models read text in a single direction, predicting each word from only the words that came before it. BERT instead conditions every token's representation on both its left and right context across all layers at once. This lets a single representation of a word account for its full surrounding sentence, which is what makes BERT strong at tasks that depend on sentence meaning.

BERT is not used directly to answer questions out of the box. It produces dense vector representations for tokens and for whole sequences. Those representations are then either fine-tuned on a labeled task or read off as features for downstream models, retrieval, and search systems.

Encoder-only Transformer focused on language understanding, not text generation.
Introduced by Google in 2018 (arXiv:1810.04805), authored by Devlin, Chang, Lee, and Toutanova.
Deeply bidirectional: each token sees both left and right context in every layer.
Outputs contextual embeddings that are fine-tuned or used as features downstream.

How BERT Is Pretrained

BERT is pretrained on large amounts of unlabeled text using two self-supervised objectives. The first is Masked Language Modeling (MLM). Roughly 15% of the input tokens are selected at random, and the model is trained to predict the original token at each selected position from the surrounding context. Because a token can attend to words on both sides, MLM forces the model to learn genuinely bidirectional representations.

To reduce a mismatch between pretraining and fine-tuning (the special mask token never appears at fine-tuning time), the 15% of selected tokens are not all replaced the same way. Of those selected positions, 80% are replaced with a special [MASK] token, 10% are replaced with a random token from the vocabulary, and 10% are left unchanged. The model must predict the correct original token in every case.

The second objective is Next Sentence Prediction (NSP). The model receives two segments, A and B, and predicts whether B is the actual sentence that followed A or a random sentence from the corpus. During training, 50% of the pairs are true continuations and 50% are random. NSP was meant to help tasks that reason over sentence pairs, such as question answering and natural language inference.

L_MLM = -Σ_{i ∈ M} log P(x_i | x_\M),    P(x_i = v | h_i) = softmax(W h_i + b)_v

The MLM loss sums the negative log-likelihood over masked positions M, where each masked token's distribution over the vocabulary is a softmax over the final hidden state h_i for that position.

MLM masks about 15% of tokens and trains the model to predict them.
Of selected tokens: 80% become [MASK], 10% a random token, 10% unchanged.
NSP trains the model to tell a true next sentence from a random one, 50/50.
Both objectives are self-supervised, so no human labels are needed for pretraining.

Architecture and Input Format

BERT was released in two main sizes. BERT-base has 12 Transformer encoder layers, a hidden size of 768, 12 self-attention heads, and about 110 million parameters. BERT-large has 24 layers, a hidden size of 1024, 16 attention heads, and about 340 million parameters. BERT-large is more accurate but heavier to run and fine-tune.

Text is split into subword units using WordPiece tokenization with a vocabulary of about 30,000 tokens. WordPiece breaks rare or unseen words into smaller known pieces, which keeps the vocabulary compact while still covering open-ended text. Each token's final input embedding is the sum of its token embedding, a segment embedding (which sentence it belongs to), and a position embedding.

Two special tokens shape every input. A [CLS] token is prepended to the sequence, and its final hidden state serves as an aggregate representation of the whole input that is commonly fed to a classifier. A [SEP] token marks the end of a segment and separates the two sentences in a pair, supporting tasks like NSP, question answering, and sentence-pair classification.

BERT-base: 12 layers, hidden size 768, 12 heads, ~110M parameters.
BERT-large: 24 layers, hidden size 1024, 16 heads, ~340M parameters.
WordPiece tokenization with a vocabulary of roughly 30,000 subword units.
[CLS] gives a pooled sequence vector; [SEP] separates segments in a pair.

Fine-Tuning and the GLUE Results

BERT popularized the pretrain-then-fine-tune recipe for natural language processing. A single pretrained model is adapted to a specific task by adding a small task-specific layer on top and continuing training on labeled data for that task. The same pretrained weights can be fine-tuned separately into a sentiment classifier, a question-answering model, or a named entity recognizer, which removed the need to design a new architecture per task.

BERT became famous because of its benchmark results. On GLUE, a collection of language understanding tasks, BERT pushed the overall score to 80.5%, an absolute improvement of 7.7 points over the previous state of the art at the time. It also set new records on the SQuAD question answering datasets and on several other tasks, which is why it was widely adopted across the field very quickly.

The code snippet below loads a pretrained BERT model with the Hugging Face Transformers library and produces contextual embeddings for a sentence. The last hidden states give one vector per token, and the [CLS] vector is often used as a sentence-level summary.

python

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

inputs = tokenizer("BERT reads context in both directions.", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# One contextual embedding per token
token_embeddings = outputs.last_hidden_state      # shape: [1, seq_len, 768]
# The [CLS] vector as a sentence-level summary
cls_embedding = outputs.last_hidden_state[:, 0]    # shape: [1, 768]

Loading BERT-base and extracting token-level and [CLS] sentence embeddings with Hugging Face Transformers.

Fine-tuning adds a small head and trains on labeled data for one task.
One pretrained checkpoint adapts to classification, QA, NER, and more.
BERT raised the GLUE score to 80.5%, a 7.7 point absolute gain.
It also set new records on SQuAD and other understanding benchmarks.

BERT vs GPT and the Descendants

BERT and GPT sit on opposite sides of the Transformer. BERT is encoder-only and bidirectional, so it sees the whole input at once and is suited to understanding tasks like classification and extraction. GPT-style models are decoder-only and autoregressive: they read left to right and predict the next token, which makes them suited to open-ended text generation. BERT cannot generate fluent text on its own, and a plain decoder-only model does not produce the same kind of bidirectional understanding representation.

BERT spawned a family of improved encoders. RoBERTa kept the architecture but removed the NSP objective, trained longer on more data with larger batches, and improved results. ALBERT shared parameters across layers and replaced NSP with a sentence-order prediction objective to cut model size. DistilBERT used knowledge distillation to make a smaller, faster model. ELECTRA replaced MLM with replaced-token detection, training the model to spot tokens swapped in by a small generator. DeBERTa added disentangled attention that encodes content and position separately.

BERT-style encoders remain in active use even as generative decoder models dominate the headlines. They power text embedding and semantic search systems, serve as rerankers in retrieval pipelines, and run as efficient classifiers and named entity recognizers where bidirectional understanding and low latency matter more than text generation.

BERT is encoder-only and bidirectional; GPT is decoder-only and autoregressive.
RoBERTa dropped NSP and trained longer; ALBERT shared parameters and used sentence-order prediction.
DistilBERT distills a smaller model; ELECTRA and DeBERTa change the pretraining task and attention.
Encoder models still drive embeddings, retrieval reranking, classification, and NER.

Key takeaways

BERT is an encoder-only Transformer from Google (2018, arXiv:1810.04805) for language understanding.
It learns deeply bidirectional representations by conditioning on both left and right context.
Pretraining uses Masked Language Modeling on ~15% of tokens plus Next Sentence Prediction.
BERT-base has ~110M parameters; BERT-large has ~340M; both use WordPiece tokenization.
Fine-tuning one pretrained model per task pushed the GLUE score to 80.5%.
Descendants like RoBERTa, ALBERT, DistilBERT, ELECTRA, and DeBERTa refine the recipe.

Frequently asked questions

BERT is a Transformer model from Google that reads a sentence in both directions at once to understand the meaning of each word in context. It is pretrained on large amounts of text by filling in masked-out words, then fine-tuned for specific tasks like classification, question answering, or named entity recognition. It produces vector representations of text rather than generating new text.

BERT is encoder-only and bidirectional, so it sees the entire input at once and excels at understanding tasks. GPT is decoder-only and autoregressive, predicting one token at a time from left to right, which makes it suited to generating text. In short, BERT is built to read and classify, while GPT is built to write.

The first is Masked Language Modeling (MLM), where about 15% of input tokens are hidden and the model predicts them from surrounding context. The second is Next Sentence Prediction (NSP), where the model decides whether one segment truly follows another. Both are self-supervised, so they need no human-labeled data.

BERT-base has 12 encoder layers, a hidden size of 768, and about 110 million parameters. BERT-large has 24 layers, a hidden size of 1024, and about 340 million parameters. BERT-large is generally more accurate but slower and more memory-intensive to fine-tune and run.

Yes. While decoder-only generative models get most attention, BERT-style encoders are still widely used for text embeddings, semantic search, retrieval reranking, classification, and named entity recognition. They are efficient and well suited to understanding tasks where low latency matters and no text generation is required.

RoBERTa removed NSP and trained longer on more data. ALBERT shared parameters to shrink the model and used sentence-order prediction. DistilBERT is a smaller distilled version, ELECTRA uses replaced-token detection instead of masking, and DeBERTa adds disentangled attention for content and position.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free