AI Foundations

Named Entity Recognition (NER)

By Aditya Kumar Jha, Engineer

Named Entity Recognition (NER) is the natural language processing task of locating spans of text that name real-world entities and classifying each into predefined types such as person, organization, location, date, or money.

What Named Entity Recognition Is

Named Entity Recognition, abbreviated NER, is a core information extraction task in natural language processing. Given a sequence of text, an NER system must do two things at once: detect the boundaries of each span that refers to a named entity, and assign that span a category from a fixed inventory of types. A named entity is a concrete, nameable thing rather than a general concept, so a string like "Marie Curie" is a PERSON entity and "Paris" is a LOCATION, while a common noun like "scientist" or "city" is not tagged.

The set of recognized types depends on the schema. The CoNLL-2003 shared task uses four coarse types: person (PER), organization (ORG), location (LOC), and miscellaneous (MISC). Broader schemas such as OntoNotes 5.0 define 18 types, adding categories like DATE, TIME, MONEY, PERCENT, GPE (geopolitical entity), and WORK_OF_ART. Domain-specific NER extends this further, for example tagging genes and diseases in biomedical text or invoice numbers and account IDs in document processing.

NER is usually framed as the front end of a larger pipeline. Detected entities feed entity linking, relation extraction, knowledge-graph construction, search indexing, and redaction of personally identifiable information. Because the output is a set of typed, located spans rather than free text, NER produces structured data that downstream systems can query and join.

  • NER both finds entity spans and labels each with a type from a fixed set.
  • A named entity is a specific nameable thing, not a generic noun.
  • Type inventories range from CoNLL-2003's 4 types to OntoNotes 5.0's 18.
  • Output is structured, typed spans that downstream tasks can consume directly.

Tagging Schemes: Turning Spans Into Token Labels

Entities can span multiple tokens, so NER is commonly reframed as token classification: assign one label to every token, then reconstruct spans from the label sequence. The standard encoding is the IOB scheme, also called BIO, which tags each token with B- (beginning of an entity of a given type), I- (inside, a continuation of the current entity), or O (outside any entity). The variant used in practice, IOB2, marks the first token of every entity with B-, so two adjacent entities of the same type are always separable.

BILOU is a richer five-label scheme that adds explicit end and singleton markers: B- for the first token of a multi-token entity, I- for inner tokens, L- for the last token, O for outside, and U- for a unit-length (single-token) entity. Teaching a model to predict both the start and the end of a span, rather than only the start, often improves boundary accuracy, which is why BILOU (sometimes written BIOES) is a common alternative to plain BIO.

Whatever scheme is chosen, decoding must respect its grammar. An I- tag should only follow a B- or I- of the same type, so inference layers or post-processing fix illegal transitions such as an I-ORG appearing with no preceding organization tag.

  • BIO/IOB tags tokens as B- (begin), I- (inside), or O (outside).
  • IOB2 marks every entity's first token with B-, separating adjacent same-type entities.
  • BILOU adds L- (last) and U- (unit) for sharper boundary learning.
  • Decoders enforce scheme grammar so illegal tag sequences are not emitted.

From Rules and CRFs to BiLSTM-CRF

Early NER systems relied on hand-written rules and gazetteers, which are curated lists of known names (cities, companies, given names). Rules and dictionaries are precise on familiar entities but brittle on novel names and ambiguous contexts, and they require constant manual maintenance. They remain useful as features or fallbacks rather than as a complete system.

Statistical sequence models replaced pure rules. The Conditional Random Field (CRF) became the standard: it scores an entire label sequence jointly, learning which tag transitions are likely, so it naturally captures constraints like "I-PER cannot start a span." CRFs operate over hand-engineered features such as capitalization, prefixes and suffixes, part-of-speech tags, and gazetteer membership.

The neural era began by feeding learned representations into a CRF. Lample et al. (2016), in "Neural Architectures for Named Entity Recognition" (NAACL 2016, arXiv:1603.01360), introduced a bidirectional LSTM that reads context in both directions, combined with character-level embeddings and a CRF decoding layer. This BiLSTM-CRF reached 90.94 F1 on CoNLL-2003 English without gazetteers, and the BiLSTM-CRF pattern became the dominant NER architecture for several years.

python
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying a U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.label_)
# Apple ORG
# U.K. GPE
# $1 billion MONEY
Extracting typed entity spans with spaCy: doc.ents yields each span's text and its predicted label.
  • Gazetteers and rules are precise but brittle and maintenance-heavy.
  • CRFs score whole label sequences, learning valid tag transitions.
  • Lample et al. (2016) paired a BiLSTM encoder with a CRF decoder.
  • That BiLSTM-CRF hit 90.94 F1 on CoNLL-2003 English without gazetteers.

Modern NER: Fine-Tuned Encoders and LLMs

Today's strongest supervised NER systems fine-tune a pretrained Transformer encoder for token classification. A model such as BERT produces a contextual vector for every token; a linear classification head on top predicts the BIO (or BILOU) tag for each token, and the model is trained end to end on a labeled corpus. Pretraining on large unlabeled text gives the encoder broad lexical and syntactic knowledge, so fine-tuned BERT-family models surpass BiLSTM-CRF on standard benchmarks.

Subword tokenization complicates the token-to-tag alignment. Encoders split words into subword pieces, so labels are typically aligned to the first subword of each word, with continuation pieces ignored in the loss or assigned a special label. The Hugging Face token-classification pipeline handles this alignment and offers aggregation strategies that merge subword predictions back into whole-entity spans.

Large language models add a different mode: zero-shot and few-shot extraction by prompting. Instead of training a tagger, a prompt instructs the model to return entities (often as JSON) for arbitrary, user-defined types without labeled examples. This is flexible for new domains and rare types, but it can hallucinate spans or types and is generally harder to evaluate strictly than a fixed-schema supervised tagger, so high-stakes pipelines still favor fine-tuned models or constrained decoding.

python
from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="dslim/bert-base-NER",
    aggregation_strategy="simple",
)
for e in ner("Marie Curie was born in Warsaw."):
    print(e["word"], e["entity_group"])
# Marie Curie PER
# Warsaw LOC
A Hugging Face token-classification pipeline; aggregation_strategy merges subword predictions into whole entities.
  • Fine-tuned encoders like BERT add a per-token classification head over BIO tags.
  • Subword tokenization requires aligning labels to the first piece of each word.
  • LLM prompting enables zero/few-shot extraction of arbitrary, user-defined types.
  • Prompted LLM extraction is flexible but can hallucinate and is harder to score exactly.

Evaluating NER: Entity-Level Precision, Recall, and F1

NER is scored at the entity (span) level, not the token level. A predicted entity counts as correct only when it is an exact match: its start boundary, end boundary, and type all agree with a gold entity. A span that overlaps a gold entity but has the wrong boundary, or the right boundary but the wrong type, is counted as both a false positive and a missed gold entity, so partial credit is not given. This strictness is the CoNLL evaluation convention established by the CoNLL-2003 shared task and implemented by the standard conlleval script.

From the counts of true positives, false positives, and false negatives, precision and recall are computed at the span level, and the headline metric is their harmonic mean, the F1 score. Precision answers "of the entities the system predicted, what fraction were exactly right," while recall answers "of the gold entities, what fraction did the system find." Reporting F1 alone can hide a precision/recall imbalance, so careful reports show all three.

Because evaluation is exact-match at the span level, the choice of tokenization and tagging scheme affects scores, and tools must decode the tag sequence into spans before comparing. Per-type F1 is also reported to reveal which categories (often MISC or rare types) drag down the overall number.

P = TP / (TP + FP), R = TP / (TP + FN), F1 = 2PR / (P + R)
Entity-level metrics: TP is exactly-matched entities, FP is predicted entities with no exact gold match, FN is gold entities the system missed. F1 is the harmonic mean of span-level precision P and recall R.
  • A predicted entity is correct only on exact match of start, end, and type.
  • Precision and recall are computed over spans, not individual tokens.
  • F1 is the harmonic mean of span-level precision and recall.
  • The CoNLL convention (conlleval) gives no partial credit for overlaps.

Where NER Is Used

NER is the extraction layer beneath many text systems. In search and analytics, recognizing that a query or document mentions a specific company, person, or place lets engines index and filter by entity rather than by keyword alone. In knowledge-graph construction, NER finds the nodes (entities) that relation extraction then connects into edges, building structured graphs from unstructured text.

Privacy and compliance workflows use NER to detect personally identifiable information (PII) such as names, addresses, phone numbers, and account identifiers so they can be redacted or masked. Document processing pipelines apply NER after optical character recognition to pull structured fields, for example vendor, date, total, and tax, from invoices, contracts, and forms, turning scanned pages into queryable records.

NER also feeds retrieval and question answering. Identifying the entities in a question or passage sharpens retrieval, grounds answers, and supports linking mentions to a canonical knowledge base. Because entities are typed and located, the same extraction can serve indexing, redaction, and graph building from one pass over the text.

  • Entity-aware search and analytics index by people, places, and organizations.
  • Knowledge graphs use NER to find the nodes that relations then link.
  • PII detection relies on NER to locate names, IDs, and contact details for redaction.
  • Document pipelines run NER after OCR to extract structured fields from forms.

Key takeaways

  • NER locates entity spans in text and classifies each into a predefined type.
  • Token-level BIO/BILOU tagging encodes multi-token spans for sequence labeling.
  • CRFs, then BiLSTM-CRF (Lample et al. 2016), advanced NER before Transformers.
  • Fine-tuned encoders like BERT and prompted LLMs are the modern approaches.
  • CoNLL-2003 and OntoNotes are the standard benchmarks and type inventories.
  • Evaluation uses exact-match, entity-level precision, recall, and F1 (= 2PR/(P+R)).

Frequently asked questions

It is an NLP task that scans text, finds the words that name a specific thing, and labels each with a category such as person, organization, location, date, or money. For example, in "Apple opened a store in Paris," it would tag "Apple" as an organization and "Paris" as a location. The output is a set of typed, located spans that other systems can use.
BIO (also called IOB) labels every token as B- (the beginning of an entity), I- (inside, a continuation of the current entity), or O (outside any entity), with a type suffix like B-PER. It turns span detection into per-token classification. The common IOB2 variant marks the first token of every entity with B-, which keeps two adjacent entities of the same type from merging.
It is scored at the entity level with precision, recall, and their harmonic mean, F1. A predicted entity counts as correct only on an exact match of start boundary, end boundary, and type, so overlapping or mistyped spans get no partial credit. This exact-match rule is the CoNLL convention implemented by the conlleval script.
A fine-tuned encoder such as BERT is trained on labeled data to predict a tag for each token from a fixed type set, giving consistent, easily scored output. An LLM can extract entities zero-shot or few-shot from a prompt for arbitrary user-defined types without training data, which is flexible but can hallucinate spans and is harder to evaluate strictly. High-stakes pipelines often prefer fine-tuned models for reliability.
CoNLL-2003 is a widely used NER benchmark built from news text that defines four entity types: person, organization, location, and miscellaneous, in English and German. OntoNotes 5.0 is a larger, more diverse corpus that defines 18 entity types, adding categories like DATE, MONEY, PERCENT, and GPE. Both are standard datasets for training and comparing NER systems.
It powers entity-aware search, knowledge-graph construction, and relation extraction by identifying the entities in text. It is also used to detect personally identifiable information for redaction, and to pull structured fields like vendor, date, and total from documents after OCR. In retrieval and question answering, entities sharpen indexing and ground answers.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free