An embedding is a dense numeric vector that represents the meaning of a piece of data, such as text, an image, or audio, so that items with similar meaning land close together in vector space. This geometry lets machines compare meaning rather than match exact words.
What an embedding is: meaning compressed into a dense vector
An embedding is a list of floating-point numbers, a vector, that encodes the meaning of an input such as a word, sentence, document, or image. Instead of storing data as raw symbols, an embedding model maps each input to a fixed-length point in a high-dimensional space, where geometric proximity corresponds to semantic similarity. Two passages that say the same thing in different words end up near each other, even when they share no literal terms.
These vectors are called dense because nearly every dimension holds a nonzero value, in contrast to sparse representations like one-hot encodings or bag-of-words counts, where most entries are zero. A dense embedding packs information about context, topic, sentiment, and relationships into a few hundred or few thousand numbers. The exact meaning of any single dimension is rarely interpretable on its own; meaning emerges from the overall position of the vector relative to others.
- Dense vector: most dimensions are nonzero, unlike sparse keyword vectors.
- Similar meaning maps to nearby points; the model learns this geometry from data.
- Individual dimensions are usually not human-interpretable in isolation.
From word embeddings to contextual and sentence embeddings
Early embeddings were word embeddings: each word in the vocabulary received one fixed vector. Word2vec, introduced by Mikolov and colleagues at Google in 2013, learned these vectors by predicting nearby words from a sliding context window (skip-gram and CBOW). GloVe, from Pennington, Socher, and Manning at Stanford in 2014, instead factorized global word co-occurrence statistics across the whole corpus. Both produced vectors with a famous property: simple vector arithmetic captured analogies and relationships between words.
Static word embeddings share one core limitation: a word gets the same vector regardless of context, so river bank and bank account collapse to one representation. Contextual models solved this. BERT and other transformer encoders generate a different vector for a word depending on the surrounding sentence, using bidirectional self-attention. Building on this, sentence embedding models such as Sentence-BERT (Reimers and Gurevych, 2019) produce a single vector for an entire sentence or passage, which is what most modern search and retrieval systems actually use.
- Word2vec (2013) and GloVe (2014): one static vector per word.
- Contextual embeddings (BERT-style): the vector changes with surrounding context.
- Sentence and document embeddings: one vector for a whole passage, ideal for retrieval.
What an embedding model does and what dimensionality means
An embedding model is a neural network trained to output a vector for each input. Modern text embedding models are typically transformer encoders fine-tuned so that semantically related pairs produce nearby vectors and unrelated pairs produce distant ones, often with a contrastive training objective. At inference time you send text in and receive a fixed-length vector out, ready to store in a database or compare against other vectors.
Dimensionality is the length of that vector. Higher dimensions can capture more nuance but cost more storage and compute per comparison. As of 2026, OpenAI's text-embedding-3-large defaults to 3,072 dimensions and text-embedding-3-small to 1,536. Some models support Matryoshka representation learning, which packs the most important information into the earliest dimensions so a vector can be truncated to a shorter length (for example 256 or 1,024) with only modest quality loss, trading a little accuracy for large savings in storage and search speed.
- Input text, output a fixed-length vector; the model is trained, not rule-based.
- More dimensions can mean more nuance but higher storage and query cost.
- Matryoshka models let you truncate vectors to shorter lengths to save space.
How similarity is measured: cosine, dot product, and distance
Once data is embedded, comparing meaning becomes a math operation on vectors. Cosine similarity measures the angle between two vectors and ignores their magnitude, so it captures whether two items point in the same semantic direction. It is the default metric for most text embedding models. The dot product multiplies and sums corresponding elements, accounting for both direction and magnitude, and is preferred when a model deliberately encodes information in vector length.
Euclidean distance measures straight-line separation in the space, with smaller distance meaning more similar. A key practical fact ties these together: if vectors are normalized to unit length, cosine similarity, normalized dot product, and Euclidean-distance ranking all produce the same nearest-neighbor ordering. Many embedding pipelines normalize vectors for exactly this reason, which is why cosine similarity has become the conventional default in semantic search.
- Cosine similarity: angle only, the standard choice for text embeddings.
- Dot product: direction and magnitude; use when magnitude carries meaning.
- On normalized vectors, cosine, dot product, and Euclidean rank identically.
Embeddings as the foundation of semantic search and retrieval
Semantic search replaces literal keyword matching with meaning matching. Documents are embedded ahead of time and stored in a vector index. At query time the question is embedded with the same model, and the system retrieves the stored vectors closest to the query vector. This finds relevant results even when the query and the document share no exact words, because proximity reflects meaning rather than spelling.
Searching every vector exhaustively is slow at scale, so vector databases use approximate nearest neighbor (ANN) algorithms to return close matches quickly. This retrieval step is the backbone of retrieval-augmented generation (RAG): an embedding-based search pulls the most relevant passages from a knowledge base, and those passages are inserted into a language model's prompt as grounding context. Embeddings are what let a RAG system answer from a specific corpus rather than from the model's frozen training data alone.
- Index documents as vectors, then embed each query the same way to find neighbors.
- Approximate nearest neighbor search keeps retrieval fast over millions of vectors.
- In RAG, embedding retrieval supplies the grounding passages for the model's answer.
Text, image, and multimodal embeddings
Embeddings are not limited to text. Image models produce vectors for pictures, audio models for sound, and code models for source code. The transformative idea behind multimodal embeddings is placing different data types in one shared vector space, so a text query can directly retrieve an image. OpenAI's CLIP, released in 2021, trained an image encoder and a text encoder jointly on roughly 400 million image-text pairs so that a photo and its caption land near each other in the same space.
Shared multimodal spaces enable searching photos with plain-language descriptions, matching captions to images, and building retrieval systems that span formats. As of 2026, embedding models that jointly handle text, images, and other modalities in a single space are increasingly common, extending the same nearest-neighbor search techniques developed for text across every kind of content.
- Separate embedding models exist for text, images, audio, and code.
- Multimodal models like CLIP put images and text in one shared space.
- Shared spaces let a text query retrieve an image, or vice versa.
Choosing an embedding model: quality, dimensions, cost, and context
Selecting an embedding model means balancing several factors. Retrieval quality is the headline concern, and public benchmarks such as the Massive Text Embedding Benchmark (MTEB) compare models across many tasks; the leaderboard shifts frequently as new models appear, so treat any specific ranking as a snapshot. Dimensionality affects both storage and query latency, since every stored vector and every comparison scales with length.
Cost matters at scale because most hosted embedding APIs charge per token of input, and a large corpus can require embedding millions of chunks. Context length, the maximum amount of text a model can embed at once, determines how large each chunk can be before it must be split. Practical selection also weighs whether to use a hosted API or a self-hosted open-weight model, and whether the model was trained on a domain similar to your data.
- Quality: consult MTEB and similar benchmarks, but verify on your own data.
- Dimensions and cost: longer vectors and larger corpora raise storage and price.
- Context length sets the maximum chunk size before text must be split.
Embeddings in AI memory and recall
AI memory systems and second-brain applications rely on embeddings to make saved information findable by meaning. When a note, document, transcript, or photo is captured, it is embedded into a vector and stored, so a later question phrased in everyday language can retrieve it through semantic search rather than exact keywords.
This pattern turns a growing personal archive into a queryable knowledge base. The retrieved vectors can then feed a language model as context, combining recall with generation. The quality of that recall depends directly on the embedding model chosen and on how the source material is divided into chunks before embedding.
- Captured notes, documents, and media are embedded and stored as vectors.
- Plain-language questions retrieve them by meaning, not exact wording.
- Retrieved vectors can supply context to a language model for answers.
Limitations: drift, domain mismatch, and chunking sensitivity
Embeddings are powerful but imperfect. Models trained on general web text can underperform on specialized domains such as law, medicine, or internal jargon, a problem known as domain mismatch; fine-tuning or a domain-specific model often helps. Embedding spaces are also model-specific and not interchangeable: a query embedded with one model cannot be compared against vectors produced by a different model, so changing models requires re-embedding the entire corpus, a form of drift to plan for.
Retrieval quality is highly sensitive to chunking, the way long documents are split before embedding. Chunks that are too large dilute meaning across many topics, while chunks that are too small lose the surrounding context needed to interpret them. Embeddings also capture correlation rather than verified truth, so semantic similarity can surface passages that are related in topic but wrong in substance, which is why retrieval is paired with careful ranking and, in RAG, with a generation step whose output should still be verified.
- Domain mismatch: general models can struggle on specialized vocabulary.
- Vectors are model-specific; switching models means re-embedding everything.
- Chunking strongly affects quality: too large dilutes, too small loses context.
Key takeaways
- An embedding is a dense vector that places similar meanings close together in space, letting machines compare meaning instead of matching exact words.
- Static word embeddings (word2vec, GloVe) gave way to contextual and sentence embeddings that vary with context and represent whole passages.
- Cosine similarity is the standard metric for text embeddings; on normalized vectors, cosine, dot product, and Euclidean ranking agree.
- Embeddings power semantic search and RAG by retrieving the passages closest to a query vector, and multimodal models like CLIP extend this across text and images.
- Quality depends on model choice, dimensionality, context length, and chunking; switching models requires re-embedding the whole corpus.
Frequently asked questions
Related terms
Related reading
Sources
- Efficient Estimation of Word Representations in Vector Space (word2vec), Mikolov et al., 2013
- GloVe: Global Vectors for Word Representation, Pennington, Socher, and Manning, EMNLP 2014
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Reimers and Gurevych, EMNLP-IJCNLP 2019
- MTEB: Massive Text Embedding Benchmark, Muennighoff et al., 2022
- OpenAI: New embedding models and API updates (text-embedding-3, dimensions, Matryoshka)
- CLIP: Learning Transferable Visual Models From Natural Language Supervision, Radford et al., 2021
- Distance Metrics in Vector Search, Weaviate
Put the idea into practice
MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.
Try MemX Free