AI Memory Layer

What is an AI memory layer?

An AI memory layer is the persistence and retrieval tier that sits between an LLM's short-term context window and long-term personalization. It captures durable facts and preferences from interactions, stores them outside the model in a database, and selectively injects the relevant items back into the prompt on future turns. The result is an agent or app that appears to remember a user across separate conversations, even though the underlying model itself remains stateless.

The need for a memory layer comes from a hard constraint: a model only knows what is inside its context window for the current request. Once a session ends or the window fills, that information is gone unless something external stored it. A memory layer is that external system. It decides what is worth keeping, where to keep it, and when to surface it, turning a one-shot chat into a continuous relationship.

Memory layers are usually organized around the cognitive-science distinction between memory types. Episodic memory records specific events ("the user asked about webhook setup on June 3"), semantic memory holds distilled facts ("the user writes TypeScript"), and procedural memory encodes learned behaviors or instructions. Vendors implement these on top of vector stores, knowledge graphs, key-value stores, or some combination, and a routing component picks the right backend for each read.

• Sits between the context window (short-term) and personalization/fine-tuning (long-term).
• Stores facts outside the stateless model so they survive across sessions.
• Commonly split into episodic, semantic, and procedural memory.
• Backed by vector databases, knowledge graphs, and/or key-value stores.
• Turns a stateless chat into a persistent, personalized experience.

How does the write path and read path work?

A memory layer has two pipelines. The write path runs after or during a conversation: it extracts candidate facts (typically with an LLM call that distills raw turns into atomic statements), embeds them into vectors, optionally links them to entities in a graph, deduplicates or updates against existing memories, and persists them. Mem0, for example, exposes a simple add operation whose pipeline extracts atomic facts, then classifies each one as add, update, delete, or no-op against what it already stores.

The read path runs before generation. Given the current user message, the system retrieves candidate memories (semantic vector search, often combined with keyword/BM25 matching and entity or recency boosting), reranks them so the most relevant rise to the top, and injects a compact selection into the prompt as context. Good systems cap how much they inject, because the goal is to spend as few tokens as possible while still surfacing what matters.

Conflict handling is where memory layers earn their keep. When a new fact contradicts an old one ("the user moved from Berlin to Lisbon"), the layer must update, supersede, or version the prior memory rather than store both. Temporal awareness matters here: Zep's design, described in its January 2025 arXiv paper, uses a temporal knowledge graph so the system can answer "what was true then" versus "what is true now" instead of returning stale facts.

Pseudo-config below sketches the two paths so the moving parts are concrete. The exact LLM, embedding model, and stores vary by vendor, but the shape is consistent across the category.

memory_layer:
  write_path:
    extract:
      model: llm            # distill turns into atomic facts
      mode: add_or_update    # dedupe against existing memories
    embed:
      model: text-embedding
    store:
      vector: true           # semantic recall
      graph: true            # entity relationships, temporal edges
      key_value: true        # exact-match lookups
  read_path:
    retrieve:
      strategy: hybrid       # vector + keyword + recency boost
      top_k: 20
    rerank:
      keep: 5                # only inject the best few
    inject:
      max_tokens: 800        # budget the context spend

• Write path: extract facts, embed, link entities, dedupe/update, store.
• Read path: retrieve, rerank, inject a small relevant slice into the prompt.
• Retrieval is usually hybrid: vector similarity plus keyword and recency signals.
• Conflicting facts must be updated or versioned, not blindly appended.
• Temporal modeling distinguishes "true then" from "true now."

How is an AI memory layer different from RAG and a vector database?

A memory layer overlaps with retrieval-augmented generation (RAG) but is not the same thing. RAG retrieves chunks from a static, mostly read-only corpus (documents, wikis, product manuals) to ground answers in external knowledge. A memory layer is read-write and personal: it continuously writes new facts derived from the ongoing interaction and retrieves them later, so its corpus grows and changes with each conversation. Put simply, RAG remembers documents, while a memory layer remembers the user.

A vector database is a component, not the whole layer. It provides approximate-nearest-neighbor search over embeddings, which is one retrieval mechanism a memory layer can use. But a memory layer adds the extraction logic, the dedupe and update rules, the entity graph, the conflict resolution, and the injection policy on top. You can build a memory layer using a vector DB, but a vector DB alone is not a memory layer.

The practical line between them is blurring. Production memory systems borrow RAG's reranking and hybrid search, and modern RAG systems borrow memory's incremental updates. The useful distinction is intent: RAG answers "what do the documents say," and a memory layer answers "what do I already know about this user and what did we agree on before."

• RAG retrieves from a static document corpus; a memory layer is read-write and personal.
• RAG remembers documents; a memory layer remembers the user across sessions.
• A vector database is one component, not the full memory layer.
• Memory layers add extraction, dedupe, conflict resolution, and injection policy.
• The two converge in practice but differ in intent and data lifecycle.

Build vs buy, and the vendor category

You can build a memory layer in-house: a vector store, an embedding model, an LLM extraction prompt, and some glue is enough for a prototype. The hard parts surface at scale, namely conflict resolution, deduplication, temporal reasoning, retrieval quality, latency, and multi-tenant isolation. Teams often start by building and switch to a vendor once these edges start costing more engineering time than the feature is worth.

An open category of memory-layer vendors emerged in 2024 and 2025. Mem0 (Apache-2.0 licensed) markets itself as a universal memory layer and reports strong scores on the LoCoMo and LongMemEval benchmarks. Zep, built on the open-source Graphiti engine, uses a temporal knowledge graph and reported 94.8% on the Deep Memory Retrieval benchmark in its January 2025 paper, against 93.4% for MemGPT. Letta, the framework that grew out of UC Berkeley's October 2023 MemGPT paper, treats the agent as an operating system that manages its own tiered memory through tool calls. Cognee is an open-source platform that builds a self-hosted knowledge graph for agent memory.

Note that these vendors benchmark on different tasks and definitions, so headline numbers are not directly comparable across products. Treat published scores as directional and validate against your own data and latency budget. The right choice depends on whether you want a self-managing agent runtime (Letta), a graph-centric temporal store (Zep, Cognee), or a drop-in add-and-search memory API (mem0).

Consumer-facing products embed their own memory layers too. ChatGPT, Claude, and Gemini each ship memory features, and personal-memory apps add a control and privacy layer on top. MemX, for example, is designed to be private by architecture, using per-user isolation and encryption at rest with on-device options rather than pooling everyone's memories in one shared store.

• Building is easy to prototype but hard at scale (conflicts, dedupe, latency, isolation).
• Mem0 is an Apache-2.0 add-and-search memory API.
• Zep uses a temporal knowledge graph via the open-source Graphiti engine.
• Letta descends from the October 2023 MemGPT paper and self-manages tiered memory.
• Vendor benchmarks use different tasks, so headline scores are not directly comparable.

How do you view, manage, or turn off an AI memory layer?

For a memory layer you build or operate, control means exposing the stored memories to the user and to operators. A well-designed system supports listing what it has saved, editing or deleting individual entries, and clearing everything. It should also support a memory-off mode where the read and write paths are skipped, so a session runs purely on the context window with nothing persisted.

For consumer products that ship a memory layer, the controls live in settings rather than code. In ChatGPT, saved-memory management and the on/off toggle are under Settings, then Personalization, per OpenAI's help docs as of mid-2026. Claude and Gemini expose comparable settings for their memory and personalization features. The exact labels change as these features evolve, so check the current settings screen rather than relying on a remembered menu path.

Privacy and governance are part of managing a memory layer, not an afterthought. Because the layer stores personal facts indefinitely, it should enforce per-user isolation so one user's memories never leak into another's retrieval, encrypt data at rest, and give clear delete semantics. A poisoned or manipulated memory store can also steer an agent's future behavior, so write-path validation and provenance tracking matter as much as the retrieval quality.

• Operators should expose list, edit, delete, and clear-all for stored memories.
• A memory-off mode should skip both read and write paths for a session.
• In ChatGPT, memory controls live under Settings, then Personalization (verify current labels).
• Per-user isolation prevents one user's memories from surfacing for another.
• Validate write-path inputs and track provenance to limit memory poisoning.