Short-Term vs Long-Term Agent Memory

Tell an agent your production database is Postgres, and forty messages later that sentence has scrolled out of context for good. Short-term memory is the live context window the model reads each turn; long-term memory is external storage it reaches into through retrieval. Naming the two tiers is the easy part. The mechanism that actually makes two-tier memory work is the layer between them: the promotion policy that decides what graduates from short-term to long-term, the decay rules that drop what no longer matters, and the recall loop that pulls long-term records back into context when relevant again. That layer is rarely explained, and it is where most agents quietly forget. Yet a fresh session next week still answers with pg_dump, not mysqldump, and nobody told it again. The difference between those two outcomes is one policy, not a bigger window. This guide traces exactly how.

That middle layer is where long-term memory lives or dies. Get the promotion and eviction policy wrong and you either save garbage forever or forget the one fact that mattered. The Postgres fact above is the thread: this guide follows it through promotion, compression, eviction, decay, and recall, so the mechanism stops being a black box.

Short-term vs long-term: the one-line difference

Short-term memory equals everything currently inside the model's context window: the system prompt, the running conversation, retrieved snippets, and tool outputs. It is fast because the model attends to it directly, and it is volatile because it vanishes the moment the window fills or the session ends. Long-term memory is anything stored outside the window, in a database the agent queries on demand. It persists across sessions but is invisible to the model until something pulls it back in.

The MemGPT paper frames this split cleanly by borrowing from operating systems. It calls the in-window tier main context, the prompt tokens the model can read during inference, and the out-of-window tier external context, data that requires an explicit retrieval step before the model can see it. The whole point of the design is to create the illusion of unlimited context while the underlying model still runs on a fixed, finite window.

Why the context window is not enough by itself

Bigger context windows do not remove the need for long-term memory, for two reasons: cost and attention quality. Every token in the window is paid for on every single turn, so stuffing an entire history into the prompt scales badly. Worse, models do not read a long window evenly: a fact buried in a 200k-token prompt can be present and still ignored.

The 'Lost in the Middle' study by Liu and colleagues showed that model accuracy is highest when relevant information sits near the beginning or the end of the input, and degrades sharply when the model has to use information buried in the middle. Performance also drops as the context grows longer overall, and this held even for models marketed as long-context. A fact technically present in a 200k-token window can still be functionally invisible. That is why the difference between the context window and a real memory system matters: presence in the prompt is not the same as usable recall.

The same mechanic explains why chatbots forget. When a session exceeds the window, the oldest turns fall out, and unless those turns were promoted to durable storage first, they are gone for good. That failure pattern gets a full treatment in why ChatGPT forgets what you told it, and the deeper distinction between raw context and a memory layer in context window vs memory.

Also on MemX

AI Explained

Agent Memory vs RAG: The Real Difference

12 min read→

AI Explained

How AI Long-Term Memory Actually Works

12 min read→

AI Explained

Does Deleting a Chat Erase AI Memory?

10 min read→

Promotion: what graduates from short-term to long-term

Promotion is the decision to copy something out of the volatile window into durable storage before it gets evicted. This is the hidden mechanism that makes long-term memory work, and it runs on a policy, not magic. The policy answers one question on each turn: is this worth keeping after the session ends?

There are two common ways to make that call. The first is agent-driven: the model itself decides, using tools, what to write. MemGPT does this through self-editing memory, where the working context is a fixed-size block of text writeable only through function calls the model issues. When the agent learns a durable fact, it calls a function to append or replace text in that block, deliberately choosing what survives.

The second is extraction-driven: a separate step reads the conversation and pulls out salient facts automatically. Mem0 takes this route, dynamically extracting and consolidating important information from ongoing conversations rather than waiting for the model to flag it by hand. In practice, a strong promotion policy filters hard. Stable preferences, identities, decisions, and corrections graduate. One-off pleasantries and transient tool output do not.

Eviction and decay: what gets dropped and when

Eviction is what removes data from the short-term window; decay is what lets long-term records fade in importance over time. They are different mechanisms with the same purpose: stop both tiers from drowning in noise.

In MemGPT, eviction from short-term memory is driven by token pressure. The recent message history sits in a FIFO queue. When usage crosses a warning threshold, around 70 percent of the window, the system signals the agent to save anything important to durable storage. When the window is full, a flush evicts roughly half of the messages from the queue. Nothing is silently lost: the system folds evicted messages into a recursive summary, built from the existing summary plus the dropped messages, which then sits at the front of the queue. That is decay and compression happening in one step.

Long-term decay is softer. A well-designed store does not just accumulate; it lets newer information revise older information. A-Mem, an agentic memory system inspired by the Zettelkasten note-taking method, builds this in through memory evolution: when a new memory is added, it can trigger updates to the attributes and links of existing memories, so the network continuously refines itself instead of freezing the first version of a fact forever. Recency, access frequency, and explicit supersession are the usual signals that govern which records stay salient and which fade.

Summarize vs store verbatim: the compression tradeoff

Every promotion faces one choice: keep the raw text exactly, or compress it into a summary. Each direction loses something. Verbatim storage preserves detail and exact wording but bloats the store and makes retrieval noisier. Summaries are compact and cheap to recall but discard specifics you might need later, and a bad summary can quietly drop the one detail that mattered.

• Store verbatim when exact wording is load-bearing: credentials, configuration values, named entities, legal or numeric facts, and direct user corrections.
• Summarize when you need the gist across many turns: the arc of a long conversation, recurring themes, or a multi-session project status.
• Hybrid approaches keep both, a short summary for fast recall plus a pointer to the raw record for when precision is required.
• Recursive summarization, as in MemGPT, is the streaming version: each batch of evicted text gets merged into one running summary so the compressed history never grows without bound.

The recall loop: pulling long-term memory back into context

Recall is the reverse of promotion: it moves a record from long-term storage back into the short-term window so the model can use it. Long-term memory is inert until this loop runs, because the model only ever reasons over what is in its context. The loop has three steps: detect that stored knowledge is relevant, retrieve the right records, and inject them into the prompt.

Retrieval runs on semantic similarity: the store converts facts into embeddings and finds the closest matches by approximate nearest-neighbor search, the same retrieval-augmented generation machinery covered in detail in how AI long-term memory works. The part specific to two-tier memory is what feeds that search.

The recall loop is also where a bad promotion policy comes back to bite you. If the store is full of low-value verbatim chatter, similarity search surfaces noise, and you reintroduce the lost-in-the-middle problem by crowding the window with marginally relevant text. Clean promotion is what keeps recall sharp. For a fuller walkthrough of the storage-plus-retrieval architecture, see how AI long-term memory works.

Lifecycle of a single fact across both tiers

Trace one fact, 'the user's production database is Postgres, not MySQL,' through the whole system. The two-tier framing usually skips this walkthrough.

• Arrival: the user mentions Postgres mid-conversation. The fact now lives only in short-term memory, sitting in the recent-message queue inside the context window.
• Promotion: the policy evaluates it. A future session that assumed MySQL would generate wrong commands, so the fact clears the promotion bar. It is written to long-term storage, either by the model calling a memory function or by an extraction step pulling it out.
• Compression choice: because the exact engine name is load-bearing, it is stored close to verbatim ('DB engine: Postgres'), not summarized into a vague 'user discussed databases.'
• Eviction from short-term: later turns fill the window. The original Postgres message is evicted from the FIFO queue and folded into a recursive summary. It is gone from short-term memory, but safe in long-term storage.
• Decay and supersession: weeks later the user migrates to a managed Postgres service. The new fact links to and revises the old record rather than contradicting it, the kind of memory evolution A-Mem describes.
• Recall: in a brand-new session the user asks for a backup command. The query embeds, nearest-neighbor search matches the stored Postgres record, and the loop injects it back into the context window. The agent answers with pg_dump, not mysqldump, having never been told again in this session.

Every arrow in that lifecycle is a policy decision, not an automatic property of having two tiers. Promotion decided the fact was worth keeping. The compression choice decided it stayed precise. Decay decided the migration revised rather than duplicated. Recall decided it came back at the right moment. Remove any one of these and the two-tier diagram stops working in practice.

Data structures that hold each tier

Short-term and long-term memory use different data structures because they optimize for different things: order and recency versus semantic lookup.

• Short-term: a token-bounded sequence, typically a FIFO message queue plus a small fixed-size working block of text. The queue preserves conversational order; the working block holds a handful of always-present facts the agent edits directly.
• Long-term, semantic: a vector database. Records become embeddings and are retrieved by approximate nearest neighbor search on meaning rather than keywords.
• Long-term, relational: a knowledge graph linking entities and relationships, useful for multi-hop questions where the answer chains across several facts. Mem0's graph variant and A-Mem's linked notes both take this shape.
• Long-term, raw log: a plain message database, MemGPT's recall storage, that keeps the full transcript so nothing is irrecoverably lost even after summarization.

Getting the structures and policies right pays off measurably. Mem0 reports a 26 percent relative improvement in an LLM-as-a-judge accuracy metric over OpenAI's memory feature on the LOCOMO benchmark, while also cutting p95 latency by 91 percent and saving more than 90 percent of token cost compared with passing full context. Selective promotion plus targeted recall beats stuffing everything into the window on accuracy, speed, and cost at the same time.

Building this middle layer yourself means owning the promotion policy, the eviction and decay rules, the summarize-versus-verbatim logic, and the retrieval loop, then keeping them consistent across every app a user touches. MemX provides that as a persistent memory layer: facts promoted once are stored, decayed by recency and supersession, and recalled into context across sessions and tools, so the short-term and long-term tiers stay connected without each application reinventing the policy.