Memory Management

TL;DR

An LLM is stateless — it remembers nothing between API calls. Every "memory" an agent appears to have is something your code re-supplies on each request. Memory management is the discipline of deciding what to keep, where to keep it, and what to inject back into the context window for the next turn.

The mental model has two axes:

• Short-term memory = the context window. Finite, fast, expensive per token, gone when the request ends. This is the agent's working memory / RAM.
• Long-term memory = an external store (vector DB, document store, SQL, a plain file). Effectively unbounded, cheap to store, but you must retrieve the right slice to bring it into context.

Cognitive-science taxonomy maps onto agents cleanly:

• Working memory — the active context window (what the model can "see" right now).
• Episodic memory — logs of what happened (past conversations, action traces).
• Semantic memory — facts about the world / the user (vectorized docs, a knowledge graph).
• Procedural memory — how to behave (system prompt, tool definitions, learned guidelines).

The core techniques for staying inside the context window: sliding window, summarization, retrieval (RAG over memory), and TTL eviction. Every one is a cost/latency vs. fidelity tradeoff, and the tokens you carry are literally your bill: total prompt = uncached + cache-write + cache-read tokens.

---

Simple explanation + analogy

Memory in an agent is a desk and a filing cabinet.

• The desk is the context window. You can only fit so much on it. Everything on the desk is instantly usable, but the desk is small and "renting" desk space (tokens) is expensive.
• The filing cabinet is long-term storage. It holds everything, cheaply, forever — but nothing in the cabinet helps you until you walk over, find the right folder, and put it on the desk.

Working with an agent is a constant juggle: as the desk fills up, you must decide what to sweep off (sliding window), what to shrink into a sticky-note summary (summarization), and what to go fetch from the cabinet because it's suddenly relevant (retrieval). And old folders you'll never touch again should be shredded on a schedule (TTL) so the cabinet doesn't cost a fortune.

The whole art is: keep the desk small, keep the cabinet cheap, and move the right folder onto the desk at the right moment.

---

Diagram

                        AGENT MEMORY ARCHITECTURE

   ┌──────────────────────── SHORT-TERM (context window) ──────────────────────┐
   │  [ system prompt ] [ tool defs ] [ summary ] [ recent turns ] [ new msg ] │
   │  procedural mem      procedural    compressed   sliding window   input     │
   │  ◄──────────────── finite token budget B (e.g. 200K) ──────────────────►   │
   └───────────────▲────────────────────────────────────────▲──────────────────┘
                   │ retrieve (top-k)                        │ append + evict
                   │                                         │
   ┌───────────────┴──────────────┐          ┌───────────────┴──────────────┐
   │       LONG-TERM STORE         │          │   IN-MEMORY SESSION BUFFER    │
   │                               │          │   (per session_id)            │
   │  ┌─────────┐  ┌────────────┐  │          │   messages[-WINDOW:]          │
   │  │ SEMANTIC│  │  EPISODIC  │  │          │   last_active timestamp       │
   │  │ vectors │  │ chat logs  │  │          │        │                      │
   │  │ (facts) │  │ (events)   │  │          │        ▼ TTL purge            │
   │  └─────────┘  └────────────┘  │          │   evict if idle > TTL         │
   └───────────────────────────────┘          └───────────────────────────────┘
        unbounded, cheap, must retrieve             bounded, fast, expires

   Techniques:  sliding window · summarization · retrieval (RAG) · TTL eviction

---

How it works (deep)

Short-term = the context window (working memory)

Because the API is stateless, "short-term memory" is just the message list you resend each turn. It holds the procedural memory (system prompt, tool definitions), any running summary, and the recent turns. It is bounded by the model's context window and — more practically — by your token budget, since every token in it is billed on every call.

The failure mode is obvious: append every turn forever and you either hit the context limit or your cost/latency explode. So you manage it with four techniques.

Technique 1 — Sliding window

Keep only the last W messages (or last W turns). Oldest messages fall off the front as new ones arrive. Simple, deterministic, O(1) per turn, and it bounds cost. The cost: you lose anything older than the window — the agent forgets the beginning of a long conversation. Good default for chat where recency dominates.

Technique 2 — Summarization

When the buffer gets large, replace the old turns with an LLM-generated summary, then keep summarizing recursively (a "rolling summary"). This preserves gist across a much longer horizon than a raw window at a fraction of the tokens. Costs: an extra LLM call (latency + money), and lossy compression — details and exact quotes get smoothed away. Best when long-range coherence matters more than verbatim recall. Modern APIs offer server-side compaction that does this for you when the context nears a threshold.

Technique 3 — Retrieval (RAG over memory)

Store everything in a long-term vector/doc store; each turn, embed the query and pull the top-k most relevant memories into context. This scales to effectively unlimited history because you only pay for what you retrieve. This is how semantic and episodic long-term memory are actually used. Costs: an embedding + vector-search round trip (latency), and retrieval-quality risk (miss the relevant memory → the agent "forgets" despite having it stored). Best when the total history is huge but only a small, query-dependent slice is relevant per turn.

Technique 4 — TTL eviction

Long-term memory that lives forever costs forever. Attach a time-to-live to sessions/entries and purge idle ones on a schedule. This bounds storage cost and, for privacy-sensitive apps, bounds data retention. This is the pattern in the code below.

The four cognitive memory types (interview gold)

The key insight (MemGPT's framing): working memory is not one type among equals — it's the mandatory activation space every other type must pass through before the model can act. Episodic/semantic/procedural memory are inert until something pages them into the context window. MemGPT's whole contribution was treating this like an OS's virtual memory: explicit page-in / page-out between the context window and external stores.

---

The math

Context-window token budget

Let B be the usable token budget (model context window minus your desired output headroom). On each turn the prompt must satisfy:

$$ T_{\text{sys}} + T_{\text{tools}} + T_{\text{summary}} + \sum_{i=1}^{W} T_{\text{turn}i} + T{\text{retrieved}} + T_{\text{output}} ;\le; B $$

Every term is a lever:

• T_sys + T_tools — fixed procedural overhead. Keep it stable and cache it (see below).
• T_summary — compressed history; you trade an LLM call for a huge drop in this term.
• Σ T_turn — the sliding window; bounded by W.
• T_retrieved — top-k memories; bounded by k and chunk size.
• T_output — reserve enough or the answer truncates.

Sliding-window cost. If each turn averages t̄ tokens and you keep a window of W turns, per-request history cost is ≈ W · t̄ and is constant regardless of conversation length. A naive append-everything strategy grows linearly: turn n costs ≈ n · t̄. Over an N-turn conversation, naive total is O(N²), sliding window is O(N·W).

The token bill. Total prompt tokens split into three price tiers:

$$ T_{\text{total}} = T_{\text{uncached}} + T_{\text{cache-write}} + T_{\text{cache-read}} $$

Cache reads cost roughly 0.1× the base input price and cache writes about 1.25× (for a 5-minute TTL). This is why keeping the stable prefix (system prompt, tool defs, frozen summary) byte-identical at the front of the context is a memory-management decision, not just a perf trick: a stable prefix caches, and cached history is ~10× cheaper than re-sending it cold.

---

Real code

This is a ConversationMemory — a per-session buffer built on LangChain's InMemoryChatMessageHistory, with a sliding window and TTL eviction. It's the whole short-term-memory story in ~70 lines.

"""
Per-session conversation memory backed by LangChain's InMemoryChatMessageHistory.

For production, swap to RedisChatMessageHistory:
    from langchain_redis import RedisChatMessageHistory
    def get_session_history(session_id):
        return RedisChatMessageHistory(session_id, redis_url=REDIS_URL)

TTL: sessions idle longer than CHAT_SESSION_TTL are evicted by purge_idle_sessions(),
which is called periodically from the startup lifespan in main.py.
"""
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import AIMessage, HumanMessage
from core.config import CHAT_MEMORY_WINDOW, CHAT_SESSION_TTL   # defaults: 20, 3600

# { session_id: (history, last_active_timestamp) }
_store: Dict[str, Tuple[InMemoryChatMessageHistory, float]] = {}

def get_session_history(session_id: str) -> InMemoryChatMessageHistory:
    """Return (or create) the history for a session, refreshing its TTL."""
    if session_id not in _store:
        _store[session_id] = (InMemoryChatMessageHistory(), time.time())
    history, _ = _store[session_id]
    _store[session_id] = (history, time.time())        # touch → refresh idle clock
    return history

def add_exchange(session_id: str, human: str, ai: str) -> None:
    """Append one human/AI pair and enforce the sliding window."""
    history = get_session_history(session_id)
    history.add_message(HumanMessage(content=human))
    history.add_message(AIMessage(content=ai))
    if len(history.messages) > CHAT_MEMORY_WINDOW:      # ← SLIDING WINDOW
        history.messages = history.messages[-CHAT_MEMORY_WINDOW:]

def purge_idle_sessions() -> int:
    """Evict sessions idle longer than CHAT_SESSION_TTL. Returns count removed."""
    cutoff = time.time() - CHAT_SESSION_TTL            # ← TTL EVICTION
    stale = [sid for sid, (_, last) in _store.items() if last < cutoff]
    for sid in stale:
        del _store[sid]
    return len(stale)

What to notice, from an interview standpoint:

1. The sliding window is one line — history.messages[-CHAT_MEMORY_WINDOW:] (keep last 20 messages). That's the entire cost bound on short-term memory.
2. TTL is decoupled from access — every get_session_history touches the timestamp, and a background job (purge_idle_sessions, wired into the FastAPI startup lifespan) sweeps anything idle past CHAT_SESSION_TTL (1 hour). This bounds memory and storage, not just context.
3. The docstring names the production swap — InMemoryChatMessageHistory → RedisChatMessageHistory. In-memory dies with the process; Redis survives restarts and scales horizontally. Same interface, one function changed. That "how do I make this durable?" answer is what interviewers probe.

For long-term memory, the workflow goes to a vector store (semantic memory over destination/preference vectors) and retrieves the top-k relevant items per query — the retrieval technique above — rather than stuffing everything into the window.

---

Real-world example

A single-agent trip-recommendation workflow cleanly separates the memory types. Short-term working memory is the ConversationMemory buffer above: a 20-message sliding window with a 1-hour TTL, keyed by session_id. Long-term semantic memory is a vector store (preference vectors + destination embeddings); the recommendation chain retrieves the relevant destinations per query rather than carrying them in context. Episodic memory (what a group decided) is persisted to SQLite as Task/TaskComment — the durable record of an event, retrieved on demand. Notice the split by cost profile: the hot, recency-dominated conversation lives in a cheap bounded window that expires; the large, query-dependent knowledge lives in a store you retrieve from; the durable decisions go to SQL. That's the whole taxonomy realized in one app — and the reason a group chat can run for many turns without the token bill growing quadratically or the process leaking sessions forever.

A fixed multi-step review pipeline shows the procedural + episodic end of the spectrum: its "memory" is a versioned, checksummed run artifact bundle under runs/<run_id>/ — every node's structured output persisted so a run can be inspected, resumed, or re-scored. That's episodic memory as an append-only, reproducible log, not a chat buffer — the right shape when the "conversation" is a multi-step pipeline whose intermediate state must survive and be auditable.

---

Interview questions companies actually ask

Q1. LLMs are stateless — so how does a chatbot "remember" my name? [easy] Your application resends the relevant history in the prompt on each call. The "memory" is entirely in the message list you reconstruct per request; the model itself holds no state between API calls. (Working memory in LLMs)

Q2. Distinguish short-term and long-term memory in an agent. [easy] Short-term = the context window (working memory): finite, fast, expensive per token, ephemeral. Long-term = an external store (vector DB, doc store, SQL): effectively unbounded, cheap, durable — but you must retrieve the relevant slice into context to use it. (Memory in the age of AI agents)

Q3. Name the four memory types and give an agent example of each. [medium] Working (active context window), episodic (what happened — chat/action logs), semantic (facts — vector DB / knowledge graph), procedural (how to behave — system prompt, tool defs, weights). E.g. working = current turn; episodic = "user rejected sushi last week"; semantic = "user is vegan"; procedural = "search before answering price questions." (Types of AI agent memory, Semantic vs episodic vs procedural)

Q4. Compare sliding window vs summarization vs retrieval. When each? [medium] Sliding window: keep last W turns — O(1), deterministic, but forgets anything older; use for recency-dominated chat. Summarization: compress old turns into a rolling summary — preserves gist far longer, costs an LLM call, lossy; use when long-range coherence matters. Retrieval: store all, pull top-k per query — scales to unbounded history, costs an embedding+search round trip, risks retrieval misses; use when history is huge but only a small slice is relevant per turn.

Q5. What is MemGPT's core idea? [medium] Treat the context window like an OS's virtual memory: it's finite RAM, and external stores are disk. The system explicitly pages information in and out of the context window as needed. The reframe: working memory is the mandatory activation space every other memory type must pass through — nothing helps until it's paged in. (Memory systems in AI agents)

Q6. Write the context-budget inequality and name each lever. [hard] T_sys + T_tools + T_summary + Σ T_turn + T_retrieved + T_output ≤ B. Levers: keep sys/tools stable & cached; summarize to shrink history; bound the window (W); bound retrieval (k, chunk size); reserve output headroom. Every term you shrink is tokens you don't pay for on every call.

Q7. How does memory strategy change your cost bill? [hard] Append-everything grows history cost linearly per turn (O(N²) over an N-turn chat); a sliding window makes it constant (O(N·W)). Beyond that, total tokens split into uncached / cache-write / cache-read tiers — cache reads are ~0.1× and writes ~1.25× the base input price. Keeping a byte-stable prefix (system prompt, tool defs, frozen summary) at the front makes it cacheable, so re-sent history costs ~10× less. In production traces, system prompts dominate input tokens — huge caching leverage. (LLMOps guide 2026)

Q8. Why attach a TTL to memory, and how do you implement it without hurting active sessions? [medium] TTL bounds storage cost and data-retention exposure. Implement it by timestamping last-access, refreshing on every touch, and running a periodic background sweep that evicts only sessions idle past the TTL (see purge_idle_sessions above). Active sessions keep getting touched, so they're never swept.

Q9. An in-memory session store loses everything on restart. How do you make it durable and scalable? [medium] Swap the backing store behind the same interface — e.g. InMemoryChatMessageHistory → RedisChatMessageHistory (or a DB). Redis survives restarts, is shared across instances (so any worker can serve any session), and supports native TTL. The application code that reads/writes memory doesn't change — only the store factory.

Q10. Retrieval-based memory failed to surface a relevant fact. How do you debug it? [hard] It's a retrieval-quality problem, not a "forgetting" problem — the fact is stored but wasn't fetched. Check: embedding model / query quality, chunk size and boundaries (fact split across chunks), top-k too small, similarity threshold too strict, or a stale/missing index entry. Add a recency or importance re-ranker, or a hybrid (keyword + vector) retriever. Measure it with a retrieval eval set (does top-k contain the gold memory?) — the same gold-set discipline you use for the agent's answers.

---

When to use / tradeoffs

Rules of thumb: default to a sliding window; add summarization when conversations outgrow it; add retrieval when the knowledge (not the chat) is large; always attach TTL to durable stores; keep the prompt prefix byte-stable so it caches. Don't reach for a vector DB when a 20-message window solves it, and don't try to cram a knowledge base into the context window when retrieval is what you want.

---

Summary + related articles

Memory management is the art of keeping the context window (working memory) small while backing it with long-term stores (semantic, episodic, procedural) you retrieve from on demand. The techniques — sliding window, summarization, retrieval, TTL — are each a cost/latency vs. fidelity trade, and the token budget inequality is the equation that governs all of them. The ConversationMemory above (20-message window + 1-hour TTL, swappable to Redis) is the pattern in miniature; the token bill is the reason to care.

Related articles:

• Production Agents (6-5) — prompt caching, context management, and cost control at scale.
• Agent Evaluation (6-5) — evaluating retrieval quality and the cost/latency of memory.
• Autonomous Agents (6-5) — why long-horizon autonomy makes memory the bottleneck.
• Tool Use (6-1) & Orchestration (6-4) — where tool defs (procedural memory) and shared state live.