Long-Term Persistence, Summarization, and Scratchpads

Every time an AI agent handles a new request, it starts completely blank. It does not remember the customer's name from two messages ago, what they called about last week regarding a damaged order, or whether they prefer email or phone notifications. This chapter explores how to solve this limitation by engineering context, managing short-term conversation memory, and persisting long-term state across sessions.

1. Why Do We Need Agent Memory? (The Stateless Problem)

A common mistake in agent design interviews is assuming that Large Language Models (LLMs) possess built-in memory of a conversation.

Note: LLM API calls are fundamentally stateless. The model does not "remember" past turns. Instead, the entire conversation history is re-sent to the model on every single API call, forcing the model to reread everything from scratch within its fixed token context window.

Without explicit memory management, agents suffer from severe functional gaps:

• Zero Session Continuity: If a customer chats for an hour and exceeds the model's context window, the API call fails entirely, crashing the chat session mid-stream.
• Zero Cross-Session Continuity: When a session ends, the context resets to zero. If the same customer returns tomorrow, the agent has no memory of past interactions, forcing the user to repeat themselves from scratch.

2. Core Mental Model: Managing Conversation History as a Resource

Instead of treating the conversation history as an unmanaged list that grows infinitely until it breaks, production-grade architectures treat conversation history as a tightly managed resource.

Real-Life Analogy: The Human Receptionist vs. A Scratchpad

Think of an LLM as a receptionist with severe short-term memory loss who relies entirely on a running clipboard. Every time a caller speaks, the receptionist reads the entire clipboard from the very first greeting. If the clipboard gets too long, it takes too long to read, costs too much money in paper, and eventually runs out of physical space. To fix this, the receptionist must periodically take older notes off the clipboard, summarize the key points (e.g., "Customer is John Doe, called about a broken widget, refund pending"), and archive the detailed transcripts in a filing cabinet for future reference.

3. Naive Append vs. Production-Grade Memory Management

In a naive implementation, developers simply append every user message and model response to an in-memory list and pass the entire array into every API request.

# Naive Approach: Unbounded Message List
messages = []

def chat_with_agent(user_input):
    messages.append({"role": "user", "content": user_input})
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages
    )
    messages.append({"role": "assistant", "content": response.content})
    return response.content

The Overflow Problem in Practice

As the conversation progresses through multiple turns, token consumption explodes:

• Turn 1: Message list consumes ~2,500 tokens.
• Turn 15: Message list grows to ~23,000 tokens.
• Turn 30: Message list reaches ~48,000+ tokens, exceeding the token threshold or context limit, causing the API call to fail entirely.

4. Short-Term Memory Strategies: Sliding Windows and Summarization

To prevent context overflow within a single session, engineers use specific truncation and compression patterns.

Strategy A: Sliding Window Memory

A sliding window keeps only the most recent $N$ turns and drops everything older.

class SlidingWindowMemory:
    def __init__(self, max_turns=10):
        self.messages = []
        self.max_turns = max_turns

    def add_message(self, role, content):
        self.messages.append({"role": role, "content": content})
        # Trim older messages if we exceed the limit (each turn = user + assistant)
        if len(self.messages) > self.max_turns * 2:
            self.messages = self.messages[-self.max_turns * 2:]

• Trade-off: Simple to implement and caps token usage. However, if a critical fact was established in turn 2 and you are now on turn 30, that early context is permanently lost.

Strategy B: Sliding Window with Summarization

Before dropping older turns, an agentic summarizer compresses them. When token estimates cross a specific threshold, the code takes the older half of the conversation, sends it to a smaller, faster model (e.g., gpt-4o-mini), and replaces the old turns with a concise summary.

• What the summary preserves: Customer's name, core issue, decisions made, and tools invoked.
• Token savings: Reduces 20,000 tokens of raw chat history down to a dense 500–2,000 token summary.

5. Comparison of Short-Term Memory Approaches

6. Long-Term Memory: Bridging Sessions with Databases

While short-term summarization manages active chats, everything is wiped when the session ends. Long-term memory moves critical information out of the conversation stream and into persistent storage databases. When a new session starts, the agent queries the database and injects relevant context before the customer even speaks.

Data Categorization and Storage Mapping

7. Context Engineering & Reasoning Scratchpads

Beyond conversation history and user profiles, complex agentic reasoning requires dedicated working memory spaces.

Context Engineering

Context engineering is the rigorous discipline of deciding what fits inside the limited context window and what gets left out. It encompasses:

• Dynamic prompt assembly based on user intent.
• Selective tool definition injection (avoiding listing 20+ tools at once to prevent token waste and degradation of tool selection accuracy).
• RAG chunk filtering and re-ranking.

Reasoning Scratchpads

For complex multi-step tasks (e.g., debugging code or financial planning), agents utilize a hidden reasoning scratchpad.

• The agent outputs intermediate thoughts within a structured block (e.g., <scratchpad> ... </scratchpad> or a thought JSON field) before generating the final action or tool call.
• This hidden workspace allows the model to break down complex problems step-by-step without exposing raw internal monologues to the end user unless required.

Summary and Key Takeaways

• Stateless LLM Architecture: LLM API calls have no built-in memory; the entire conversation history must be re-sent with every single request.
• The Overflow Hazard: Unmanaged conversation histories grow exponentially, eventually exceeding token limits and causing mid-session agent crashes.
• Sliding Windows: Truncates older messages to cap token usage, but risks permanently losing crucial early context.
• Summarization Pattern: Compresses older conversation turns using a smaller background model, preserving critical facts while reclaiming massive token space.
• Long-Term Persistence: Relies on databases (PostgreSQL, Redis) to store user profiles, interaction history, and preferences across separate sessions.
• Context Engineering Discipline: Treats the context window as a constrained resource by selectively including only relevant tool definitions, retrieved documents, and memory blocks.
• Reasoning Scratchpads: Provides agents with a hidden intermediate workspace to plan complex actions and process multi-step workflows accurately.