Episodic and Procedural Memory Frameworks

This chapter explores the architectural patterns, trade-offs, and engineering implementations of memory systems within Large Language Model (LLM) applications. It covers the transition from basic short-term, RAM-bound conversational buffers to advanced, compressed summary buffers, token-constrained windows, and persistent vector-based long-term memory stores.

1. The Memory Problem in LLM Architectures

Large Language Models are inherently stateless. Each API call to an LLM is independent, meaning the model does not natively remember previous interactions. To simulate a continuous conversation, developers must pass the historical context back to the model with each new query.

However, this approach introduces a fundamental engineering challenge: the context window constraint.

CODE

User Query (Turn N) 
       + Historical Context (Turns 1 to N-1) 
       = Total Context Sent to LLM

As the conversation grows, the volume of historical tokens increases. This leads to:

Escalating API Costs: Most LLM providers charge per input and output token.
Latency Spikes: Processing larger context windows increases the time-to-first-token (TTFT) and overall generation latency.
Context Window Exhaustion: Every model has a hard limit on the maximum number of tokens it can process in a single execution.

To manage these constraints, engineers must implement memory management strategies that balance context retention (accuracy) against token consumption (cost and latency).

2. Sliding Window Memory

Sliding Window Memory restricts the active context to a fixed number of recent interaction "turns." A single turn typically consists of a user message and the corresponding assistant response.

The Eviction Mechanism ("Big Boss Eviction")

When the conversation history exceeds the pre-configured window size ( $K$ ), the oldest interaction is evicted from the active context window to make room for the newest turn.

Structure & Concept Flow

Initial State (Window Size K = 4):

[ Message 1 ] -> [ Message 2 ] -> [ Message 3 ] -> [ Message 4 ]

New Message (Message 5) Arrives:

[ Message 1 ] (EVICTED)

[ Message 2 ] -> [ Message 3 ] -> [ Message 4 ] -> [ Message 5 ] (NEW ACTIVE WINDOW)

Note: Eviction does not mean the message is permanently deleted from the application's database; rather, it is excluded from the payload sent to the LLM for the current turn.

The Forgetting Problem (Information Loss)

While Sliding Window Memory provides strict control over token usage, it introduces a severe forgetting problem. If a user shares a critical fact (e.g., their name or project requirements) in Message 1, and the conversation extends to Message 5, the LLM will no longer have access to Message 1. When queried about that information, the LLM will state, "I do not have this information."

3. Summary Memory & Summary Buffer Memory

To mitigate the hard cut-offs of sliding windows, Summary Memory compresses historical context instead of discarding it.

The Summarization Cycle

Instead of passing raw historical messages, the system uses an LLM to generate a running summary of the conversation. When a new user message arrives, the LLM receives the old summary and the new messages to generate the response, updating the running summary in the process.

Structure & Concept Flow

[Old Summary] + [New Messages]

[Summarizer LLM]

[New Compressed Summary] -> Saved to Summary Store

Summary Buffer Memory

Summary Buffer Memory combines the sliding window and summarization approaches. It maintains a raw buffer of the most recent messages. Once the buffer exceeds a specified token threshold ( $T$ ), the oldest messages are evicted from the active buffer and compiled into a running summary via an LLM summarization call.

Structure & Concept Flow

New User Message

Check Buffer Size (Token Threshold)

+---> If Exceeded: Evict Oldest Messages ---> Summarizer LLM (Merges Old Summary + Evicted) ---> Update Summary Store

| |

+---> If Not Exceeded: Keep in Buffer |

| |

v v

Prompt Assembly <---------------------------------------------------------------------------------------+

(System Prompt + Summary Store + Buffered Messages + New Message)

Target LLM Generates Response

4. The Mechanics of Summary Buffer Memory

The core advantage of Summary Buffer Memory is context preservation. However, it introduces a unique set of engineering challenges:

Lossy Compression & Progressive Degradation

Summarization is inherently a lossy compression technique. As the conversation progresses over dozens of turns, the running summary is repeatedly summarized. This progressive summarization can lead to:

Context Drift: Gradual loss of specific details, numbers, or names.
Semantic Decay: The summary becomes too generalized to be useful for precise tasks.
Latency and Cost Overheads: Every eviction event triggers an auxiliary LLM call to update the summary, increasing operational costs and response latency.

The KV Cache Bottleneck

In production environments, maintaining long-term context in memory is constrained by the GPU's Key-Value (KV) Cache. The KV cache stores the attention keys and values for past tokens to avoid recomputing them during generation.

As context grows, the KV cache can consume massive amounts of GPU memory. Advanced engineering techniques, such as shrinking the KV cache, are often employed to drop less important attention heads or tokens, allowing systems to maintain long-term context without exhausting GPU resources.

5. Token Buffer Memory

Token Buffer Memory manages short-term history based on an exact token budget rather than the number of conversational turns.

Sliding Window vs. Token Buffer Memory

Sliding Window Memory uses turns as its control unit. Because message lengths vary, the actual token count per payload is highly unpredictable.
Token Buffer Memory uses tokens as its control unit. It guarantees that the active context payload will never exceed a strict token limit.

Mathematical Formulation

The total input tokens sent to the LLM during a call can be calculated using the following formula:

\begin{aligned} \text{Input Tokens} &= \text{System Prompt Tokens} + \min(\text{History Tokens}, \text{Max Buffer Tokens}) \end{aligned}

This formula ensures strict budget precision, preventing unexpected API billing spikes and context window overflows.

LangChain Implementation

In LangChain, this pattern is implemented using the ConversationTokenBufferMemory class. Below is a realistic implementation showing how to restrict memory based on token limits using a local or remote LLM.

PYTHON

from langchain_openai import ChatOpenAI
from langchain.memory import ConversationTokenBufferMemory
from langchain.chains import ConversationChain

# Initialize the LLM
llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)

# Initialize Token Buffer Memory with a strict limit of 100 tokens
memory = ConversationTokenBufferMemory(
    llm=llm, 
    max_token_limit=100, 
    return_messages=True
)

# Create the Conversation Chain
conversation = ConversationChain(
    llm=llm,
    memory=memory,
    verbose=True
)

# First interaction
conversation.predict(input="Hi, my name is Alice. I am a Senior Security Engineer.")

# Second interaction (forces eviction of older tokens if limit is exceeded)
conversation.predict(input="Can you recommend three security guardrail frameworks for LLMs?")

# Inspect the current state of the memory buffer
print(memory.load_memory_variables({}))

6. Vector Store Memory: Transitioning to Long-Term Storage

While sliding windows, summary buffers, and token buffers are excellent for short-term memory (session-bound, typically living in volatile RAM), they fail when applications require persistence across sessions.

Structure & Concept Flow

Short-Term Memory (RAM-bound) ---> Resets when session ends

Long-Term Memory (Vector DB) ---> Persists indefinitely across sessions

Vector Store Memory provides a persistent, scalable solution for long-term memory:

Structure & Concept Flow

User Query

+---> Vector Search Query ---> [ Vector Database ] (ChromaDB, Pinecone, FAISS)

| |

| v

| Retrieve Top-K Semantically Similar Past Interactions

| |

v v

[ Prompt Assembly ] <-------------------+

Target LLM (Generates contextually enriched response)

Core Workflow

Tokenization & Embedding: Conversational turns are converted into high-dimensional vector embeddings using an embedding model.
Indexing: These vectors are stored in a specialized vector database (e.g., Chroma DB, FAISS, Pinecone, Weaviate, Milvus).
Semantic Retrieval: When a user sends a new query, the system embeds the query and performs a cosine similarity or Euclidean distance search against the vector store.
Context Injection: The most relevant historical interactions are retrieved and injected into the prompt template as background context.

7. Comparative Synthesis of Short-Term Memory Techniques

Memory Strategy	Control Unit	Budget Precision	Latency Profile	Primary Failure Mode	Best Use Case
Sliding Window	Conversational Turns ( $K$ )	Approximate (varies with message length)	Low (no auxiliary LLM calls)	Hard context loss (abrupt forgetting of older turns)	Simple, short-lived chat sessions with predictable turn lengths.
Summary Buffer	Token Threshold ( $T$ )	Moderate	High (requires periodic summarization LLM calls)	Lossy compression, semantic decay, and context drift	Long, complex conversations where high-level context must be preserved.
Token Buffer	Raw Tokens ( $N$ )	Exact	Low (no auxiliary LLM calls)	Hard eviction of oldest tokens once the budget is exhausted	Cost-sensitive production applications with strict API budget limits.
Vector Store	Semantic Relevance	Variable (depends on retrieval parameters)	Moderate (depends on database search latency)	Retrieval of irrelevant context due to semantic overlap issues	Multi-session applications, personalized assistants, and long-term user profile retention.

8. Real-World Engineering Trade-offs & Niche Optimization

When designing production-grade AI agents, choosing the right memory architecture requires balancing several engineering constraints:

RAM-Bound Session Memory vs. Persistent Storage

Short-term memory techniques (Sliding Window, Summary Buffer, Token Buffer) typically reside in the application server's RAM or a fast cache like Redis.

Pros: Extremely fast read/write times; zero database overhead.
Cons: Memory resets when the session ends or the server restarts.
Ideal Use Cases: Temporary customer support bots, single-session Q&A tools, and transactional tools.

Hybrid Memory Architectures

To build robust, production-ready systems, engineers rarely rely on a single memory technique. Instead, they implement Hybrid Memory Systems:

Structure & Concept Flow

+-----------------------+

| New User Message |

+-----------------------+

+---------------+---------------+

| |

v v

+---------------------+ +---------------------+

| Short-Term Memory | | Long-Term Memory |

| (Token Buffer) | | (Vector Database) |

+---------------------+ +---------------------+

| |

| (Exact recent context) | (Semantic historical facts)

+---------------+---------------+

+-----------------------+

| Prompt Assembly |

+-----------------------+

| Target LLM |

+-----------------------+

This hybrid approach ensures the model has exact, high-fidelity access to the immediate conversation flow while retaining the ability to recall key facts from past sessions.

9. Summary and Key Takeaways

Statelessness of LLMs: LLMs do not natively retain state; memory must be managed externally by appending historical context to each API request.
Sliding Window Eviction: This approach maintains a fixed number of conversational turns. It is simple to implement but suffers from sudden information loss when older turns are evicted.
Summary Buffer Compression: This technique compresses older interactions into a running summary. While it preserves high-level context, it introduces latency overhead and suffers from progressive semantic decay.
Token Buffer Precision: By managing history using exact token counts, Token Buffer Memory provides predictable API costs and prevents context window exhaustion.
Short-Term Memory Boundaries: Memory strategies that run in RAM are session-bound and reset upon session termination, making them ideal for temporary or transactional chatbots.
Vector Store Persistence: Vector databases enable long-term, persistent memory across multiple sessions by storing and retrieving semantically relevant past interactions.
Hybrid Memory is Standard: Production-grade AI systems combine short-term token buffers (for immediate conversational flow) with long-term vector stores (for historical recall) to optimize cost, latency, and accuracy.