Entity Memory and Knowledge Graph Extraction: Advanced LLM Memory Architectures and Lineage

This chapter explores the architectural evolution of Large Language Model (LLM) memory systems. It traces the lineage of memory from simple, stateful message lists to sophisticated hybrid models, analyzing the trade-offs between context preservation, computational token costs, and engineering complexity.

1. The Evolution of LLM Memory: The Memory Lineage

In AI engineering, memory is not a monolithic component. To build production-grade agentic workflows, developers must understand the Agent Memory Lineage—the evolutionary path of how LLMs retain, discard, and synthesize information over time.

While basic agentic loops are easy to construct using modern foundation models, the true engineering differentiator lies in the surrounding infrastructure: memory management, security, governance, and context engineering. Over the next 3 to 5 years, the most significant opportunities in AI engineering will center on this infrastructure layer rather than the raw cognitive capabilities of the models themselves.

2. Conversation Buffer Memory (CBM): The Naive Baseline

Conversation Buffer Memory (CBM) represents the earliest and simplest form of LLM memory. It was widely used in early, non-agentic LLM applications (such as early LangChain implementations).

Architectural Design

CBM maintains a raw, sequential list of every message exchanged during a session.

• The system message is pinned at the first index position (Index 0).
• Subsequent user inputs and AI responses are appended sequentially to the end of the list.

Trade-offs

• Pros: Perfect Recall. Because every word of the conversation is preserved verbatim, the LLM has access to the exact historical context.
• Cons: Context Bloat. As the conversation continues, the size of the prompt sent to the LLM grows exponentially, leading to prohibitive API costs and latency spikes.

3. The Problem of Context Bloat and Token Economics

Using raw Conversation Buffer Memory as a standalone memory layer in production is financially and computationally unsustainable.

As the conversation progresses, the token count increases non-linearly. Each new turn requires sending the entire accumulated history back to the LLM API.

The Mathematics of Token Bloat

In a typical 10-turn conversation, the token footprint scales dramatically:

• Turn 1: 149 tokens
• Turn 10: 976 tokens

This represents a $6.6\times$ to $7\times$ growth factor in token consumption over just ten turns.

If a system handles thousands of concurrent users, this $7\times$ multiplier translates directly to exponential API cost scaling and degraded response times. Therefore, production systems must implement more sophisticated, hybrid memory layers.

4. Sliding Window Memory: The Fixed-Horizon Approach

To mitigate context bloat, Sliding Window Memory introduces a strict eviction policy based on a fixed conversational horizon, defined by the parameter $K$.

How It Works

The system only retains the last $K$ turns (where 1 turn = 1 User message + 1 AI response). As new messages arrive, the oldest messages are permanently evicted from the active context window.

Limitations

While Sliding Window Memory caps token costs at a predictable upper bound, it suffers from complete historical amnesia. Any context, preference, or entity introduced prior to the last $K$ turns is permanently lost to the model.

5. Summary Buffer Memory: The Hybrid Memory Model

Summary Buffer Memory is a hybrid architecture designed to solve the trade-offs of both Conversation Buffer Memory (high cost, perfect recall) and Summary Memory (low cost, lossy recall).

It maintains the most recent conversational turns verbatim while compressing older history into a dynamically updated running summary.

Dual-Region Design

This dual-region design splits the context window into two distinct zones:

1. The Verbatim Buffer: Keeps the most recent messages word-for-word, ensuring precise short-term context.
2. The Summary Store: Holds a running summary of all evicted messages, providing long-term historical awareness.

6. The Mental Model: The Long Phone Call Analogy

To understand why Summary Buffer Memory is highly effective, consider how humans process long-term conversations.

Real-Life Analogy: The Long Phone Call

Imagine you are on a 30-minute phone call with a friend:

• Short-Term Memory (Verbatim Buffer): You can recall the exact last two or three sentences your friend said word-for-word. This allows you to respond naturally to their immediate point.
• Long-Term Memory (Summary Store): You cannot recall the exact sentences spoken 20 minutes ago. However, you retain a synthesized mental summary of the key facts discussed (e.g., "They bought a new car," "They are planning a trip to Spain").

Summary Buffer Memory mimics this human cognitive process by combining precise short-term recall with synthesized long-term context.

7. Under the Hood: The Summary Buffer Workflow and Prompt Assembly

The execution flow of a Summary Buffer Memory system relies on a continuous background loop of eviction, summarization, and assembly.

The Step-by-Step Execution Flow

Prompt Assembly Formula

The final prompt payload sent to the LLM is constructed dynamically:

8. Engineering Challenges: Transition Thresholds and Token Budget Splits

Implementing Summary Buffer Memory in production requires careful tuning of two core engineering variables:

1. The Transition Threshold

The transition threshold is the exact point (measured in tokens or message count) at which messages age out of the active buffer and are sent to the summarizer.

• Too low: Messages are summarized too quickly, causing the agent to lose immediate conversational nuance.
• Too high: The system behaves like a raw conversation buffer, risking context bloat and high API costs.

2. The Token Budget Split

Developers must decide how to partition the total available context window. For example, in an $8\text{K}$ token context window, a developer might allocate:

• $2\text{K}$ tokens for the running summary.
• $4\text{K}$ tokens for the verbatim buffer.
• $2\text{K}$ tokens reserved for the system prompt and model generation.

Enterprise Suitability

While Summary Buffer Memory is highly effective for medium-scale enterprise solutions (such as coaching bots, customer support agents, and management assistants), it is not a silver bullet. Massive-scale systems (e.g., large retail banking customer service systems handling millions of users) require more advanced, decoupled memory architectures to scale efficiently.

9. Comparative Synthesis of Core Memory Architectures

10. Advanced Memory Paradigms: Episodic, Semantic, Procedural, and Self-Reflection

As AI engineering moves beyond basic buffers, production architectures are transitioning to cognitive memory frameworks. These systems mimic human memory structures and use specialized routing layers to query the correct memory store.

The Cognitive Memory Stack

• Episodic Memory: Focuses on capturing the sequence of events, experiences, and specific user interactions over time.
• Semantic Memory: Retains structured facts, concepts, and relationships, often extracted and stored as a Knowledge Graph (Entity Memory).
• Procedural Memory: Stores the rules, execution steps, and workflows that the agent must follow to complete tasks.
• Self-Reflection Memory: Allows the agent to evaluate its own performance, analyze past mistakes, and update its execution strategies. This approach is heavily utilized by advanced coding assistants (such as Claude Code) to refine code generation.
• Memory Routing: An LLM-driven routing layer that analyzes incoming prompts to determine which memory store (episodic, semantic, or procedural) to query for the most relevant context.

11. Summary and Key Takeaways

• Infrastructure is the Differentiator: The next phase of AI engineering (3 to 5 years) will be defined by memory infrastructure, context engineering, security, and governance, rather than raw foundation model capabilities.
• Raw Buffers Do Not Scale: Conversation Buffer Memory provides perfect recall but suffers from severe context bloat, showing up to a $7\times$ token increase over just 10 turns.
• Sliding Windows Introduce Amnesia: Sliding window memory caps token costs but completely erases historical context beyond the configured $K$ threshold.
• Summary Buffer is a Cognitive Hybrid: By keeping recent messages verbatim and summarizing older history, Summary Buffer Memory balances short-term precision with long-term awareness.
• The Phone Call Mental Model: Summary Buffer Memory mirrors human memory: we remember the last few sentences verbatim (short-term) but retain only synthesized facts from earlier in the conversation (long-term).
• Tuning is Critical: The primary engineering challenges of hybrid memory are setting the transition threshold (when to evict and summarize) and managing the token budget split.
• Advanced Memory is Multi-Dimensional: Modern agentic architectures are moving toward specialized memory stores—episodic, semantic (knowledge graphs), procedural, and self-reflection—coordinated by intelligent memory routing layers.