Add Memory to your AI Agents | Context Management for LLMs

Large Language Models (LLMs) are inherently stateless; each API invocation executes in isolation without native retention of prior interactions. To build multi-turn conversational agents and complex agentic workflows, engineers must implement external context management architectures that store, format, and reinject conversation history into the model's prompt window. This chapter explores the mechanics of memory management in LLM systems, contrasting raw message retention with dynamic context summarization using LangChain.

1. The Core Problem: Stateless LLMs

By default, an LLM treats every incoming request as an independent event. When a user conducts a multi-turn conversation, the model does not store previous interactions on the server.

CODE

Turn 1:
User: "John is my colleague and he works as a software engineer."
LLM:  "Got it! How can I help you with John?"

Turn 2 (Without Memory):
User: "What is John's profession?"
LLM:  "I don't have any context about who John is. Could you please provide more details?"

Without an external state container:

The LLM cannot resolve co-references (e.g., "he", "that", "earlier").
Multi-step tasks lose state between user prompts.
User experience degrades because the user must repeatedly supply background information.

2. Mental Model: Context Injection

To give the illusion of memory, the host application maintains a historical message store and injects relevant prior turns into the prompt context payload before sending the request to the model.

Structure & Concept Flow

User submits Query

Application retrieves History from State Store

Application formats Payload: [System Prompt + History + Current Query]

LLM receives combined context and generates response

Application appends Query & AI Response back to State Store

When memory is equipped, Turn 2 functions correctly:

CODE

Turn 2 (With Memory):
Context Injected: "System: You are a helpful assistant. | User: John is my colleague and a software engineer. | AI: Understood."
User: "What is John's profession?"
LLM:  "John is a software engineer."

3. Two Core Strategies for Context Management

Context management techniques fall along a spectrum between absolute fidelity and token efficiency:

Full Chat History Injection (Naive Raw Memory): Appends every preceding human and AI message directly into the input array of the prompt.
Summarized Chat History (Context Compression): Distills older conversational turns into a compact summary once the history crosses a defined threshold, ensuring the model retains essential facts without overflowing the token budget.

4. Strategy 1: Naive Full Chat History Injection

In the raw history approach, the application keeps an array of interchange objects (HumanMessage and AIMessage) and appends them sequentially to the prompt on every request.

Structure & Concept Flow

Message 1 (Human): "Can you explain dynamic programming under 50 words?"

Message 2 (AI): "Dynamic programming breaks complex problems into subproblems..."

Message 3 (Human): "What did I ask you earlier?"

Combined Payload sent to LLM: [Message 1, Message 2, Message 3]

LLM Response: "You asked me to explain dynamic programming under 50 words."

Implementation: Raw History Injection with LangChain

PYTHON

import os
from dotenv import load_dotenv
from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

load_dotenv()

# Initialize Model and Base System Prompt
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Answer all questions to the best of your ability."),
    ("placeholder", "{messages}")
])

chain = prompt | llm

# Initialize State Store
chat_history = []

def interrogate_model(query: str, history: list, preserve_chat_history: bool = True) -> str:
    """
    Invokes the model with or without historical context based on a boolean flag.
    """
    current_message = HumanMessage(content=query)
    
    if preserve_chat_history:
        # Prepend historical turns before the current query
        messages_to_send = history + [current_message]
    else:
        # Isolate the current query (Stateless behavior)
        messages_to_send = [current_message]
        
    response = chain.invoke({"messages": messages_to_send})
    return response.content

# Turn 1: Seed the conversation
query_1 = "Can you explain dynamic programming under 50 words?"
response_1 = chain.invoke({"messages": [HumanMessage(content=query_1)]})

# Update history store
chat_history.append(HumanMessage(content=query_1))
chat_history.append(AIMessage(content=response_1.content))

# Turn 2: Query memory with flag toggling
query_2 = "What did I ask you earlier?"

# Case A: Memory Disabled
stateless_reply = interrogate_model(query_2, chat_history, preserve_chat_history=False)
# Output: "I'm sorry, but I don't have access to past interactions..."

# Case B: Memory Enabled
stateful_reply = interrogate_model(query_2, chat_history, preserve_chat_history=True)
# Output: "You asked me to explain dynamic programming under 50 words."

5. The Limitations of Raw History Retention

While raw history injection is straightforward to implement, it creates critical engineering bottlenecks in production systems:

Context Window Exhaustion: Every LLM operates within a fixed context limit (e.g., 8k, 32k, 128k tokens). Unbounded histories eventually exceed this limit, causing runtime exceptions.
Quadratic Cost Scaling: API pricing scales linearly with the total tokens sent per request. In a 20-turn conversation, Turn 1 is transmitted and billed 20 separate times:

\text{Total Tokens Billed} \approx \sum_{i=1}^{N} \text{Tokens}(\text{Turn}_i) \propto \mathcal{O}(N^2)

Latency Degradation: Time-to-First-Token (TTFT) increases proportionally with input prompt length.
Attention Dilution ("Lost in the Middle"): LLMs struggle to reliably retrieve specific facts buried in extremely long conversational payloads.

6. Strategy 2: Summarized Chat History

To mitigate context window saturation, systems employ Context Compression. Instead of passing every individual turn, the system tracks message count or token usage. When a defined threshold is reached (e.g., $\ge 4$ turns), an auxiliary summarization chain compresses older messages into a compact semantic digest.

Structure & Concept Flow

Raw History Exceeds Threshold (Length >= 4)

Summarization Prompt: "Distill the chat history into a single summary message."

Auxiliary LLM Call generates Context Summary

Payload for Main Query: [System Prompt + Context Summary + Current Query]

LLM generates informed response with minimal token overhead

7. Step-by-Step Implementation of Summarized Memory

The following pattern demonstrates a complete workflow with dynamic threshold-based summarization:

PYTHON

import os
from dotenv import load_dotenv
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

load_dotenv()

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Answer all questions to the best of your ability."),
    ("placeholder", "{messages}")
])

chain = prompt | llm

def summarize_chat_history(history: list) -> str:
    """
    Summarizes raw message history into a concise representation 
    if history length exceeds 4 messages.
    """
    if len(history) >= 4:
        summary_prompt = (
            "Distill the above chat messages into a single summary message. "
            "Include as many specific details as you can."
        )
        messages_to_summarize = history + [HumanMessage(content=summary_prompt)]
        summary_response = chain.invoke({"messages": messages_to_summarize})
        return summary_response.content
    return ""

def interrogate_model_with_summary(query: str, history: list, debug: bool = False) -> AIMessage:
    """
    Constructs the prompt payload dynamically using either raw history 
    or a compressed summary based on length thresholds.
    """
    current_message = HumanMessage(content=query)
    chat_messages = []
    
    if len(history) >= 4:
        if debug:
            print("\n[DEBUG] Threshold exceeded (>= 4 messages). Summarizing context...")
        summary_content = summarize_chat_history(history)
        
        # Inject the compressed context as a descriptive system/human context block
        summarized_context_message = HumanMessage(
            content=f"Summary of conversation so far: {summary_content}"
        )
        chat_messages = [summarized_context_message, current_message]
        
        if debug:
            print(f"[DEBUG] Compressed Payload Injected: {summary_content}\n")
    else:
        chat_messages = history + [current_message]
        
    response = chain.invoke({"messages": chat_messages})
    return response

def chat_with_model(query: str, history: list, debug: bool = False):
    """
    High-level orchestrator that executes the query and appends interactions 
    to the mutable history array.
    """
    
ai_response = interrogate_model_with_summary(query, history, debug=debug)
    
    # Update state store with raw turns
    updated_history = list(history)
    updated_history.append(HumanMessage(content=query))
    updated_history.append(AIMessage(content=ai_response.content))
    
    return ai_response.content, updated_history

8. End-to-End Multi-Turn Execution Trace

Here is an end-to-end trace running three sequential turns using chat_with_model:

PYTHON

history = []

# Turn 1: Low-context bootstrap
print("--- Turn 1 ---")
reply_1, history = chat_with_model(
    "Can you explain dynamic programming under 50 words?", 
    history, 
    debug=True
)
print(f"AI: {reply_1}\n")
# History size: 2 items (1 Human, 1 AI)

# Turn 2: Second domain inquiry
print("--- Turn 2 ---")
reply_2, history = chat_with_model(
    "Can you explain graph data structures under 50 words?", 
    history, 
    debug=True
)
print(f"AI: {reply_2}\n")
# History size: 4 items (2 Human, 2 AI) -> Reaches threshold

# Turn 3: Interrogation requiring contextual memory
print("--- Turn 3 ---")
reply_3, history = chat_with_model(
    "Can you tell me what did I ask till now?", 
    history, 
    debug=True
)
print(f"AI: {reply_3}")

Execution Log Breakdown:

CODE

--- Turn 1 ---
AI: Dynamic programming is a method for solving complex problems by breaking them down into simpler subproblems, solving each subproblem once, and storing their solutions to avoid redundant computations.

--- Turn 2 ---
AI: A graph data structure consists of nodes (vertices) connected by edges. They are used to represent pairwise relationships between objects and can be directed, undirected, weighted, or unweighted.

--- Turn 3 ---
[DEBUG] Threshold exceeded (>= 4 messages). Summarizing context...
[DEBUG] Compressed Payload Injected: The user requested concise explanations (under 50 words) for two concepts: dynamic programming and graph data structures. The assistant provided definitions focusing on subproblem memoization and node-edge relational topologies respectively.

AI: You asked me to explain two concepts: dynamic programming and graph data structures, both under a 50-word constraint.

9. Comparative Synthesis: Memory Strategies

Evaluation Dimension	Naive Full History Injection	Summarized Context Compression
Token Efficiency	Poor; grows continuously with conversation length ( $\mathcal{O}(N)$ per request)	High; bounds context size to a predictable, fixed ceiling ( $\mathcal{O}(1)$ relative to ceiling)
Context Fidelity	100% exact verbatim retention	Semantic retention; may lose non-critical peripheral tokens
Latency Profile	TTFT increases monotonically over conversation turns	Constant TTFT for main query; requires an auxiliary call on compression steps
System Complexity	Minimal (list append operations)	Moderate (triggers compression chains based on thresholds)
Failure Modes	Runtime context window overflow (`400 BadRequest`)	Potential loss of fine-grained detail during summarization

10. Summary and Key Takeaways

LLMs are Stateless: Large Language Models retain zero conversational state between distinct API invocations. Any multi-turn continuity is an application-level abstraction created by context injection.
Context Injection Mechanics: Conversation history is maintained within the application layer (e.g., Python runtime, Redis, SQL) and passed as an array of structured messages (SystemMessage, HumanMessage, AIMessage) alongside each new query.
The Raw History Trade-off: Sending raw message histories maintains complete verbatim accuracy, but introduces quadratic cumulative token costs, latency spikes, and risks context-window overflow.
Context Compression via Summarization: Compacting chat history into a semantic summary once an interaction crosses a size threshold (e.g., $\ge 4$ messages) keeps prompts concise while preserving essential facts.
Threshold Automation: Intelligent agent workflows dynamically route history through secondary distillation chains, swapping raw transcripts for structured summaries when managing long-running conversations.
Foundation of Agentic Memory: Effective context compression is the foundational building block for advanced agent architectures, including conversational memory, long-term memory retrieval (RAG), and persistent agent workflows.

Add Memory to your AI Agents | Context Management for LLMs | LangChain

Add Memory to your AI Agents | Context Management for LLMs

1. The Core Problem: Stateless LLMs

2. Mental Model: Context Injection

3. Two Core Strategies for Context Management

4. Strategy 1: Naive Full Chat History Injection

Implementation: Raw History Injection with LangChain

5. The Limitations of Raw History Retention

6. Strategy 2: Summarized Chat History

7. Step-by-Step Implementation of Summarized Memory

8. End-to-End Multi-Turn Execution Trace

Execution Log Breakdown:

9. Comparative Synthesis: Memory Strategies

10. Summary and Key Takeaways