Vector Ingestion, Embedding Models, and Chunk Optimization

1. Why Do We Need Long-Term Memory in Agentic Systems?

In production AI architectures, conversational memory cannot live solely in a single short-term session buffer. When a user interacts with an AI agent across multiple distinct sessions or days, the system must bridge those sessions seamlessly to maintain continuity and provide an efficient, personalized experience.

Real-Life Analogy: The Bank Teller vs. The Amnesiac Receptionist

Imagine walking into a bank. If you dealt with an amnesiac receptionist every single day, you would have to re-explain your entire account history, open loan applications, and past disputes from scratch. By contrast, a professional bank teller opens your customer profile instantly before you even finish speaking, referencing your active tickets and pending refunds. Long-term memory provides AI agents with this exact institutional continuity.

Consider the practical behavioral difference in a customer support environment:

Without Long-Term Memory: The customer states, "I called yesterday about my damaged headphones." The AI responds, "I do not have any record of a previous conversation. Could you describe this issue again?" (Frustrating, repetitive, and wasteful of tokens).
With Long-Term Memory: The AI has the open ticket, customer name, and pending refund status pre-loaded before the conversation begins. It responds, "I can see your refund is still pending approval. Let me check the status for you."

This reduces resolution time, eliminates friction, and drastically lowers token costs per session by avoiding redundant conversational turns.

2. Long-Term Memory Architecture & Data Types

To implement robust long-term memory, an agentic system categorizes data into specialized persistence stores rather than dumping everything into flat files or raw message logs.

Structure & Concept Flow

Incoming Customer Interaction

+--> Active Issues / Open Tickets ----> PostgreSQL (Real-time updates)

+--> Behavioral Patterns ------------- Periodic Batch Job (Repeat callers, high-value status)

+--> Conversation Transcripts -------- Vector DB / Document Store (Structured summaries)

Core Data Categories

Active Issues: Open tickets and pending refunds live in relational databases (such as PostgreSQL) and update in real time.
Behavioral Patterns: Computed attributes (e.g., whether someone is a repeat caller or a high-value customer) that get updated periodically via background batch jobs.
Structured Summaries: Rather than storing raw transcripts of 200 messages (which create noise and bloat context), the agent stores structured summaries containing:
- Issue type
- Resolution actions taken
- Customer sentiment
- Commitments made

Note: Short-term memory manages the current active session, while long-term memory bridges sessions together. Both are required for a production-grade agentic workflow.

3. The Retrieval and Injection Pattern

The Large Language Model (LLM) does not connect directly to the database. Instead, application code handles retrieval and injects the context directly into the system prompt at the start of a session.

Sequence Flow

Structure & Concept Flow

Session Start

Application Code queries DB (Customer profile, recent tickets, active issues)

Application Code formats retrieved data into a clean text block

Text block injected into LLM System Prompt

LLM generates context-aware response without querying DB directly

4. The Resource Allocation Problem & Context Engineering

The context window is a fixed-size resource (e.g., 128,000 tokens in modern models), and all operational components compete fiercely for space inside it.

Core Components Competing for Context Space

System prompt & persona definitions
Tool definitions and schemas
Retrieved documents from RAG (Retrieval-Augmented Generation)
Long-term context injected from databases
Growing conversation history with every user/assistant turn

Context Engineering is the discipline that manages this competition and allocation—functioning identically to memory management in an operating system, where a fixed amount of RAM is shared across multiple competing processes.

5. Worked Example: Token Budget Breakdown (128K Context Window)

Let us analyze a concrete token budget allocation for a customer support AI agent operating inside a 128,000 token limit:

Structure & Concept Flow

Total Context Capacity: 128,000 Tokens

+-- Fixed Allocations (4,200 Tokens Committed Before Conversation Starts):

| +-- System Prompt: 2,000 tokens

| +-- Tool Definitions: 1,200 tokens

| +-- Response Generation Reserve: 1,000 tokens

+-- Per-Session Allocations:

| +-- Customer Context (Database): ~1,000 tokens per query

| +-- RAG Retrieval (3 chunks): ~3,000 tokens

+-- Remaining Budget for Conversation History: ~119,800 tokens

Assuming a blended average of 1,400 tokens per conversational turn (accounting for a mix of simple Q&A turns and heavier turns involving tool calls and reasoning), the theoretical maximum is roughly 85 turns.

6. Proactive Context Compression Strategies

Waiting until the context window is 95% full is dangerous; a single tool call returning a large dataset can easily consume 3,000 to 4,000 tokens in one exchange and blow past the limit before compression runs. Therefore, budgeting should trigger compression at roughly 60% capacity (e.g., around 71,888 tokens used, triggering summarization around turn 51).

When compression triggers, three layered strategies come into play:

Compression Strategy	Mechanism	Cost / Complexity	Rationale
1. Selective Action	Replace "no results found" tool outputs with short placeholders; strip verbose reasoning traces.	Cheapest (No extra LLM call needed)	Preserves core conclusions while eliminating dead weight.
2. Summarization	Compress the older half of the conversation history into a concise summary (~2,000 tokens).	Moderate (Requires one additional LLM call)	Preserves key facts while reclaiming massive block space.
3. Document Compression	Extract only relevant sentences from a `1,000`-token RAG chunk down to `200` relevant tokens before injection.	Highest Complexity	Recovers significant budget by filtering out irrelevant retrieved noise.

7. Reasoning Scratchpads: Decoupling Thinking from Output

When complex tasks require deep evaluation, exposing raw internal monologue degrades the user experience. A Reasoning Scratchpad is a separate, hidden working space where the agent processes intermediate reasoning before outputting the final user-facing response.

Before vs. After Scratchpad Implementation

Without Scratchpad (Exposed Monologue):

"Order date is June 2nd. Current date is June 5th. 3 days is within the 15-day return window. Product category is eligible. Order amount is below auto-approval threshold. Therefore, your refund has been initiated..."
With Scratchpad (Clean End-User Experience):

"Your refund of $45 has been initiated. It will appear in your account within three to five business days."

Implementation Trade-offs

Two-LLM-Call Approach:
- Call 1 asks the model to think step-by-step in a scratchpad block.
- Call 2 receives the scratchpad output as internal context and generates the clean response.
- Trade-off: Doubles latency and cost.
Single-Call Structured Output:
- The model outputs reasoning in one JSON field and the response in another.
- Trade-off: More efficient, but requires strict, reliable structure output parsing.

8. Summary and Key Takeaways

Short-Term vs. Long-Term Memory: Short-term memory manages active session turns, while long-term memory bridges past interactions across multiple sessions.
Structured Summaries Over Raw Logs: Storing structured attributes (issue type, resolution, sentiment, commitments) prevents context bloat compared to saving raw chat transcripts.
Retrieval and Injection Pattern: Application code queries databases and injects structured text into system prompts; the LLM does not query databases directly.
Context Engineering as OS Memory Management: Fixed-size context windows (128K tokens) require strict token budgeting across system prompts, tools, RAG chunks, and history.
Trigger Compression Early: Set compression thresholds around 60% capacity (e.g., Turn 51) to prevent large tool outputs from overflowing the context limit.
Layered Compression: Combine selective action on tool results, conversation summarization, and document chunk compression to reclaim token budgets.
Reasoning Scratchpads: Hide complex multi-step logical derivations from the end user, presenting only polished, actionable outcomes.
Selective Scratchpad Usage: Apply scratchpads to complex eligibility checks, financial refunds, and multi-step reasoning, while bypassing them for simple factual lookups.