Hybrid Retrieval, PGVector Indexing, and Re-ranking

1. Why Do We Need This? (The Limitations of LLMs and Statelessness)

When designing production-grade agentic AI systems, developers encounter foundational limitations in Large Language Models (LLMs). Understanding these constraints is critical for system architecture and technical interviews.

Statelessness: LLMs retain zero memory between API calls. Every invocation requires passing the entire conversation context explicitly.
Knowledge Cutoffs: Model weights are frozen at training time. An LLM trained in January has no awareness of today's updated corporate refund policies, inventory status, or pricing adjustments.
Hallucination Risk: When faced with missing information, models generate plausible-sounding but factually incorrect text.

Retrieval-Augmented Generation (RAG) directly solves these limitations by fetching relevant external documents at query time and injecting them into the prompt, grounding the model's response in verifiable data.

2. Core Mental Model and Architecture of Agent Memory

Managing memory in AI agents requires balancing short-term conversational context with long-term persistent storage.

Structure & Concept Flow

[Customer Request / Incoming Message]

+-----------------------+

| Token Budget Manager | <--- Triggers compression at 60% capacity

+-----------------------+

| |

v v

[Short-Term Memory] [Long-Term Memory]

(Sliding Window + (SQL Database: Profiles,

Summarization) Preferences, Past Sessions)

Real-Life Analogy: The Customer Support Representative

Think of an LLM as a human support agent working with a blank desk at the start of every phone call.

Short-term memory is the scratch paper on the desk where the agent notes down what the customer said in the last five minutes.
Long-term memory is the customer relationship management (CRM) database open on the computer screen, showing that this customer called twice last week and prefers email communication.
Scratch pads are private calculation notes the agent writes down before giving an official refund decision.

3. Four Common Memory Mistakes in Production & Interviews

Production systems and technical interviews frequently expose architectural flaws in how memory and context are managed.

Mistake	Description / Failure Mode	Production Fix
1. No Memory Management	Passing the entire raw conversation history on every API call without token tracking. Around turn 50-60, degradation begins; around turn 80, the API returns a context length exceeded error.	Token budget allocation with a sliding window and automated summarization.
2. Storing Everything in Long-Term Memory	Saving every raw message, tool result, and agent response to a database, then injecting 50 pages of raw history on return visits. Signal gets buried in noise.	Store structured summaries (issue type, resolution, action taken, commitments made) instead of raw transcripts.
3. Zero Cross-Session Memory	Starting every session completely fresh. Customers repeat themselves across multiple calls, causing frustration.	Persistent storage for customer profiles and past interactions retrieved at session start.
4. Using Short-Term Memory for Durable Preferences	Relying on chat history for preferences (e.g., "I prefer email"). The preference vanishes in the next session.	Intentional design: extract durable preferences and write them to a long-term database store.

4. Context Engineering and Token Budget Allocation

Context engineering is the deliberate allocation of the LLM's fixed context window across competing operational categories.

Structure & Concept Flow

+-------------------------------------------------------------------------+

| 128,000 Token Model Budget |

+-------------------+-----------------+------------------+----------------+

+-------------------+-----------------+------------------+----------------+

Key Quantitative Benchmarks for Interviews

Total Model Budget: Modern models offer a 128,000-token window, leaving roughly 120,000 tokens for active conversation after fixed allocations.
Compression Threshold: Trigger summarization and sliding window pruning at 60% of the remaining budget (roughly around turn 51).
Summarization Cost: Generated summaries consume 500 to 2,000 tokens per cycle.
Long-Term Context Injection: Injected user profiles and historical summaries consume 500 to 1,500 tokens.

5. Scratch Pads and Audit Trails

For complex multi-step reasoning tasks (such as evaluating refund eligibility), agents utilize scratch pads.

Dual Purpose of Scratch Pads

Response Quality: Forcing explicit step-by-step reasoning improves logical coherence.
Audit Trail: Creates a traceable record of how the agent reached a specific conclusion, which is essential for observability and compliance audits.

Note: Single-step FAQ responses and speed-critical interactions where extra latency is unacceptable should bypass scratch pads.

6. Interview Reference Card: Anti-Patterns vs. Strong Answers

When questioned in system design interviews, avoid naive statements and articulate production-grade architectural solutions.

Naive Interview Statement	Why It Fails	Strong Production Answer
"The LLM will remember our conversation."	Ignores model statelessness; fails system design scrutiny.	"The LLM is stateless. I manage context explicitly by appending turns to a message list using a sliding window to stay within budget."
"I will just use a bigger context window."	Larger context windows incur significantly higher costs and latency, and do not solve cross-session persistence.	"I optimize token usage via sliding windows and structured summaries while reserving large windows strictly for dense RAG retrieval."
"I will store everything in a vector database."	Confuses semantic vector retrieval with structural state management and transactional user profiles.	"I use a relational SQL database for structured session state and user profiles, reserving vector stores strictly for semantic document retrieval."
"Memory is just passing the chat history."	Ignores the two-layer architecture separating short-term and long-term memory.	"I implement a two-tier memory architecture combining short-term conversational sliding windows with long-term persistent database profiles."

7. Connecting Memory, RAG, and Cost Optimization

The memory system acts as the foundational layer upon which subsequent architectural modules are built:

Structure & Concept Flow

Section 5: Context Engineering & Token Budgets

+---> Section 6: RAG (Retrieval Augmented Generation)

| |--- Largest consumer of context budget

| |--- Competes directly with chat history & tool results

+---> Section 10: Observability (Auditing scratch pad reasoning)

+---> Section 11: Cost Optimization (Managing summarization token expenses)

RAG is the single largest consumer of the context budget. Every retrieved text chunk consumes tokens in the prompt, creating a direct engineering tradeoff between how much historical context is retained and how much reference documentation can be injected.

Summary and Key Takeaways

Statelessness Management: LLMs require explicit context injection; agents must manage conversation history programmatically using sliding windows.
Two-Tier Memory Architecture: Combine short-term session chat history with long-term persistent SQL storage for user profiles and durable preferences.
Token Budget Discipline: Allocate budget categories explicitly, triggering compression and summarization at 60% capacity to prevent overflow errors.
Structured Summaries over Raw Transcripts: Store extracted issue types, resolutions, and commitments rather than hundreds of raw dialogue turns.
Scratch Pads for Complex Reasoning: Utilize hidden intermediate reasoning steps to improve decision quality and establish audit trails for compliance.
RAG as the Grounding Layer: Overcome LLM hallucinations and knowledge cutoffs by retrieving external policy documents at query time.
Context Window Competition: RAG document chunks compete directly with chat history, tool definitions, and system prompts for limited token budget space.
Interview Positioning: Never claim LLMs remember state; articulate precise sliding-window, token budgeting, and dual-tier storage strategies.