Agent Boundaries, Failure Modes, and Resilient Retries

This chapter explores how production-grade AI systems balance knowledge retrieval strategies (RAG vs. Fine-Tuning), architect knowledge bases for multi-document enterprise environments, scale into multi-agent coordination patterns, and maintain robust boundaries under failure conditions.

1. RAG vs. Fine-Tuning: Core Mental Model and Decision Framework

When building domain-specific AI applications, teams often reach for fine-tuning under the misconception that it injects factual data. The foundational distinction in agent architecture separates behavior from knowledge:

• Fine-Tuning: Modifies how the model behaves—its writing style, reasoning patterns, or domain-specific terminology.
• Retrieval-Augmented Generation (RAG): Gives the model dynamic access to facts it was not trained on, allowing instant updates without retraining.

Note: If a model "does not know" company data, RAG is almost always the correct first architectural step. Fine-tuning should be reserved for persistent behavioral or linguistic gaps that cannot be solved via prompt engineering and few-shot examples.

Production Hybrid Approach

Enterprise systems rarely rely on just one strategy. Instead, production architectures layer these mechanisms:

1. RAG: For large, volatile factual knowledge bases (catalogs, support articles).
2. Fine-Tuning: For specialized tone, strict persona enforcement, or proprietary domain terminology.
3. Long Context / System Prompts: For small, critical reference materials such as absolute safety rules and core system instructions.

2. Designing a Customer Support Knowledge Base (Capacity & Chunking)

To understand real-world capacity constraints, consider a mid-scale e-commerce customer support agent requiring access to four distinct categories of internal documentation:

Total raw content across the knowledge base equals approximately 650,000 tokens. Because this exceeds both GPT-4's context window (128k tokens) and Claude's window (200k tokens), long-context loading is unviable, making RAG mandatory.

Total Chunks and Embedding Economics

Using the strategies outlined above, the system generates roughly 5,330 total chunks to embed and store:

• FAQ Articles: 200 chunks
• Policy Documents: 80 chunks
• Product Catalog: 5,000 chunks
• Shipping Documentation: 50 chunks

At OpenAI's pricing of $0.02 per million tokens for text embedding models, embedding the entire 650,000-token knowledge base costs:

Re-embedding the entire knowledge base daily costs under $5.00 per year, rendering embedding expenses negligible.

3. Retrieval Configuration and Hybrid Search Parameters

Effective retrieval balances semantic intent with strict keyword exactness (such as model serial numbers or order IDs).

• Hybrid Search Alpha: Set to 0.65, slightly favoring semantic similarity while preserving keyword precision.
• Top-K Retrieval: Retrieves the top 3 chunks per query to balance context quality against the token budget.
• Relevance Threshold: A minimum threshold of 0.55 discards low-relevance results to prevent noise injection.
• Metadata Filtering: Routes queries dynamically (e.g., FAQ queries to FAQ chunks, product queries to catalog chunks) when user intent is explicit.

4. Token Budget Allocation for RAG Interactions

Maintaining ample context headroom ensures the agent can sustain extended multi-turn conversations without performance degradation.

This headroom easily supports average conversation lengths of 5 to 8 turns before context pressure becomes a factor.

5. Complete Query Walkthrough: Step-by-Step Flow

To observe the RAG architecture in motion, trace a customer asking: "Can I return headphones I bought two weeks ago?"

Real-Life Analogy: The Corporate Librarian

Think of the RAG system as a meticulous corporate librarian. When a customer asks a question, the librarian doesn't memorize every invoice and return policy in the warehouse (which would cause mental overload). Instead, they look up the exact three folders relevant to "headphones" and "returns," hand only those open folders to the agent, and say: "Base your answer strictly on these pages." If policy updates next week, the librarian swaps out Folder A for Folder B—requiring zero retraining of the agent's core reasoning engine.

6. Interview Reference Sheet: Core Concepts Summary

7. Common Interview Pitfalls and How to Avoid Them

• Pitfall 1: Recommending Fine-Tuning for Frequently Updated Data.

  • Interviewer Question: "How would you handle updating FAQ data that changes weekly?"
  • Incorrect Answer: "Fine-tune the model on the new FAQs."
  • Why it fails: Fine-tuning is static, slow, and expensive.
  • Correct Answer: "Use RAG, because document updates propagate instantly without retraining."

• Pitfall 2: Over-Retrieval Token Waste.

  • Incorrect Answer: "Retrieve the top 20 chunks and feed them all into the prompt to be safe."
  • Why it fails: It exhausts the context window, introduces noise, and degrades reasoning accuracy ("lost in the middle" phenomenon).
  • Correct Answer: "Retrieve the top 3 to 5 chunks filtered by a strict minimum relevance threshold (e.g., 0.55)."

8. Essential Key Numbers for System Design Interviews

Keep these operational figures memorized for technical interviews:

• 1,536 dimensions: Vector size for OpenAI's standard small embedding model (text-embedding-3-small).
• $0.02 / million tokens: Cost of generating embeddings.
• 512 tokens: Default chunk size for policy documents, paired with a 100-token overlap.
• 3 to 5 chunks: Optimal retrieval count for standard AI agent generation.
• 0.55: Minimum relevance threshold to filter out noise.
• 0.65: Hybrid search alpha setting balancing semantic and keyword matching.
• 200–500 ms: Additional latency introduced by neural rerankers.
• 2 to 3: Number of few-shot examples required (keeping total examples under 1,000 tokens).

9. Transition to Multi-Agent Systems and Coordination Patterns

Up to this point, our architecture has focused on single-agent systems—featuring one system prompt, one set of tools, and a single LLM handling all tasks. For many workflows, a single agent is optimal.

However, as system requirements grow, single agents encounter severe scaling bottlenecks:

• System prompts become excessively long.
• Tool lists expand beyond reliable function-calling capacity.
• The LLM struggles to juggle disparate responsibilities, leading to degraded decision quality.

To overcome these limits, enterprise architectures transition to multi-agent systems, utilizing three primary coordination patterns:

1. Supervisor-Worker (Hierarchical): A central controller router delegates sub-tasks to specialized worker agents.
2. Orchestrator Pipelines (Sequential): Agents execute tasks in a defined linear or DAG workflow.
3. Peer-to-Peer Communication: Autonomous agents negotiate and pass control directly among themselves based on real-time state.

Summary and Key Takeaways

• Separate Knowledge from Behavior: Use RAG for volatile factual knowledge and fine-tuning exclusively for persistent tone, style, and behavioral constraints.
• Tailor Chunking to Document Structure: Keep FAQs whole, split policies and shipping guides recursively by headers, and treat product catalog entries as standalone items.
• Embrace Hybrid Search: Balance semantic intent (alpha = 0.65) with keyword exactness to successfully retrieve technical terms and serial numbers.
• Protect Token Budgets: Restrict top-K retrieval to 3–5 high-relevance chunks (>= 0.55 threshold) to maintain ample conversational headroom.
• Zero-Retraining Updates: Up-to-date document stores allow instant behavioral adjustments in production without modifying model weights.
• Avoid Over-Retrieval: Dumping dozens of chunks into a prompt wastes context space and triggers attention degradation.
• Recognize Single-Agent Limits: When tool lists and prompt lengths bloat, transition cleanly to multi-agent supervisor or pipeline architectures.