Multi-Agent Systems: Supervisor-Worker & Peer Orchestration

Overview

This chapter explores advanced architectural patterns for equipping Large Language Model (LLM) agents with external knowledge and behavioral guidelines. We examine the mechanics of Retrieval-Augmented Generation (RAG), search strategies (similarity, hybrid, and reranking), the separation of factual knowledge from behavioral conditioning via few-shot prompting, and a comparative analysis of RAG, Fine-Tuning, and Long-Context architectures for production agentic systems.

1. Relevance Score Thresholds in Vector Retrieval

When querying a vector database, an embedding function converts a user question into a vector, optionally filters by document type, and queries the database for the top-$K$ results. However, not every retrieved chunk is relevant. Establishing strict relevance score thresholds prevents hallucination and token waste.

Note: Setting a minimum relevance threshold (e.g., 0.55) ensures the agent receives clean context. If no chunks pass the threshold, the agent should explicitly acknowledge that it lacks the required information rather than hallucinating an answer.

2. Three Retrieval Patterns: Trade-Offs and Architecture

Basic vector similarity search is a starting point, but production systems require more robust retrieval patterns. We evaluate three primary approaches:

Pattern 1: Pure Similarity Search

• Mechanism: Embeds the question, finds closest vectors in the database, and returns top results based on cosine similarity.
• Strengths: Excellent semantic understanding (e.g., matching "how do I send back a product" to a chunk on "return policy" despite zero overlapping keywords).
• Weaknesses: Fails at exact keyword matching (e.g., looking up exact product serial numbers like model XR7200).

Pattern 2: Hybrid Search

• Mechanism: Combines vector similarity with traditional keyword matching (BM25 or full-text search). Final ranking blends both scores using a weighted parameter $\alpha$ (alpha).
• Mathematical Weighting:
  • An $\alpha$ of 1.0 represents pure vector search.
  • An $\alpha$ of 0.0 represents pure keyword search.
  • An $\alpha$ between 0.6 and 0.7 works well for most production applications.
• Strengths: Captures both semantic intent and exact alphanumeric strings (e.g., matching product ID XR7200 via keyword index and return instructions via semantic vector search).
• Weaknesses: Requires maintaining two separate indexes and tuning the blending weight ($\alpha$).

Pattern 3: Reranking (Two-Stage Retrieval)

• Mechanism:
  1. Stage 1: Fast similarity search retrieves a large candidate set (e.g., top 20 results).
  2. Stage 2: A cross-encoder model reranks those candidates by processing the query and each candidate chunk together as a single input.
• Strengths: Significantly higher precision than bi-encoder cosine similarity because the cross-encoder evaluates query-document token interactions jointly.
• Weaknesses: Computationally expensive. Cross-encoders are $10\text{ to }50\times$ slower per comparison, adding roughly $200\text{ to }500\text{ ms}$ of latency.
• Production Recommendation: Use hybrid search as the default for customer support. Add reranking exclusively for compliance-sensitive queries (e.g., fraud interpretation, refund eligibility) where retrieval accuracy directly governs critical agent decisions.

3. Knowledge (RAG) vs. Behavior (Few-Shot Examples)

A foundational architectural distinction in agent design is separating facts from behavior:

• RAG (Provides Facts): Retrieves specific policies, product details, and order information dynamically at query time. Content changes with every query.
• Few-Shot Examples (Provides Behavior): Teaches the agent format, tone, and reasoning approach. Examples remain static and are included in every prompt.

Behavioral Templates in Action

Consider two customer interactions handled by an AI agent:

1. Late Order Query:
   • Thought: Customer is asking about order status. I need to look up their order.
   • Action: Call order_lookup().
   • Response Format: Provide shipping date, carrier, estimated arrival, and tracking number.
2. Product Return Request:
   • Thought: I need to check the order date against the return policy before processing.
   • Action: Call order_lookup() and knowledge_search(return_policy).
   • Response Format: State policy window, compare order date, and communicate next steps clearly.

Production Token Economics of Few-Shot Examples

Few-shot examples consume fixed tokens on every single LLM call regardless of query relevance:

• Example calculation: Three detailed examples at 300 tokens each consume 900 tokens per call. At 10,000 calls per day, this equals 9 million tokens per day spent on examples alone.
• Rule of Thumb: Keep examples concise. Two to three well-chosen examples outperform six verbose ones. If examples exceed 1,000 tokens total, refactor and trim them.

4. Comparing Knowledge Injection Strategies: RAG vs. Fine-Tuning vs. Long-Context

When an LLM lacks required knowledge, architects choose between three primary strategies.

5. Architectural Decision Framework

Selecting the correct architecture depends entirely on update frequency, data size, and the nature of the information:

• RAG: Ideal for large, frequently updated factual knowledge bases.
• Fine-Tuning: Ideal for instilling behavioral patterns, tone of voice, or specialized linguistic formatting.
• Long-Context: Ideal for small, static documents that must be fully accessible across every token interaction without retrieval overhead.

Summary and Key Takeaways

• Relevance Filtering: Set minimum vector similarity thresholds (e.g., 0.55) to block noisy chunks and prevent model hallucination.
• Hybrid Search Supremacy: Combine vector similarity (semantic understanding) with BM25 keyword matching (exact product IDs) using a weighted parameter $\alpha$ (typically $0.6$ to $0.7$).
• Two-Stage Reranking: Apply computationally heavier cross-encoder rerankers only to compliance-sensitive queries where precision outweighs latency costs.
• Facts vs. Behavior: Use RAG for dynamic factual retrieval and static few-shot examples for behavioral reasoning templates.
• Token Budgeting: Keep few-shot examples concise (< 1,000 tokens total) to control exponential token costs at scale.
• Architectural Selection: Match your knowledge strategy to your data constraints—use RAG for dynamic large knowledge bases, Fine-Tuning for style and behavior, and Long-Context for small, static reference documents.