Parallel Tool Execution, Async Workflows, and MCP

Building reliable, production-grade AI agents requires moving beyond basic prompt engineering into robust software architecture. This chapter covers essential engineering patterns for agentic systems—focusing on tool definition design, managing token constraints at scale, enforcing structured outputs, handling tool execution failures gracefully, and implementing advanced self-correction loops.

1. Why Do We Need Production-Grade Tool Design?

In toy demos, an LLM receiving a raw text prompt and firing off arbitrary text functions might seem sufficient. However, production systems fail immediately when inputs and outputs are unconstrained.

Without explicit schema definitions, strict formatting hints, and robust runtime validation, Large Language Models will output ambiguous natural language instead of machine-parseable arguments.

Structure & Concept Flow

User Prompt -> LLM receives ambiguous prompt

LLM outputs freeform text

(e.g., "I'm sure customer is C91182")

Application Code fails to parse into Function Arguments

Silent Propagation / System Crash

Note: Production readiness means designing every tool from the LLM’s perspective—defining precise operational triggers, enforcing strict parameter types, and guaranteeing deterministic data interchange formats.

2. Core Principles of Tool Definition

To maximize tool selection accuracy and prevent execution errors, adhere to two foundational design principles:

Principle 1: Write Descriptions from the LLM's Perspective

Descriptions must instruct the model when to execute the tool, rather than merely stating what data the tool returns.

Poor Description: "Returns customer order data." (Focuses only on output)
Robust Description: "Use this tool when the customer asks about order status, delivery tracking, or shipment history." (Explicitly defines the behavioral trigger)

Principle 2: Strict Parameter Typing and Constraints

Every parameter must be explicitly defined using rigid types and validation bounds:

Primitive Types: Explicitly declare string, integer, boolean, or float.
String Formats: Enforce regex patterns (e.g., C-\d{4} for customer IDs).
Enumerated Values: Use enums for fixed value sets (e.g., severity fields constrained to low, medium, high, critical).
Required vs. Optional: Explicitly mark which parameters are mandatory and which can be omitted.

3. The Token Cost and Scaling Dilemma

A major practical constraint in agentic architecture is the context window. Every tool definition consumes tokens before a single user conversation even begins.

Structure & Concept Flow

Number of Tools Token Consumption Selection Accuracy & Behavior

-------------------------------------------------------------------------

3 to 5 tools 500 – 800 tokens High accuracy; optimal reasoning.

6 to 10 tools 1,000 – 2,000 tokens Good accuracy; distinct descriptions required.

10+ tools Degraded Similar tools begin to get confused.

20+ tools Severe degradation Requires Dynamic Tool Loading.

System Interview Insight

When designing an agent's tool set during a technical interview, always start with a core set of 4 to 6 tools. If your architecture demands more than 10 tools, the interviewer will immediately probe: "How do you handle tool selection at scale?" Demonstrating knowledge of dynamic tool loading proves you build for production, not just demos.

4. Dynamic Tool Loading Architecture

When an agent needs access to dozens or hundreds of specialized tools, loading all of them into every prompt destroys token efficiency and causes model confusion. The solution is Dynamic Tool Loading.

Structure & Concept Flow

User Intent: "Where is my package?"

Intent Classification Layer

+---> Matches intent: "Order & Shipping"

Dynamic Injector loads ONLY Order-Related Tools (4-6 tools)

LLM evaluates prompt against filtered, compact tool set

By classifying the user's intent first and injecting only the relevant subset of tools into the active context window, you keep active tool counts constant (around 4 to 6 per request) regardless of whether your system possesses 10 or 1,000 total tools.

5. Structured Output Enforcement Mechanisms

When an LLM invokes a tool, the output must be valid JSON matching a rigid schema. Freeform text cannot be reliably parsed into programmatic function arguments.

The industry relies on four primary enforcement techniques, ranked from highest to lowest reliability:

Enforcement Technique	Mechanism	Reliability	Pros & Cons
1. Native Function Calling	Provider API (OpenAI, Anthropic, Google) enforces schema on their end.	Highest	Guaranteed structured tool call objects; native platform support.
2. JSON Mode	Instructs model to respond only in JSON.	High	Guarantees valid JSON syntax, but fields or types may drift from schema.
3. Pedantic Enforcement	Python data models (e.g., Pydantic) validate raw LLM outputs in code.	High	Catches type mismatches, missing fields, and format errors at parse time.
4. Prompt-Based	Instructions inside system prompt ("Respond in JSON with fields X, Y").	Lowest	Highly prone to adding markdown wrappers, missing fields, or breaking syntax.

Concrete Python Model Validation Example

Using a validation library like Pydantic catches type mismatches immediately at parse time, preventing subtle bugs from propagating silently through your application.

PYTHON

from typing import Optional
from pydantic import BaseModel, Field, ValidationError

class OrderLookupParams(BaseModel):
    customer_id: str = Field(..., description="Required customer identifier, format C-XXXX")
    order_id: Optional[str] = Field(None, description="Optional specific order ID")

class RefundDecision(BaseModel):
    eligible: bool = Field(..., description="Whether the customer is eligible for a refund")
    reason: str = Field(..., description="Detailed justification for the decision")
    amount: float = Field(..., description="Calculated refund amount")
    requires_approval: bool = Field(..., description="Flag indicating if human manager approval is needed")

# Example validation check
try:
    # Imagine LLM returned {"eligible": "yes", ...} (string instead of boolean)
    raw_llm_output = '{"eligible": "yes", "reason": "Damaged goods", "amount": 49.99, "requires_approval": false}'
    validated_data = RefundDecision.model_validate_json(raw_llm_output)
except ValidationError as e:
    print(f"Validation failed: {e}")

Without validation, a type mismatch like passing "yes" (string) instead of true (boolean) propagates silently. Code treats "yes" as truthy in some contexts and falsy in others, creating nearly impossible-to-reproduce bugs.

6. Self-Correction via Parse-with-Retry Loops

When validation fails, do not crash the application. Because LLMs are inherently capable of self-correction when given precise error feedback, implement a parse-with-retry pattern.

PYTHON

import json
from pydantic import ValidationError

def parse_with_retry(llm_client, prompt, model_class, max_retries=2):
    current_prompt = prompt
    for attempt in range(max_retries + 1):
        response = llm_client.generate(current_prompt)
        try:
            return model_class.model_validate_json(response)
        except ValidationError as e:
            if attempt == max_retries:
                raise RuntimeError(f"Failed to parse LLM output after {max_retries} retries: {e}")
            
            # Append exact error feedback back into conversation for self-correction
            current_prompt += (
                f"\n\n[System Error: Your previous output failed validation: {e}. "
                f"Please fix the schema errors and output valid JSON matching the required schema.]"
            )

Two retries are typically more than enough for the model to correct formatting or type errors. If it exhausts retries, escalate to fallback logic rather than guessing.

7. Production Failure Modes: The Danger of Hallucinated Success

Tools call external APIs, databases, and microservices. Network timeouts, rate limits, and 503 service outages will happen in production.

The Silent Failure Trap

The most dangerous failure mode is not a visible crash—it is an agent silently treating a failed tool call as a success and generating a response based on data it never received.

Structure & Concept Flow

External API Times Out

Tool returns Empty Response / None

LLM instinct: "Be helpful!" -> Fills in blanks from training weights

Hallucination: Tells customer "Your order has shipped!" (Zero data backing it)

8. Five Production Error-Handling Patterns

To prevent hallucinations and build resilient agents, implement these five error-handling patterns:

Retry with Exponential Backoff: Handle transient failures (timeouts, 503 errors) by waiting incrementally before retrying.
- Attempt 1: Wait 1 second -> Retry
- Attempt 2: Wait 2 seconds -> Retry
- Attempt 3: Wait 4 seconds -> Stop (Exponential increase prevents overwhelming an already struggling downstream service).
Fallback to Alternative Source: Route requests to cached data or secondary read replicas if primary APIs go down.
Graceful Degradation: Handle non-critical tool failures gracefully. If a auxiliary knowledge base is down but a customer asks about order status, fulfill the order status check and inform the user that recommendations are temporarily unavailable.
Escalation to Human: When automated retries and fallbacks are exhausted, stop looping and connect the customer directly to a human support agent.
Inform and Wait: When an action cannot proceed without missing data, explicitly state what is happening to the user and offer alternative assistance.

9. Designing Robust Tool Error Returns

When writing your tool execution wrapper, never return an empty string or ambiguous None on failure.

Structure & Concept Flow

Bad Approach:

Tool crashes/times out -> Returns "" (Empty String)

LLM interprets "" as "Tool executed successfully with no results" -> Hallucinates response.

Good Approach:

Tool crashes/times out -> Returns Explicit Error Object:

{"status": "error", "code": "TIMEOUT", "message": "Order service unavailable after 3 attempts."}

LLM reads explicit error -> Understands data is missing -> Accurately reports system status to user.

10. Summary and Key Takeaways

Tool Description Design: Write tool descriptions from the LLM's perspective, instructing the model when to invoke the tool rather than merely summarizing output data.
Parameter Constraints: Use strict typing (string, integer, boolean), explicit regex formats, and enums for parameters. Always designate required versus optional fields.
Token Economics & Scaling: Keep active tool sets around 4 to 6 tools per request to maximize accuracy and conserve token context windows.
Dynamic Tool Loading: For large systems with 20+ tools, use intent classification layers to dynamically inject only relevant tools per request.
Structured Output Enforcement: Rely on native provider function calling, JSON mode, and Pydantic schema validation over fragile prompt-based formatting.
Parse-with-Retry Loops: Catch validation errors programmatically, feed exact error messages back into the conversation context, and allow 1-2 retries for LLM self-correction.
Mitigating Hallucinations: Never let an agent treat failed tool executions as successful data returns. Always return explicit structured error objects so the model understands when data is missing.
Production Resilience Patterns: Implement exponential backoff retries, data fallbacks, graceful degradation, human escalation, and informative waiting states.