Structured Tool Schemas, Pydantic, and Parsing Defense

An LLM on its own is a closed system, limited to the static knowledge captured during its training. To interact with the real world—such as checking a live order status, processing a payment, or querying a database—the model must transition from a passive text generator into an active decision engine. This transition is enabled by Tool Calling, which serves as the concrete execution mechanism (the Act step) within the agentic Perceive-Reason-Act-Observe loop.

1. Architectural Context & Key Agentic Metrics

When designing agentic systems, selecting the right architectural pattern directly impacts predictability, latency, and cost. In production environments, system architects lean heavily toward deterministic control.

The 80/20 Rule of Agent Deployments

• Workflow Patterns (Chains & Routers): Account for roughly 80% of deployed production AI systems. These patterns prioritize predictability, structured transitions, and low latency.
• Autonomous Patterns (ReAct Loops): Account for roughly 20% of deployed systems. These are reserved for open-ended tasks where the execution path cannot be pre-determined.

Key Performance Indicators (KPIs) for Agent Design

When preparing for system design interviews, memorizing these production metrics is critical for justifying architectural trade-offs:

2. The Core Mental Model of Tool Calling

A common failure point in technical interviews is mischaracterizing how LLMs interact with external systems.

⚠️ The Fundamental Distinction

The LLM does not execute code. It does not make HTTP requests, run SQL queries, or write to file systems.

The LLM is strictly a decision maker, not an execution layer. It outputs a structured JSON object that expresses the intent to call a specific function with specific arguments. Your application code (the execution layer) receives this JSON, validates it, runs the actual code, and returns the result to the LLM.

Real-Life Analogy: Calling a Meeting

Think of tool calling like a corporate executive calling a meeting. The executive does not physically book the room, set up the projector, or compile the financial spreadsheets. Instead, the executive issues a structured request: "Schedule a Q4 budget review with the finance team on Tuesday at 2 PM."

An administrative assistant (the application code) processes this request, validates the availability of the team, books the room (executes the action), and reports back to the executive with the results.

3. The Step-by-Step Tool Execution Loop

The integration of tool calling into the agentic loop follows a strict, synchronous cycle of data exchange.

Detailed Step Walkthrough

1. Schema Registration: The application defines the available tools (names, descriptions, and parameter schemas) and sends them to the LLM alongside the user's prompt.
2. Intent & Argument Generation: The LLM reads the schemas, determines which tool is appropriate, and outputs a structured JSON payload containing the target function name and its arguments.
3. Execution: The application intercept layer parses the JSON, validates the arguments against the schema, and executes the actual function (e.g., database query, API call).
4. Observation Feedback: The application captures the return value of the function and sends it back to the LLM as a new contextual message (the "observation").
5. Synthesis: The LLM processes the observation and either requests another tool (continuing the loop) or generates a natural language response for the user.

4. Naive vs. Production-Grade Tool Schemas

The LLM relies entirely on the semantic information provided in the tool schema to determine when and how to use a tool. Poorly designed schemas lead to tool selection failures, hallucinated arguments, and system crashes.

Schema Comparison

5. Implementation with Pydantic & OpenAI Function Schemas

Using raw JSON schemas to define tools is error-prone and difficult to maintain. In production, developers use Pydantic to define robust, self-validating data models that can be automatically converted into LLM-compatible tool schemas.

Production Code: Defining and Parsing Tools

The following Python implementation demonstrates how to define a tool schema using Pydantic (v2), generate the OpenAI-compatible function definition, and safely parse the model's response.

from typing import Type, Dict, Any
from pydantic import BaseModel, Field, ValidationError
import json

# ==========================================
# 1. Define the Tool Schema using Pydantic
# ==========================================
class OrderLookupArgs(BaseModel):
    """
    Input arguments for the order lookup tool.
    """
    customer_id: str = Field(
        ..., 
        description="The unique customer identifier formatted as 'C-' followed by 5 digits (e.g., C-12345).",
        pattern=r"^C-\d{5}$"
    )
    include_history: bool = Field(
        default=False,
        description="Whether to include the historical orders from the past 12 months."
    )

# ==========================================
# 2. Convert to OpenAI Tool Specification
# ==========================================
def get_openai_tool_spec() -> Dict[str, Any]:
    return {
        "type": "function",
        "function": {
            "name": "order_lookup",
            "description": (
                "Retrieve active order details, delivery status, tracking numbers, "
                "and return eligibility for a specific customer. Use this tool whenever "
                "a customer inquires about where their package is or their order history."
            ),
            "parameters": OrderLookupArgs.model_json_schema()
        }
    }

# ==========================================
# 3. Mock Execution Layer (The Actual Code)
# ==========================================
def execute_order_lookup(customer_id: str, include_history: bool) -> Dict[str, Any]:
    # In a real application, this would query a database or external CRM API
    print(f"[SYSTEM] Executing database query for {customer_id} (History: {include_history})")
    return {
        "status": "success",
        "data": {
            "order_id": "98765",
            "status": "Shipped",
            "carrier": "FedEx",
            "tracking_number": "1Z999AA10123456784",
            "estimated_delivery": "2026-03-30"
        }
    }

6. Parsing Defense & Error Handling Strategies

When an LLM attempts to call a tool, the output JSON can fail in several ways:

1. Malformed JSON: The output is truncated or contains syntax errors.
2. Validation Failures: The JSON is valid, but the arguments violate schema constraints (e.g., a missing required field or an invalid string pattern).
3. Execution Failures: The arguments are valid, but the underlying system throws an error (e.g., database timeout or record not found).

Defensive Parsing Implementation

This code demonstrates how to handle these failures gracefully, providing feedback to the LLM so it can self-correct (the Reflection pattern).

def handle_llm_tool_call(raw_llm_output: str) -> str:
    """
    Parses, validates, and executes a tool call from raw LLM output.
    Returns a string observation to be sent back to the LLM.
    """
    try:
        # Step 1: Parse raw string to JSON
        tool_call = json.loads(raw_llm_output)
        function_name = tool_call.get("name")
        arguments_raw = tool_call.get("arguments")
        
        if function_name != "order_lookup":
            return f"Error: Unknown tool '{function_name}'. Available tools: 'order_lookup'."
            
        # Step 2: Validate arguments against Pydantic schema
        # (Handles cases where LLM sends invalid formats like "customer_id": "12345" instead of "C-12345")
        validated_args = OrderLookupArgs.model_validate(arguments_raw)
        
        # Step 3: Execute the actual function
        result = execute_order_lookup(
            customer_id=validated_args.customer_id,
            include_history=validated_args.include_history
        )
        return json.dumps(result)

    except json.JSONDecodeError:
        # Defense against malformed JSON
        return "Error: The tool call payload was not valid JSON. Please retry using valid JSON syntax."
        
    except ValidationError as ve:
        # Defense against schema validation violations
        # We return the exact validation errors so the LLM can self-correct
        error_details = ve.errors()
        readable_errors = [f"{err['loc'][0]}: {err['msg']}" for err in error_details]
        return f"Error: Argument validation failed. Details: {'; '.join(readable_errors)}. Please correct the arguments and retry."
        
    except Exception as e:
        # Defense against execution-level failures
        return f"Error: The tool failed to execute due to an internal system error: {str(e)}."

# ==========================================
# Execution Examples
# ==========================================

# Example A: Valid LLM Output
valid_output = '{"name": "order_lookup", "arguments": {"customer_id": "C-99812", "include_history": true}}'
print("--- Example A (Valid) ---")
print("Observation:", handle_llm_tool_call(valid_output))

# Example B: Invalid Argument Format (Triggers Pydantic Pattern Validation)
invalid_output = '{"name": "order_lookup", "arguments": {"customer_id": "invalid-id-123"}}'
print("\n--- Example B (Invalid Format) ---")
print("Observation:", handle_llm_tool_call(invalid_output))

7. Summary and Key Takeaways

• The LLM is a Decision Engine, Not an Executor: The LLM only generates structured JSON text specifying which tool to call and with what arguments. The application code is responsible for validation and execution.
• Production Prefers Control: Roughly 80% of deployed agentic systems use predictable workflow patterns (chains and routers), while only 20% use autonomous ReAct loops.
• Keep ReAct Budgets Tight: Limit ReAct loops to 5–15 steps and cap reflection/self-correction attempts at 2 to avoid diminishing returns, high latencies, and runaway token costs.
• Descriptions are the Steering Wheel: The LLM relies heavily on tool and parameter descriptions to determine when to trigger a tool. Vague descriptions lead to incorrect tool selection.
• Implement Defensive Parsing: Always wrap tool execution in robust try-except blocks. Use Pydantic to validate incoming arguments, and return precise validation error messages back to the LLM so it can self-correct.