Hands-On Code: Multi-Tier Model Fallbacks & Async Tools

This chapter bridges foundational theory and production-grade engineering by building a complete customer support AI agent from scratch. Moving past simple framework abstractions, we examine how primitive agent loops operate underneath popular tools like LangChain, CrewAI, and Microsoft AutoGen, layering on state management, tool execution, and architectural constraints.

1. Interview Architecture & Key Production Metrics

In an elite system design or AI engineering interview, rushing straight into code or hand-waving failure modes are fatal errors. Interviewers evaluate candidates across a strict four-part cadence: requirements for 3 minutes, architecture diagrams for 5 minutes, deep dives for 15 minutes, and trade-offs for 5 minutes.

When handling uncertainty, candidates must reason through trade-offs rather than guessing blindly, and adapt smoothly when the interviewer introduces new constraints.

Common Interview Pitfalls to Avoid

• Starting with components before requirements: Jumping straight into database choices or LLM models without clarifying latency, scale, and budget.
• Presenting a perfect system: Claiming zero latency or 100% reliability. All architectures involve trade-offs.
• Skipping failure scenarios: Ignoring rate limits, token timeouts, tool crashes, and hallucinated arguments.

Key System Numbers and Production Parameters

Production agent design is governed by strict quantitative bounds. The reference customer support agent in our architecture adheres to the following baseline metrics:

2. The Core Mental Model: From Theory to Code

Every agent framework—whether LangChain, CrewAI, or AutoGen—wraps the exact same underlying execution primitive. Understanding this core primitive lets you debug any framework, evaluate whether a library adds genuine value or just unnecessary abstraction, and build custom architectures tailored to precise requirements.

3. The 25-Line Minimum Agent Loop (Observe -> Decide -> Act -> Update -> Stop)

To understand how agents work under the hood, we start with the simplest possible implementation: a 25-line Python function executing the core agent loop.

import openai
from typing import List, Dict, Any, Callable

def run_agent(
    user_message: str, 
    system_prompt: str, 
    tools: List[Callable], 
    max_iterations: int = 5
) -> str:
    # 1. Initialize message list (Conversation State)
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ]
    
    for iteration in range(max_iterations):
        # 2. DECIDE: Call LLM with messages and tool definitions
        response = openai.chat.completions.create(
            model="gpt-4",
            messages=messages,
            tools=tools if tools else None
        )
        
        response_message = response.choices[0].message
        messages.append(response_message) # UPDATE step
        
        # 3. STOP CONDITION: Check if LLM requested tool calls
        if not response_message.tool_calls:
            return response_message.content
            
        # 4. ACT: Execute requested tools
        for tool_call in response_message.tool_calls:
            tool_result = execute_tool(tool_call) # Pseudo-code execution
            
            # 5. OBSERVE: Feed tool output back to LLM as a new message
            messages.append({
                "tool_call_id": tool_call.id,
                "role": "tool",
                "name": tool_call.function.name,
                "content": str(tool_result)
            })
            
    return "Error: Maximum iteration budget exceeded without resolution."

Mapping Code to Loop Steps

• Decide: The OpenAI API call where the LLM evaluates the message history and decides whether to answer or call a tool.
• Act: Iterating over tool_calls and executing the corresponding backend function.
• Observe: Appending the tool output back into the message array with the tool role so the LLM can ingest the new information.
• Update: Appending assistant responses and tool outputs to the conversation history array.
• Stop Condition: Checking whether tool_calls is empty. If empty, the loop terminates and returns the final text.

4. Limitations of the Naive Loop

While the 25-line loop illustrates core mechanics, it breaks down in production environments due to three missing pillars:

1. No Data Access (Without Tools): If a user asks "What is the status of order 12345?" and no tools are defined, the agent cannot query external databases. It can only hallucinate or state that it lacks access.
2. No State Management: The loop lacks tracking for iteration counts, execution lifecycles, tracing IDs, or accumulated token costs.
3. No Memory Management: When conversation length exceeds the context window, the model loses historical context or hits token rate limits.

5. Conversation State vs. Execution State

A critical architectural distinction in production AI systems is separating Conversation State from Execution State. Blurring this distinction introduces severe bugs, such as wasting tokens by feeding internal application flags to the LLM, or losing vital context.

State Categorization Table

6. Implementing Production State Management

To track metrics beyond plain message lists, we encapsulate execution state into a dedicated Python dataclass. This separates application plumbing from conversational history.

import uuid
from dataclasses import dataclass, field
from typing import List, Dict, Any, Optional

@dataclass
class AgentState:
    # Conversation State
    messages: List[Dict[str, Any]] = field(default_factory=list)
    
    # Execution State (Managed by Application, Never Sent to LLM)
    iteration: int = 0
    status: str = "running"  # running, completed, failed, waiting_for_human
    trace_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    pending_action: Optional[Dict[str, Any]] = None
    token_count: int = 0
    cost_usd: float = 0.0
    max_iterations: int = 5

Why This Separation Matters

• Token Efficiency: Internal counters (iteration, cost_usd, trace_id) are never sent to the LLM context window, saving valuable token costs.
• Debuggability: Every log entry and API trace carries the trace_id, enabling end-to-end request tracing across distributed microservices.
• Human-in-the-Loop Interruption: When an action exceeds the auto-approval threshold (e.g., a refund greater than $50.00), the status flips to waiting_for_human and the tool call payload is safely stored in pending_action.

7. Summary and Key Takeaways

• Interview Rigor: Master the 3-minute requirements, 5-minute architecture, 15-minute deep dives, and 5-minute trade-off structure while avoiding common pitfalls like skipping failure modes.
• Core Unit Economics: Design systems around strict constraints: 3-second latency budgets, $0.02$ to $0.035$` per interaction limits, 60/40 model routing splits, and 30% cache hit rates.
• Underlying Primitives: Every major agent framework (LangChain, CrewAI, AutoGen) wraps the same core Observe -> Decide -> Act -> Update -> Stop execution loop.
• Separation of Concerns: Strictly segregate Conversation State (sent to the LLM) from Execution State (managed by application code for iteration tracking, cost metering, and tracing).
• Loop Budgets: Always enforce hard iteration limits (5 to 15 steps) to prevent runaway agent loops and infinite tool-calling recursion.
• Human-in-the-Loop Controls: Implement threshold-based approval gates (such as $50.00 auto-approval limits) to intercept high-risk tool executions before they impact production systems.
• Traceability: Attach a unique trace_id UUID to every agent interaction to ensure robust logging and observability across asynchronous tool calls.