Dynamic Routing, Intent Classifiers, and Multi-Step Handlers

This chapter explores the architectural patterns used to build robust, production-grade AI agents. It contrasts dynamic runtime execution patterns like ReAct (Reasoning and Acting) with static build-time patterns like Prompt Chaining, detailing how Chain of Thought (CoT) serves as an underlying reasoning engine to ensure auditability, safety, and accuracy.

1. The ReAct (Reason-Act) Loop & Dynamic Execution

In complex customer support and operations scenarios, tasks are rarely linear. A simple, fixed workflow cannot handle situations where the next step depends entirely on what the previous step uncovered.

For example, troubleshooting a damaged item is a highly dynamic process that requires an agent to adapt its path based on real-time discoveries:

Structure & Concept Flow

Customer reports damaged item

Agent looks up order details

Agent checks warranty status

Agent retrieves return policy

Agent verifies customer address

Agent processes return/refund

Because each action is contingent on the output of the prior action, the agent must dynamically navigate its tools. The ReAct (Reason-Act) pattern solves this by structuring the agent's execution into iterative cycles of reasoning, action execution, and observation.

2. Loop Budgets and Graceful Degradation

A major risk of giving an LLM dynamic control over tool execution is the potential for infinite loops. If an agent gets confused, encounters unexpected tool outputs, or fails to parse a response, it can enter a cycle of repeatedly calling tools and reasoning in circles. This not only degrades the user experience but also rapidly burns through API tokens.

To prevent this, production systems implement two critical safety patterns:

Loop Budget (Max Steps): A hard ceiling on the number of reasoning-action cycles the agent can perform. For instance, setting a max_steps parameter to 10 ensures that the agent cannot execute more than 10 iterations before terminating.
Graceful Degradation: Instead of crashing or throwing an unhandled exception when the loop budget is exhausted, the agent executes a clean fallback path. It returns a polite, structured message acknowledging its limitation and hands off the context to a human agent or a fresh system.

Real-Life Analogy

Think of a loop budget like a customer service representative who has a policy: "If you cannot resolve a customer's issue within 10 minutes, do not keep them on hold indefinitely. Transfer them immediately to a senior specialist with all the notes you have collected so far."

Implementation of a ReAct Loop with Budget and Fallback

The following Python code demonstrates how to implement a ReAct loop with a hard step budget and a graceful degradation fallback:

PYTHON

import time
from typing import Dict, Any, List

class ReActAgent:
    def __init__(self, tools: Dict[str, Any], max_steps: int = 10):
        self.tools = tools
        self.max_steps = max_steps
        self.fallback_message = (
            "I was unable to fully resolve this issue within the allowed steps. "
            "Let me connect you with a fresh specialist who can assist you further."
        )

    def run(self, user_query: str) -> str:
        step_count = 0
        context: List[Dict[str, str]] = [{"role": "user", "content": user_query}]
        
        print(f"[System] Starting ReAct loop with a budget of {self.max_steps} steps.")
        
        while step_count < self.max_steps:
            step_count += 1
            print(f"\n--- Reasoning Step {step_count}/{self.max_steps} ---")
            
            # Simulate LLM deciding whether to call a tool or return a final answer
            llm_decision = self._call_llm_decision_engine(context)
            
            if llm_decision["type"] == "final_answer":
                print("[Agent] Final Answer reached.")
                return llm_decision["content"]
            
            elif llm_decision["type"] == "tool_call":
                tool_name = llm_decision["tool_name"]
                tool_args = llm_decision["tool_args"]
                
                print(f"[Agent] Decided to call tool: '{tool_name}' with args: {tool_args}")
                
                # Execute tool and get observation
                try:
                    observation = self.tools[tool_name](**tool_args)
                    print(f"[Observation] Tool output: {observation}")
                except Exception as e:
                    observation = f"Error executing tool: {str(e)}"
                    print(f"[Observation] Tool execution failed: {observation}")
                
                # Append reasoning and observation to context
                context.append({"role": "assistant", "content": f"Thought: {llm_decision['thought']}\nAction: {tool_name}({tool_args})"})
                context.append({"role": "system", "content": f"Observation: {observation}"})
                
        # If loop budget is exhausted, degrade gracefully
        print(f"\n[System] Loop budget of {self.max_steps} exceeded! Initiating graceful degradation.")
        return self.fallback_message

    def _call_llm_decision_engine(self, context: List[Dict[str, str]]) -> Dict[str, Any]:
        # Mocking LLM behavior for demonstration
        # In a real system, this calls OpenAI/Anthropic tool-calling APIs
        last_msg = context[-1]["content"]
        if "Observation" in last_msg and "Order #12345 found" in last_msg:
            return {
                "type": "final_answer",
                "content": "Your order #12345 has been located and is currently in transit."
            }
        else:
            return {
                "type": "tool_call",
                "thought": "I need to look up the customer's order details to proceed.",
                "tool_name": "lookup_order",
                "tool_args": {"order_id": "12345"}
            }

# Mock tool dictionary
mock_tools = {
    "lookup_order": lambda order_id: f"Order #{order_id} found. Status: Shipped."
}

# Run the agent
agent = ReActAgent(tools=mock_tools, max_steps=3)
final_response = agent.run("Where is my order?")
print(f"\n[Final Response]: {final_response}")

3. Chain of Thought (CoT) Prompting as a Reasoning Technique

It is vital to distinguish between Chain of Thought (CoT) and the ReAct pattern. Chain of Thought is not an execution architecture; it is a reasoning technique that operates inside an execution architecture.

Instead of allowing an LLM to jump directly from a user query to a final answer, CoT instructs the model to explicitly write out its step-by-step reasoning before outputting the final response.

Why CoT Matters: The Laptop Refund Example

Consider a customer asking for a refund on a laptop purchased 60 days ago. The company's return policy states that electronics must be returned within 30 days.

Naive Approach (Without Chain of Thought)

CODE

User: "I bought a laptop 2 months ago and want a refund."
Agent: "Refund approved! Please send the laptop back." (WRONG)

Problem: The agent skipped policy checks, ignored the purchase date, and approved an ineligible refund. There is no reasoning trail to explain why or how this decision was made.

Decoupled Production Approach (With Chain of Thought)

CODE

User: "I bought a laptop 2 months ago and want a refund."
Agent (Internal Thought Process):
  1. The order was placed approximately 60 days ago.
  2. The return policy allows returns within 30 days for electronics.
  3. 60 days exceeds the 30-day window.
  4. The electronics category has no extended return window.
  5. Conclusion: The refund is not eligible.
Agent (Public Response): "I'm sorry, but because the laptop was purchased 60 days ago, it falls outside our 30-day return window for electronics. We cannot process a refund." (CORRECT)

Production Value: Auditability

In production systems, AI decisions have real financial and customer relationship consequences:

False Positives: Approving a refund that should have been denied costs the company direct revenue.
False Negatives: Denying a refund that should have been approved damages customer trust and retention.

By forcing the agent to generate a Chain of Thought, you create a clear audit trail. If an agent makes a mistake, engineers can trace exactly where the reasoning went wrong:

Was it a failure in the policy lookup?
Was it an error in the date/data calculation?
Was it a category classification error?

4. Comparing Chain of Thought (CoT) vs. ReAct

While both patterns involve step-by-step processing, they operate at different levels of the agentic stack. The table below synthesizes their key differences:

Dimension	Chain of Thought (CoT)	ReAct Pattern
Core Mechanism	Reasons step-by-step, then answers.	Reasons, acts (calls tools), observes, and repeats.
Execution Scope	Occurs within a single LLM call.	Occurs across multiple LLM calls and tool executions.
External Interaction	Cannot call external tools or APIs; relies entirely on information present in the prompt context.	Actively invokes external APIs, databases, and tools between reasoning steps.
Relationship	Serves as the internal reasoning engine.	Serves as the outer execution loop.

How They Work Together

CoT and ReAct are complementary. A robust ReAct agent uses Chain of Thought within each reasoning step. When the agent pauses to think between tool calls, that thinking is CoT reasoning.

Structure & Concept Flow

ReAct Loop Step

+--> [LLM Call]

+--> Chain of Thought (CoT) Reasoning (Internal Thought)

+--> Decides Action (Tool Call)

Execute Tool -> Observe Output -> Next ReAct Loop Step

5. Prompt Chaining: Decomposing Complex Objectives

Large Language Models perform significantly worse as the number of simultaneous objectives increases. Asking a model to perform multiple distinct tasks in a single prompt—such as classifying an issue, drafting a response, and reviewing its own tone—forces the model to juggle conflicting success criteria, leading to degraded output quality.

Prompt Chaining solves this by breaking a complex task into a linear sequence of highly focused LLM calls.

Structure & Concept Flow

User Complaint

[Step 1: Classification LLM] ---> Outputs: Low, Medium, High, or Critical

[Step 2: Drafting LLM] ---> Uses classification to draft a tailored response

[Step 3: Tone Review LLM] ---> Reviews and refines the draft against guidelines

Final Approved Response

Each step in the chain has:

A single, dedicated job.
A specific system prompt optimized exclusively for that job.
A clear, easily testable success definition.

This modularity makes debugging straightforward. If the tone of the final response is poor, you can isolate and debug Step 3 without modifying or risking regressions in the classification (Step 1) or drafting (Step 2) logic.

6. Implementing a Prompt Chain in Python

The following code demonstrates a production-style implementation of a three-step prompt chain for handling customer complaints:

PYTHON

import os
from typing import Dict, Any

# Mock LLM client wrapper for demonstration
class MockLLMClient:
    def call(self, system_prompt: str, user_input: str) -> str:
        # Simple routing logic to simulate different LLM responses
        if "classify" in system_prompt.lower():
            if "broken" in user_input.lower() or "fire" in user_input.lower():
                return "CRITICAL"
            return "MEDIUM"
        elif "draft" in system_prompt.lower():
            severity = user_input.split("|")[0].strip()
            complaint = user_input.split("|")[1].strip()
            if severity == "CRITICAL":
                return f"We are deeply sorry about your experience. We have escalated your issue regarding: '{complaint}'."
            return f"Thank you for reaching out. We are looking into your issue: '{complaint}'."
        elif "review" in system_prompt.lower():
            # Ensure professional tone
            draft = user_input
            return f"[APPROVED TONE] {draft}"
        return "Default response"

def complaint_pipeline(complaint_text: str) -> Dict[str, str]:
    llm = MockLLMClient()
    
    # Step 1: Classify Severity
    system_prompt_1 = (
        "You are an intent classifier. Analyze the customer complaint and output "
        "exactly one word representing the severity: LOW, MEDIUM, HIGH, or CRITICAL."
    )
    severity = llm.call(system_prompt_1, complaint_text).strip()
    
    # Step 2: Draft Response based on Severity
    system_prompt_2 = (
        "You are a customer support writer. Draft a response to the customer complaint "
        "that matches the provided severity level. Keep it concise."
    )
    draft_input = f"{severity} | {complaint_text}"
    draft_response = llm.call(system_prompt_2, draft_input)
    
    # Step 3: Review Tone and Guidelines
    system_prompt_3 = (
        "You are a quality assurance editor. Review the drafted response. "
        "Ensure it is empathetic, professional, and contains no internal jargon. "
        "Output the finalized, polished response."
    )
    final_response = llm.call(system_prompt_3, draft_response)
    
    return {
        "severity": severity,
        "draft": draft_response,
        "final_response": final_response
    }

# Execution
complaint = "My laptop caught fire while charging!"
pipeline_result = complaint_pipeline(complaint)

print(f"Original Complaint: {complaint}")
print(f"Step 1 (Severity): {pipeline_result['severity']}")
print(f"Step 2 (Draft):    {pipeline_result['draft']}")
print(f"Step 3 (Final):    {pipeline_result['final_response']}")

7. Trade-offs of Prompt Chaining: The Latency Budget

While prompt chaining dramatically improves output quality and system maintainability, it introduces a significant trade-off: latency.

Each step in a prompt chain requires a separate network round-trip to the LLM provider. If each LLM call takes approximately 1 to 2 seconds, the total latency of the pipeline scales linearly with the number of steps:

T_{\text{total}} = \sum_{i=1}^{N} t_i

For a 3-step chain:

T_{\text{total}} \approx 1\text{s} + 1\text{s} + 1\text{s} = 3\text{s}

In customer-facing applications (such as live chat widgets) where users expect a response within a few seconds, a 3-second latency budget is highly noticeable. Adding a fourth or fifth step to the chain must be carefully evaluated: does the marginal increase in quality justify the additional latency penalty?

8. Prompt Chaining vs. ReAct: Build-Time vs. Runtime Decisions

The fundamental difference between Prompt Chaining and the ReAct pattern lies in who controls the execution path and when that path is decided.

Feature	Prompt Chaining	ReAct Pattern
Path Determination	Build-Time (Static)	Runtime (Dynamic)
Controller	The Developer	The LLM
Execution Flow	Step 1 $\rightarrow$ Step 2 $\rightarrow$ Step 3. The path never changes based on intermediate discoveries.	The LLM dynamically decides which tool to call next based on the results of the previous step.
Predictability	High. Easy to test, trace, and benchmark.	Lower. Requires strict guardrails (loop budgets) to prevent non-deterministic behavior.
Best Used For	Fixed, known sequences of steps where latency and reliability are critical.	Exploratory tasks where the exact sequence of steps cannot be known in advance.

Architectural Decision Rule

Use Prompt Chaining if your business process always follows the same sequence of steps regardless of the input content.
Use ReAct if the path to resolution depends entirely on what you discover along the way.

Summary and Key Takeaways

ReAct (Reason-Act) is a dynamic runtime pattern where the LLM decides the sequence of actions based on intermediate tool observations.
Loop Budgets (e.g., max_steps = 10) are mandatory in production ReAct implementations to prevent infinite loops and runaway token costs.
Graceful Degradation ensures that when an agent exhausts its loop budget, it transitions cleanly to a fallback message or human handoff instead of crashing.
Chain of Thought (CoT) is an internal reasoning technique, not an execution architecture. It runs inside LLM calls to improve accuracy and provide an auditable decision trail.
Prompt Chaining decomposes complex, multi-objective tasks into a static sequence of single-purpose LLM calls, improving reliability and debugging.
The Latency Trade-off is the primary drawback of prompt chaining; because latency accumulates linearly ( $1\text{s}$ to $2\text{s}$ per step), developers must balance chain depth against user experience requirements.
Build-Time vs. Runtime: Choose Prompt Chaining for static, predictable workflows; choose ReAct for dynamic, exploratory workflows.