Evaluator - Optimizer Design Pattern in Agentic Workflows

The Evaluator-Optimizer design pattern is an agentic workflow architecture that decouples content generation from quality control. Instead of returning an initial LLM output directly to the user, the architecture routes the output to an evaluation node—either an automated LLM with structured output or a human-in-the-loop (HITL)—which measures response quality against predefined benchmarks. If the output fails these checks, explicit feedback is routed back to the generator to refine the output iteratively until it meets acceptable quality standards.

1. Why Do We Need the Evaluator-Optimizer Pattern?

Single-pass Large Language Model (LLM) calls often fail when tasks require nuanced criteria, stylistic standards, or exact factual constraints. A naive generation call operates without a self-correction mechanism:

In complex production workflows, returning raw responses can lead to quality degradation, formatting errors, or off-target answers. The Evaluator-Optimizer pattern addresses this by introducing a closed-loop feedback mechanism:

1. Separation of Concerns: The generator focuses purely on creation based on prompt parameters, while the evaluator focuses exclusively on critical assessment against specific criteria.
2. Iterative Optimization: Rather than starting from scratch upon failure, the generator receives actionable, stateful feedback to guide subsequent output generations.
3. Flexible Evaluation Modes: Evaluation can be seamlessly switched between automated LLM execution and human inspection depending on the risk tier or task complexity.

2. Core Architectural Workflow

The workflow cycles between two primary nodes—the Generator Node and the Evaluator Node—guided by conditional routing edges.

Automated LLM Evaluator Flow

Human-in-the-Loop (HITL) Evaluator Flow

3. Defining State and Structured Data Contracts

To maintain continuity across feedback loops in LangGraph, we must define a shared workflow state. This state tracks the core content, input topic, targeted feedback, and evaluation verdicts across loop iterations.

Workflow State Schema

from typing import TypedDict, Optional
from pydantic import BaseModel, Field
from typing_extensions import Literal

# Shared Graph State across all nodes
class WorkflowState(TypedDict):
    topic: str
    joke: str
    feedback: Optional[str]
    funny_or_not: Optional[str]

# Structured Output Schema for the LLM Evaluator Node
class FeedbackState(BaseModel):
    grade: Literal["funny", "not funny"] = Field(
        description="Determine whether the generated joke is funny or not funny."
    )
    feedback: str = Field(
        description="If the joke is not funny, provide constructive feedback on how it can be improved."
    )

Note: Enforcing structured output via FeedbackState ensures that the evaluator LLM returns parseable, typed outputs (grade and feedback) rather than unstructured freeform text.

4. Implementing the LLM Generator Node

The generator node dynamically adjusts its generation prompt based on whether feedback exists in the shared state. On the first iteration, it generates content based purely on the topic. On subsequent loops, it incorporates the evaluator's feedback to guide refinement.

from langchain_core.messages import SystemMessage, HumanMessage
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.7)

def llm_generator(state: WorkflowState) -> dict:
    topic = state.get("topic")
    feedback = state.get("feedback")
    
    if feedback:
        prompt = (
            f"Write a joke about the topic: '{topic}'.\n"
            f"CRITICAL FEEDBACK FROM PREVIOUS ATTEMPT: {feedback}\n"
            f"Please revise the joke to address the feedback."
        )
    else:
        prompt = f"Write a joke about the topic: '{topic}'."
        
    response = llm.invoke([HumanMessage(content=prompt)])
    
    # Return updated state dictionary key
    return {"joke": response.content}

5. Variant 1: Automated LLM Evaluator Node

In the automated variant, an LLM acts as the judge. By binding the output to our FeedbackState schema using .with_structured_output(), the model is forced to return a grade and actionable feedback.

# Bind structured schema to evaluator model
evaluator_llm = llm.with_structured_output(FeedbackState)

def llm_evaluator(state: WorkflowState) -> dict:
    joke = state.get("joke")
    
    prompt = (
        f"You are an expert comedy critic. Evaluate the following joke:\n\n"
        f"'{joke}'\n\n"
        f"Determine if it is funny or not funny. If it is not funny, provide clear feedback."
    )
    
    result: FeedbackState = evaluator_llm.invoke([HumanMessage(content=prompt)])
    
    return {
        "funny_or_not": result.grade,
        "feedback": result.feedback
    }

6. Variant 2: Human-in-the-Loop (HITL) Evaluator Node

For high-stakes environments, human judgment can replace the automated evaluator. Using LangGraph's interrupt() function, graph execution pauses at the evaluator node, allowing an external human operator to inspect the current state and issue a verdict.

from langgraph.types import interrupt

def human_evaluator(state: WorkflowState) -> dict:
    joke = state.get("joke")
    
    # Execution halts here and waits for external user feedback via thread resume
    user_verdict = interrupt({
        "question": f"Is this joke funny?\nJoke: '{joke}'",
        "instructions": "Respond with 'funny' or 'not funny'."
    })
    
    return {
        "funny_or_not": user_verdict.get("grade"),
        "feedback": user_verdict.get("feedback", "")
    }

7. Dynamic Graph Construction & Conditional Edges

Routing control relies on a conditional edge function. If the state shows funny_or_not == "funny", execution proceeds to completion. Otherwise, the flow routes back to the generator.

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver

def decide_next_node(state: WorkflowState) -> str:
    if state.get("funny_or_not") == "funny":
        return "accepted"
    else:
        return "rejected"

def build_workflow(human_in_the_loop: bool = False):
    builder = StateGraph(WorkflowState)
    
    # Add shared Generator Node
    builder.add_node("generator", llm_generator)
    
    # Dynamically inject Evaluator Node based on flag
    if human_in_the_loop:
        builder.add_node("evaluator", human_evaluator)
    else:
        builder.add_node("evaluator", llm_evaluator)
        
    # Set entry point and primary edge
    builder.add_edge(START, "generator")
    builder.add_edge("generator", "evaluator")
    
    # Conditional routing based on evaluation outcome
    builder.add_conditional_edges(
        "evaluator",
        decide_next_node,
        {
            "accepted": END,
            "rejected": "generator"
        }
    )
    
    # Compile graph with checkpointing memory (required for HITL state persistence)
    memory = MemorySaver()
    return builder.compile(checkpointer=memory)

8. Workflow Visualization

When dynamically compiling the graph based on the human_in_the_loop flag, unused nodes remain disconnected while active nodes form the core execution loop.

Active LLM Evaluator Configuration (human_in_the_loop = False)

Active HITL Evaluator Configuration (human_in_the_loop = True)

9. Execution Walkthroughs

Walkthrough 1: Automated Evaluation Run

# Compile workflow with automated evaluator
app = build_workflow(human_in_the_loop=False)

config = {"configurable": {"thread_id": "auto_run_1"}}
initial_input = {"topic": "nuts"}

# Stream graph execution
for event in app.stream(initial_input, config=config):
    print(event)

Trace Output:

1. Generator Execution: Produces joke: "Why brought the nerd a ladder to the bar? Because he heard the drinks were on the house."
2. Evaluator Execution: Evaluates candidate joke against schema. Returns funny_or_not = "funny".
3. Conditional Router: Evaluates decide_next_node -> returns "accepted".
4. Completion: Graph hits END.

Walkthrough 2: Human-in-the-Loop Refinement Trace

# Compile workflow with HITL enabled
app = build_workflow(human_in_the_loop=True)
config = {"configurable": {"thread_id": "hitl_run_1"}}

# Step 1: Initial invocation triggers execution up to the human interrupt
initial_input = {"topic": "Apple"}
app.invoke(initial_input, config=config)

# Graph halts at 'evaluator' node and exposes state interrupt:
# Generated Joke: "Why did the iPhone go to therapy? Because it couldn't stop comparing itself to all other models."

# Step 2: Human inspects output and submits negative verdict ("not funny")
from langgraph.types import Command

app.invoke(
    Command(resume={"grade": "not funny", "feedback": "Too cliché. Avoid iPhone therapy tropes."}),
    config=config
)

# Step 3: Graph resumes, loops back to generator, and produces revision:
# Revised Joke: "Why did Apple go to therapy? Because it kept comparing itself to other fruits."

# Step 4: Human accepts revised output
app.invoke(
    Command(resume={"grade": "funny", "feedback": ""}),
    config=config
)
# Graph hits END.

10. Comparative Analysis: LLM vs. Human Evaluators

11. Broader Context in Agentic Architecture Literature

The Evaluator-Optimizer pattern is a foundational building block in production AI systems, as highlighted in key agentic architecture literature (e.g., Anthropic's research on agentic design patterns). It works alongside three other core design paradigms:

• Prompt Chaining: Standard linear execution where output $A$ becomes input $B$.
• Routing: Classifies an input and directs it to a specialized downstream worker.
• Orchestrator-Worker: A central manager agent breaks down a task, delegates sub-tasks to worker agents, and synthesizes their outputs.
• Evaluator-Optimizer: A two-node feedback loop focused specifically on iterative quality improvement.

12. Summary and Key Takeaways

• Decoupled Roles: Separating content generation from output evaluation prevents model bias and improves compliance with quality benchmarks.
• Closed Feedback Loop: Evaluator node failure generates structured feedback, which is injected into the state to guide the generator's next attempt.
• Structured Output Data Contracts: Using Pydantic schemas (e.g., FeedbackState) guarantees that LLM evaluators return structured grades and actionable feedback.
• Flexible Evaluator Implementations: The evaluator node can easily switch between an automated LLM call and a human reviewer (human_in_the_loop).
• State Persistence via Checkpointers: Implementing Human-in-the-Loop approval requires state persistence engines (e.g., MemorySaver) to pause and resume graph threads using a thread_id.
• Dynamic Graph Assembly: Workflows can dynamically attach either an LLM or Human evaluator node at compile time based on system configuration flags.
• Architectural Synergy: The Evaluator-Optimizer design pattern complements Prompt Chaining, Routing, and Orchestrator-Worker patterns in agentic workflows.