Sampling Determinism, Hallucinations, and Output Control

This chapter explores the fundamental mechanisms of controlling Large Language Model (LLM) outputs, focusing on temperature, Top P, and structured output schemas. It establishes a core engineering thesis: every significant agentic architecture pattern exists to compensate for a structural limitation of the underlying LLM. Additionally, this chapter provides practical strategies for model selection, cost optimization, and dynamic model routing in production environments.

1. The Mechanics of Token Selection: Temperature and Determinism

At their core, LLMs are next-token predictors that output a probability distribution over a vocabulary of tokens. To control how the model selects the next token from this distribution, developers use the Temperature parameter.

Temperature is a hyperparameter that scales the logits (the raw, unnormalized scores output by the model's final layer) before applying the softmax function to calculate token probabilities. Mathematically, the scaled probability $P(x_i)$ of token $x_i$ is defined as:

Where $z_i$ represents the logit for token $i$, and $T$ represents the temperature.

Temperature Ranges and Production Behaviors

2. Top P (Nucleus Sampling) vs. Temperature

While Temperature scales the entire probability distribution, Top P (also known as Nucleus Sampling) limits the pool of candidate tokens based on their cumulative probability mass.

For example, setting $\text{Top P} = 0.1$ instructs the model to sort the vocabulary by probability in descending order and discard all tokens outside the top 10% of the cumulative probability mass. The model then samples only from this restricted pool.

Production Best Practice

In production agent architectures, set the temperature to $0.0$ or $0.2$ and leave Top P at its default value ($1.0$). Tuning both parameters simultaneously introduces unnecessary complexity and makes debugging token distribution issues significantly harder without providing any clear performance benefit.

3. Production Code Pattern: Deterministic vs. Creative Tasks

In an agentic system, different tasks require different levels of determinism. Classification and tool execution require absolute consistency ($\text{Temperature} = 0.0$), whereas drafting user-facing responses benefit from a slight natural variation ($\text{Temperature} = 0.2$).

The following Python code demonstrates how to implement these distinct configurations using a unified interface:

import os
from openai import OpenAI

# Initialize client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "mock-key"))

def classify_customer_intent(user_query: str) -> str:
    """
    Classifies customer intent. Must be 100% deterministic.
    Uses Temperature = 0.0.
    """
    system_prompt = (
        "You are a precise intent classification utility. "
        "Classify the input into exactly one of these categories: "
        "[REFUND, ORDER_STATUS, TECHNICAL_SUPPORT, GENERAL_INQUIRY]."
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_query}
        ],
        temperature=0.0  # Absolute determinism
    )
    return response.choices[0].message.content.strip()

def draft_customer_response(intent: str, customer_context: str) -> str:
    """
    Drafts a response to the customer. Allows for slight natural variation.
    Uses Temperature = 0.2.
    """
    system_prompt = (
        f"You are an empathetic customer support agent. Draft a response "
        f"addressing the customer's issue. Category: {intent}."
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": customer_context}
        ],
        temperature=0.2  # Slight natural variation, highly coherent
    )
    return response.choices[0].message.content.strip()

# Example Execution Flow
if __name__ == "__main__":
    query = "Where is my package? It was supposed to arrive yesterday."
    
    # 1. Deterministic Classification
    detected_intent = classify_customer_intent(query)
    print(f"Detected Intent (Temp 0.0): {detected_intent}")
    
    # 2. Natural Response Generation
    context = f"User Query: {query} | Order Status: Delayed in transit. Expected delivery: 2 days."
    draft_reply = draft_customer_response(detected_intent, context)
    print(f"Drafted Response (Temp 0.2):\n{draft_reply}")

4. LLM Limitations as Architectural Drivers

A foundational paradigm shift in agent engineering is understanding that every significant agentic architecture decision exists to compensate for a structural limitation of the underlying LLM.

When you view agent design through this lens, architectures stop being arbitrary design patterns to memorize and instead become logical engineering responses to specific physical constraints.

5. The Four Core LLM Limitations

To design robust agentic systems, you must design around four fundamental limitations of LLMs:

1. Hallucination

• The Problem: The model generates text that is syntactically fluent, authoritative, and highly confident, but factually incorrect.
• The Cause: Hallucination is not a "bug" that will be fully resolved in future model versions; it is structural to how next-token prediction works. The model is mathematically optimized to produce plausible text, not factually accurate text.
• Real-World Analogy: A highly charismatic job applicant who confidently invents answers to technical questions during an interview. They sound incredibly convincing, but their answers have no basis in reality.
• Production Risk: A support agent confidently tells a customer that a product has a "2-year manufacturer warranty" when the actual warranty is only 90 days, exposing the company to legal and financial liability.

2. Context Window Bounds

• The Problem: LLMs have a strict upper limit on the number of tokens they can process in a single execution step.
• The Cause: Attention mechanisms scale quadratically or near-quadratically with context length. In a long, multi-turn conversation (e.g., 50 turns), older messages are evicted from the active context window.
• Production Risk: Once early messages are evicted, the model loses all memory of those interactions. The customer is forced to repeat information they already provided, leading to a frustrating user experience.

3. Absence of Real-Time Knowledge

• The Problem: LLMs are frozen in time. They have no awareness of events, policy changes, inventory updates, or database modifications that occurred after their training data cutoff date.
• Production Risk: If a customer asks, "Is this item currently in stock?", an LLM without external tools must either refuse to answer or hallucinate a false inventory status.

4. Latency and Cost

• The Problem: LLM inference is slow and expensive. A single API call can take anywhere from 0.5 to 5 seconds and incurs a per-token cost.
• The Cause: Autoregressive decoding requires generating tokens sequentially, one by one, which is highly compute-intensive.
• Production Risk: If an agentic workflow chains eight sequential LLM calls to resolve a single user query, the total latency will exceed 10 seconds, and the cost will scale linearly with the cumulative token volume. This is unacceptable for real-time user interfaces.

6. Mapping Limitations to Architectural Solutions

The architecture of modern agentic systems is directly mapped to these four limitations. The table below serves as a blueprint for the engineering patterns required to build production-grade agents:

7. Model Selection: The Five Engineering Dimensions

Selecting the right model for an agent is not about choosing the highest-ranked model on a public leaderboard. It is a multi-dimensional engineering trade-off across five key vectors:

                  [Reasoning Quality]
                         / \
                        /   \
            [Latency]  /     \  [Context Window]
                      /       \
                     /_________\
             [Cost per Token]  [Availability/SLA]

1. Reasoning Quality: The model's ability to handle complex logic, follow multi-step instructions, and accurately parse complex schemas.
2. Context Window Size: The maximum token capacity for input and output.
3. Latency: Time-to-first-token (TTFT) and overall token generation speed.
4. Cost per Token: The pricing structure for input and output tokens (typically measured per million tokens).
5. Availability & SLA: Rate limits, uptime guarantees, regional hosting options, and compliance standards.

8. The Model Routing Pattern

In production environments, using a single, high-tier model (like GPT-4o or Claude 3.5 Sonnet) for every single task is highly inefficient. Instead, production systems use Model Routing: dynamically directing tasks to different models based on the complexity of the request.

Model Cost Comparison (Representative Pricing)

• Premium Model (e.g., GPT-4o / Claude 3.5 Sonnet): $\approx \char36 2.50 \text{ to } \char36 3.00$ per million input tokens.
• Utility Model (e.g., GPT-4o-mini): $\approx \char36 0.15$ per million input tokens.
• The Delta: The premium model is roughly 17 times more expensive than the utility model.

9. Production Code Pattern: Dynamic Model Routing

The following code demonstrates a production-grade implementation of the Model Routing pattern. It uses a fast, cost-effective model to classify the user's intent and then routes the query to either a lightweight utility model or a high-reasoning model depending on the complexity of the task.

import os
from typing import Dict, Any
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "mock-key"))

def classify_complexity(user_query: str) -> str:
    """
    Quickly classifies the query complexity.
    Runs on a cheap model with Temp 0.0 for maximum speed and consistency.
    """
    prompt = (
        "Analyze the user query. Classify it as 'SIMPLE' if it is a basic "
        "informational request, status check, or FAQ. Classify it as 'COMPLEX' "
        "if it involves disputes, complaints, multi-step troubleshooting, or "
        "negotiation. Respond with exactly one word: SIMPLE or COMPLEX."
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": user_query}
        ],
        temperature=0.0
    )
    return response.choices[0].message.content.strip().upper()

def handle_simple_task(user_query: str) -> str:
    """
    Handles simple tasks using a fast, cheap model.
    """
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Provide a quick, direct response."},
            {"role": "user", "content": user_query}
        ],
        temperature=0.0
    ).choices[0].message.content

def handle_complex_task(user_query: str) -> str:
    """
    Handles complex tasks using a high-reasoning model (e.g., Claude or GPT-4o).
    Uses a system prompt designed for nuanced, empathetic, and policy-precise responses.
    """
    # In a real system, you might route to Claude 3.5 Sonnet here.
    # For this example, we use the premium gpt-4o model.
    return client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system", 
                "content": "You are a senior customer resolutions specialist. Provide an empathetic, highly detailed, and policy-compliant resolution."
            },
            {"role": "user", "content": user_query}
        ],
        temperature=0.2
    ).choices[0].message.content

def route_and_execute(user_query: str) -> Dict[str, Any]:
    """
    Orchestrates the routing and execution flow.
    """
    complexity = classify_complexity(user_query)
    
    if complexity == "SIMPLE":
        selected_model = "gpt-4o-mini"
        output = handle_simple_task(user_query)
    else:
        selected_model = "gpt-4o"
        output = handle_complex_task(user_query)
        
    return {
        "complexity": complexity,
        "routed_model": selected_model,
        "response": output
    }

# Example Usage
if __name__ == "__main__":
    # Test Case 1: Simple Query
    result_1 = route_and_execute("What time does your support desk close?")
    print(f"Query 1 Route: {result_1['complexity']} -> {result_1['routed_model']}")
    
    # Test Case 2: Complex Query
    result_2 = route_and_execute("I want a full refund immediately. Your software crashed and ruined my client presentation!")
    print(f"Query 2 Route: {result_2['complexity']} -> {result_2['routed_model']}")

10. Financial and Operational Impact of Model Routing

To understand the financial impact of model routing, let's look at a production system handling 10,000 customer interactions per day.

Scenario A: Monolithic Architecture (GPT-4o Only)

• Volume: 10,000 interactions/day
• Average Cost per Interaction: $\char36 0.0125$ (based on input/output token ratios on GPT-4o)
• Daily Cost: $10,000 \times \char36 0.0125 = \char36 125$
• Monthly Cost (30 Days): $\char36 3,750$

Scenario B: Routed Architecture (60% Simple / 40% Complex)

• Simple Queries (60%): Routed to GPT-4o-mini.
  • Average Cost per Interaction: $\char36 0.00075$
  • Daily Cost: $6,000 \times \char36 0.00075 = \char36 4.50$
• Complex Queries (40%): Routed to GPT-4o.
  • Average Cost per Interaction: $\char36 0.0125$
  • Daily Cost: $4,000 \times \char36 0.0125 = \char36 50.00$
• Total Daily Cost: $\char36 4.50 + \char36 50.00 = \char36 54.50$
• Monthly Cost (30 Days): $\char36 1,635$

Financial Summary

By implementing a routed architecture, the company achieves a 56% reduction in monthly LLM costs while maintaining identical quality and reasoning capabilities for complex edge cases.

11. Interview Strategy: Demonstrating Production-Ready Thinking

When interviewing for Agentic AI or AI Engineering roles, your answers should demonstrate that you think in terms of practical engineering trade-offs rather than abstract concepts.

Interview Question 1: "How do you ensure consistent, predictable agent behavior in production?"

• How to answer: Avoid focusing solely on prompt engineering. Instead, structure your answer around a multi-layered determinism strategy:
  1. Temperature: Set temperature to $0.0$ for all classification, routing, and tool-calling steps to ensure deterministic token selection.
  2. Structured Outputs: Enforce JSON schemas using strict structured output configurations (e.g., OpenAI's Structured Outputs or Pydantic validation) to guarantee that the output matches your system's expected data structures.
  3. Deterministic Tool Schemas: Use strictly defined JSON schemas for tools to prevent the model from hallucinating arguments.

Interview Question 2: "What are the primary challenges of building with LLMs, and how do you address them?"

• How to answer: Walk the interviewer through the four core limitations and map each one directly to its architectural solution:
  • "I view agent architecture as a set of logical solutions to the physical limitations of LLMs. For example, we address hallucinations using RAG and output validation guards. We handle context window limits through sliding-window memory systems. We solve the lack of real-time knowledge using function calling, and we manage latency and cost through parallel tool execution and dynamic model routing."

Interview Question 3: "How do you design an agentic system to be cost-effective without sacrificing quality?"

• How to answer: Introduce the Model Routing Pattern:
  • "I would implement an initial intent classification step using a fast, cost-effective model like GPT-4o-mini configured at temperature $0.0$. If the query is a simple lookup, the cheap model handles it. If it requires complex reasoning or empathy, the system dynamically routes the request to a high-tier model like Claude 3.5 Sonnet. This approach can reduce operational costs by over 50% while maintaining high response quality."

Summary and Key Takeaways

• Temperature controls token selection randomness: Temperature $0.0$ is deterministic (argmax selection), while temperature $1.0$ approaches random sampling.
• Top P limits the token pool: Top P restricts token selection to a cumulative probability mass, and tuning both Temperature and Top P simultaneously is a production anti-pattern.
• Agent tasks require different temperatures: Use temperature $0.0$ for classification and tool calling, and temperature $0.1$ to $0.3$ for natural yet consistent response generation.
• Architecture compensates for LLM limits: Every major agentic pattern (RAG, memory, tool calling) exists to address a core LLM limitation (hallucination, context limits, lack of real-time knowledge, latency/cost).
• Hallucination is structural: Hallucination is an inherent characteristic of next-token prediction, not a temporary bug. It must be managed using RAG and output validation.
• Model Routing optimizes cost and latency: Routing simple tasks to utility models (e.g., GPT-4o-mini) and complex tasks to premium models (e.g., Claude Sonnet) can reduce production costs by over 50%.
• Select models across five dimensions: Evaluate models based on reasoning quality, context window size, latency, cost per token, and availability/SLAs.