Online RAG Evaluation: Monitoring Production LLMs with LangSmith

Online evaluation is the discipline of continuously monitoring, testing, and scoring deployed Large Language Model (LLM) and Retrieval-Augmented Generation (RAG) applications against live user traffic. While offline evaluation relies on curated golden datasets to measure system regressions prior to release, online evaluation guarantees that edge cases, data drift, and operational bottlenecks are detected in real time within production environments.

1. The Offline vs. Online Evaluation Paradigm

Transitioning a RAG application from staging to production shifts the engineering focus from static benchmarking to continuous operational surveillance:

• Offline Evaluation: Conducted prior to deployment within CI/CD pipelines. It utilizes static "golden datasets" containing predefined queries, target contexts, and ground-truth references. It determines whether pipeline modifications cause performance regressions.
• Online Evaluation: Operates on live, uncurated user traffic post-deployment. The primary objective is observability, continuous quality assessment, and immediate alerting when live outputs degrade.

2. Reference-Based vs. Reference-Free Metrics

The primary technical boundary between offline and online evaluation is the availability of ground-truth reference data. In an offline setting, every sample contains human-verified target chunks and model responses. In a production online setting, arbitrary end users issue novel questions for which no predefined "correct" answer exists.

3. The Core Online Eval Architecture

A production online evaluation suite must avoid adding latency to user requests while providing complete visibility across system health. The minimal viable online suite consolidates six core metrics:

1. Zero Runtime Overhead: Tracing and evaluation are decoupled from the user-facing request path. Operational telemetry is extracted asynchronously during execution.
2. Safety Auditing: Live outputs pass through asynchronous safety classifiers to flag dangerous or brand-damaging responses.
3. Continuous Quality Scoring: Core RAG Triad metrics ensure context retrieval remains pertinent and hallucinations are caught mid-flight.

4. Instrumenting the Application with LangSmith

LangSmith provides native tracing for LLM applications. However, default auto-instrumentation often captures top-level generation components while missing retrieval and reranking steps, leading to inaccurate latency and cost reporting.

Resolving Trace Blind Spots

When LangChain or raw Python pipelines run, default hooks may trace only the ChatOpenAI invocation and StrOutputParser. The vector search and reranker execution times are excluded from the root trace:

To resolve this, explicit @traceable decorators must be attached to the retriever and the orchestration entry points.

Instrumentation Implementation

Install the client SDK:

uv add langsmith

Configure the environment variables in .env:

LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
LANGSMITH_API_KEY="lsv2_pt_..."
LANGSMITH_PROJECT="CX Doubt Solve"
OPENAI_API_KEY="sk-proj-..."

Decorate custom RAG components explicitly:

# src/reranker.py
from langsmith import traceable

class Reranker:
    def __init__(self):
        # Initialization logic
        pass

    @traceable(name="Retriever")
    def invoke(self, query: str, top_k: int = 3):
        # 1. Similarity search in vector store
        # 2. Cross-encoder scoring and sorting
        return retrieved_documents

# src/rag_pipeline.py
from langsmith import traceable
from src.reranker import Reranker
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser

reranker = Reranker()
llm = ChatOpenAI(model="gpt-4o-mini")
parser = StrOutputParser()

@traceable(name="RAG Pipeline")
def run_pipeline(question: str) -> dict:
    context_docs = reranker.invoke(question)
    
    prompt = ChatPromptTemplate.from_template(
        "Answer the question using the context:\nContext: {context}\nQuestion: {question}"
    )
    chain = prompt | llm | parser
    
    answer = chain.invoke({"context": context_docs, "question": question})
    
    return {
        "question": question,
        "context": [doc.page_content for doc in context_docs],
        "answer": answer
    }

Note: Explicitly tracing sub-components does not cause duplicate parent traces. LangSmith structures traces hierarchically, nesting decorated functions underneath the active parent run context.

5. Operational Monitoring: Latency, Cost, and Alert Semantics

Once traces stream into the observability backend, operational health is tracked via statistical aggregations:

Latency Distributions

• Average Latency: Measures total run execution time divided by total requests. Represents the experience of the broad user base.
• Percentile Latency ($P_{95}$, $P_{99}$): Measures the tail threshold below which $95\%$ or $99\%$ of requests fall. Quantifies degraded performance experienced by the least fortunate user segment.

Alerting Semantics: Average vs. Tail Percentiles

LangSmith restricts automated latency alert triggers to Average Run Execution Time over rolling time windows (e.g., 60 minutes) rather than $P_{95}$ or $P_{99}$:

• Rationale: Tail percentiles ($P_{95}$) inherently measure outlier behavior. In low-to-medium traffic distributions, sporadic outliers or long user queries can skew tail metrics without indicating systematic infrastructure collapse.
• Service Level Objectives (SLOs): Latency alert thresholds are governed by business objectives, not arbitrary engineering preferences. An interactive chatbot might enforce an average latency SLO of $< 3\text{ seconds}$, whereas a multi-step research agent might permit an SLO of $< 60\text{ seconds}$.

Cost Tracking

Input tokens and output tokens have distinct cost structures (typically output generation is billed at $3\times$ to $4\times$ the rate of input processing). Dashboards monitor:

• Total Operational Spend: Aggregate dollar cost over rolling 24-hour periods.
• Average Cost per Request: Measures efficiency per question answered:

6. Production Safety Guardrails: Automated Toxicity Detection

Online safety evaluations catch abusive inputs, hallucinations, or prompt injection exploits before reputational harm occurs.

Built-in Evaluator Configuration

LangSmith provides pre-configured evaluators for toxicity, PII leaks, and prompt injection:

1. Target: Attach to live Traces (not static Datasets).
2. Judge Model: Configure a dependable judge (such as GPT-4o).
3. Scoring Mechanism: Standardized across $[0, 100]$ or $[0.0, 1.0]$, accompanied by chain-of-thought textual justifications explaining why a score was assigned.

The Mathematics of the Sampling Rate

Evaluating $100\%$ of production queries with an auxiliary LLM judge doubles or triples operational inference costs. Instead, teams apply a uniform random sampling rate $S \in (0, 1]$:

• For high-volume services ($>100{,}000\text{ requests/day}$), an $S = 0.05$ to $0.10$ ($5\%\text{ to }10\%$) sample provides statistically valid representations of system behavior while constraining evaluation costs to a predictable fraction of total operating budget.

7. Quality Evaluation in Production: The Baseline Parity Rule

A critical architectural constraint governs the choice of evaluation tooling for online quality:

The Baseline Parity Rule: For continuous quality metrics (Contextual Relevancy, Faithfulness, Answer Relevancy), the online evaluation engine must execute the exact same code, prompt logic, and scoring models used during offline evaluation.

Why Evaluator Parity is Mandatory

Evaluators from different frameworks (e.g., DeepEval vs. LangSmith native prompts) do not share calibrated baselines:

Scenario: Switching Evaluators Between Stages
Offline Evaluation (DeepEval):    Contextual Relevancy = 0.45
Online Evaluation (LangSmith):   Contextual Relevancy = 0.55
Conclusion: INVALID. The +0.10 delta is an artifact of divergent judge prompts,
            differing tokenizers, and varying metric definitions, NOT a pipeline improvement.

Scenario: Enforcing Baseline Parity
Offline Baseline (DeepEval):     Contextual Relevancy = 0.45
Online Production (DeepEval):    Contextual Relevancy = 0.38
Conclusion: VALID. A statistically significant regression (-0.07) has occurred
            under real-world traffic conditions.

While utilizing third-party built-in judges is acceptable for safety metrics (where the objective is binary classification of explicit violations), quality metrics track nuanced continuous scores. Switching evaluation harnesses between staging and production invalidates trend analysis.

8. Background Evaluation Architecture: Continuous Asynchronous Scoring Loop

Because offline quality evaluators (like DeepEval) run outside of LangSmith's internal servers, an automated background service bridges the two systems.

Complete Implementation: eval_online.py

import os
import time
from datetime import datetime, timedelta, timezone
from langsmith import Client
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric
)
from deepeval.test_case import LLMTestCase

# Initialize Clients and Parameters
LANGSMITH_PROJECT = "CX Doubt Solve"
POLL_INTERVAL_SECONDS = 60
SAMPLING_RATE = 1.0  # 100% for demonstration; adjust based on budget in prod
EVAL_MODEL = "gpt-4o-mini"
THRESHOLD = 0.7

client = Client()

# Instantiate identical metrics used in offline evaluation
faithfulness_metric = FaithfulnessMetric(threshold=THRESHOLD, model=EVAL_MODEL)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=THRESHOLD, model=EVAL_MODEL)
contextual_relevancy_metric = ContextualRelevancyMetric(threshold=THRESHOLD, model=EVAL_MODEL)

def evaluate_and_post_feedback(run):
    """Executes DeepEval evaluations on an individual LangSmith trace run."""
    try:
        # Extract inputs and outputs from trace metadata
        inputs = run.inputs
        outputs = run.outputs

        if not inputs or not outputs:
            return

        question = inputs.get("question")
        answer = outputs.get("answer")
        context = outputs.get("context", [])

        if not question or not answer or not context:
            return

        # Build DeepEval Test Case
        test_case = LLMTestCase(
            input=question,
            actual_output=answer,
            retrieval_context=context if isinstance(context, list) else [context]
        )

        # 1. Faithfulness
        faithfulness_metric.measure(test_case)
        client.create_feedback(
            run_id=run.id,
            key="faithfulness",
            score=faithfulness_metric.score,
            comment=faithfulness_metric.reason
        )

        # 2. Answer Relevancy
        answer_relevancy_metric.measure(test_case)
        client.create_feedback(
            run_id=run.id,
            key="answer_relevancy",
            score=answer_relevancy_metric.score,
            comment=answer_relevancy_metric.reason
        )

        # 3. Contextual Relevancy
        contextual_relevancy_metric.measure(test_case)
        client.create_feedback(
            run_id=run.id,
            key="contextual_relevancy",
            score=contextual_relevancy_metric.score,
            comment=contextual_relevancy_metric.reason
        )

    except Exception as e:
        print(f"Failed to evaluate run {run.id}: {str(e)}")

def poll_and_score_recent_traces():
    """Fetches traces logged within the preceding polling window."""
    now = datetime.now(timezone.utc)
    start_time = now - timedelta(seconds=POLL_INTERVAL_SECONDS)

    # Fetch root project traces
    runs = list(client.list_runs(
        project_name=LANGSMITH_PROJECT,
        execution_order=1,  # Root runs only
        start_time=start_time
    ))

    print(f"[{now.strftime('%Y-%m-%d %H:%M:%S')}] Polled {len(runs)} traces for online evaluation.")

    for run in runs:
        # Apply sampling logic
        if SAMPLING_RATE < 1.0:
            import random
            if random.random() > SAMPLING_RATE:
                continue

        evaluate_and_post_feedback(run)

def main():
    print(f"Starting online evaluation daemon for project: '{LANGSMITH_PROJECT}'...")
    while True:
        poll_and_score_recent_traces()
        time.sleep(POLL_INTERVAL_SECONDS)

if __name__ == "__main__":
    main()

9. Visualizing Health: Custom Dashboards and Telemetry Aggregation

Once operational metadata, safety scores, and custom DeepEval feedback keys are attached to traces, they are aggregated into centralized monitoring dashboards.

Dashboard Layout Configuration

Custom monitoring consoles aggregate time-series panels over selectable windows (Last 1 Hour, Last 24 Hours, Last 30 Days):

10. The Flywheel Effect: Turning Production Failures into Golden Datasets

Online evaluation is not merely an alerting mechanism; it forms an active data flywheel that continuously hardens offline evaluation suites.

The Closed-Loop Feedback Process

1. Failure Discovery: An anomalous production question yields poor retrieval and low faithfulness ($< 0.50$).
2. Trace Triage: The trace is inspected within LangSmith. The engineer selects "Add to Dataset" and targets a managed dataset (e.g., Golden-Production-Failures).
3. Human-in-the-Loop Curation: The model's hallucinated or incorrect answer is discarded. A human subject-matter expert writes an authoritative reference response.
4. Dataset Versioning: LangSmith natively versions the dataset, tracking historical states across team commits.
5. Offline Ingestion: Offline regression suites ingest this updated dataset directly via the LangSmith SDK during local development and pre-merge CI/CD checks:

from langsmith import Client

client = Client()
# Download authoritative, production-updated golden dataset
dataset = client.read_dataset(dataset_name="Golden-Production-Failures")
examples = list(client.list_examples(dataset_id=dataset.id))

This loop guarantees that the application never regresses on identical or structurally similar production queries in future iterations.

11. Summary and Key Takeaways

• Online Eval Operates Reference-Free: Because true user queries lack predefined answers, reference-based metrics (Correctness, Completeness, Context Recall, Context Precision) cannot run online without human intervention.
• The Core Online Suite Consists of Six Metrics: Latency and Cost (Operational), Toxicity (Safety), and Contextual Relevancy, Faithfulness, and Answer Relevancy (The RAG Triad).
• Explicit Tracing is Required: Default library hooks frequently miss retrieval and reranking layers. Custom steps must be explicitly annotated using the @traceable decorator to capture true end-to-end execution times.
• Latency Alerts Track Averages, Not Tail Outliers: Operational alerting engines focus on average trace latency over rolling windows to detect systemic degradation while preventing false-positive alerts driven by isolated $P_{95}/P_{99}$ outlier queries.
• Alert Thresholds Derive from Business SLOs: Latency and cost boundaries are not arbitrary engineering variables; they are dictated by Service Level Objectives tailored to the product's interactive requirements.
• Sampling Controls Evaluator Expense: Running LLM-as-a-judge evaluations on $100\%$ of production traces causes severe cost inflation. Production architectures enforce random sampling rates ($S \in [0.05, 0.30]$) to balance statistical confidence with inference budgets.
• Preserve Metric Parity Across Boundaries: Online quality scoring must use the exact same evaluation harness, model, and prompt templates (e.g., via DeepEval) utilized in offline testing. Divergent evaluator frameworks destroy comparative baselines.
• Asynchronous Daemon Pattern for Custom Evaluators: An external polling script or scheduled cron job (eval_online.py) queries recent traces through the SDK, executes offline-grade evaluators asynchronously, and pushes structured scores back to LangSmith traces as feedback.
• Production Failures Fuel the Golden Dataset: A robust RAG lifecycle pipes production anomalies into managed, versioned datasets where human experts attach verified ground truths, continuously fortifying offline regression suites against real-world failures.