Observability in GenAI: Masterclass on LangSmith and LLM Ops

Modern Large Language Model (LLM) applications are non-deterministic, multi-stage software systems. Unlike traditional software architectures where identical inputs yield deterministic outputs, LLM-based systems can produce variable responses, introduce latency spikes, incur unexpected operational costs, or hallucinate answers under subtle parameter changes.

This masterclass explores LangSmith—a comprehensive observability, debugging, monitoring, and evaluation platform built specifically for Generative AI applications. Through real-world failure scenarios, practical code walkthroughs, and deep integrations across LangChain and LangGraph, these notes establish the operational methodologies required to turn black-box AI workflows into transparent, production-ready systems.

1. The Production Crisis of Non-Deterministic AI

To understand the necessity of specialized observability tools like LangSmith, we must examine how LLM applications fail in production environments. Traditional application monitoring tools (such as Datadog or New Relic) track system-level metrics like CPU utilization, HTTP response codes, and memory allocation. However, they lack visibility into internal model reasoning, prompt formatting, document retrieval relevance, and token consumption.

Structure & Concept Flow

Traditional Software (Deterministic)

Input A ----> [ Fixed Business Logic ] ----> Output B (Always Identical)

LLM Software (Non-Deterministic)

Input A ----> [ Multi-Stage LLM Graph ] ---> Output B (Variable / Probabilistic)

Scenario 1: Unexplained Latency Spikes in Multi-Stage Workflows

System Architecture: An AI-powered job application tool for university graduates.
Workflow Flow:
1. Parse a job description (JD) URL or PDF.
2. Retrieve student portfolio documents, resumes, and project details from Google Drive.
3. Perform skill-matching between the student profile and the JD.
4. Generate a tailored cover letter.
5. Proofread the cover letter for tone, alignment, and conversion probability.
The Failure: Baseline system latency is ~2 minutes. Suddenly, user complaints surge as end-to-end processing time spikes to 7–10 minutes.
The Debugging Dilemma: Without granular tracing, developers only see total elapsed time ( $T_{\text{total}} = 600\text{s}$ ). They cannot isolate whether the delay originated in the LLM proofreading step, the JD parsing model, or an unoptimized Google Drive API call scanning an entire cloud folder instead of a targeted directory.

Scenario 2: Exponential Cost Explosion in Autonomous Agents

System Architecture: An autonomous academic research assistant agent.
Workflow Flow:
1. Receive a research topic (e.g., "Solar Energy Innovations").
2. Fetch academic papers from Google Scholar or arXiv.
3. Study paper contents and extract key insights.
4. Summarize insights into a final structured report.
5. Provide an interactive Q&A interface over the report.
The Failure: Baseline execution cost is $\char36 0.50$ per report. Suddenly, openAI API billing dashboards show unexpected cost spikes where individual report runs consume up to $\char36 2.00$ .
The Root Cause: A minor prompt refinement intended to boost report quality ("Continue refining the report until the quality is outstanding") caused the autonomous agent to enter an infinite perfectionism loop. The agent iteratively re-downloaded, re-parsed, and re-summarized documents when it deemed the quality unsatisfactory.
The Debugging Dilemma: No code exceptions or HTTP errors were logged. The software executed successfully, but non-deterministic agent loop iterations consumed excess input/output tokens without developer visibility.

Scenario 3: Hallucinations in Enterprise RAG Systems

System Architecture: An internal HR Chatbot for enterprise onboarding (e.g., TCS freshers).
Workflow Flow:
1. Employee submits a query (e.g., "What is the leave policy during probation?").
2. Vector database retrieves relevant policy document chunks.
3. LLM synthesizes retrieved context into an answer.
The Failure: The chatbot hallucinates incorrect policy details ("Employees can take unrestricted leave and travel to Goa at any time during probation"), causing organizational misinformation.
The Debugging Dilemma: RAG failures stem from two distinct sources:
1. Retriever Failure: The vector database returned irrelevant or incomplete document chunks ( $n=1$ top- $k$ document was insufficient).
2. Generator Failure: The retriever provided correct context, but the LLM system prompt was too lenient, allowing the model to extrapolate beyond the provided text instead of outputting "I don't know".

Without step-by-step intermediate state inspection, identifying whether to tune the retriever or rewrite the generator prompt is impossible.

2. Demystifying Observability in GenAI Systems

Definition of Observability

Observability is the capability to infer the internal execution states of a system by analyzing its external outputs—specifically logs, metrics, execution traces, and intermediate payloads.

Observability vs. Monitoring:

Monitoring tells you that a system is failing (e.g., "System error rate is 12%").

Observability tells you why a system is failing, even when encountering unforeseen edge cases (e.g., "Step 3 of the prompt template received empty context because the retriever $k$ -value was set to 1").

Core Execution Metrics Tracked by LangSmith

When an LLM application runs under LangSmith observation, the platform captures granular execution telemetry at every boundary:

Metric Category	Telemetry Captured	Practical Utility
Inputs & Outputs	Raw prompt strings, system messages, variable injections, generated completions.	Inspect exact payloads sent to model APIs and raw outputs before parsing.
Intermediate States	Retriever document chunks, tool invocation parameters, agent scratchpads.	Isolate state corruption or context loss across multi-step chains.
Execution Latency	End-to-end trace runtime and individual sub-step duration ( $T_{\text{step}}$ ).	Identify slow API calls, bottleneck nodes, or runaway loops.
Token Usage & Cost	Prompt tokens, completion tokens, dollar cost derived from model pricing cards.	Monitor spending, optimize prompt lengths, and select cost-effective models.
Exception Tracing	Stack traces, API rate-limit errors, schema parsing failures.	Rapidly locate broken code paths and failed schema validations.
Tags & Metadata	Model identifiers (`gpt-4o-mini`), framework versions, custom user IDs.	Filter, search, and benchmark execution variations across deployments.
User Feedback	Binary thumbs up/down ratings, explicit user score overlays.	Align telemetry data with real-world user satisfaction metrics.

3. Core Architecture & Conceptual Hierarchy

LangSmith organizes operational telemetry into a three-tiered structural hierarchy: Projects, Traces, and Runs.

Structure & Concept Flow

Project: Enterprise HR Chatbot

+-- Trace 1 (User Request Lifecycle: "What is the notice period?")

| |

| +-- Run 1: PromptTemplate Formatting

| +-- Run 2: VectorStore Document Retrieval

| +-- Run 3: ChatOpenAI Invocation (GPT-4o)

| +-- Run 4: StrOutputParser Execution

+-- Trace 2 (User Request Lifecycle: "Explain health insurance")

+-- Run 1: PromptTemplate Formatting

+-- Run 2: VectorStore Document Retrieval

+-- ...

Project: The top-level logical container representing an LLM application or environment (e.g., RAG-Chatbot-Prod, Sequential-Report-App). All execution logs route to a designated project.
Trace: A single end-to-end execution lifecycle of an application, triggered by a top-level user input or API invocation. A trace encapsulates the entire path from initial request to final response.
Run: An individual execution unit within a trace. Each constituent component of an application—a prompt template construction, an LLM call, a document retriever search, a custom Python function, or an output parser—is logged as a distinct Run with its own inputs, outputs, latency, and token metrics.

4. Setting Up the Environment & API Configurations

LangSmith integrates seamlessly with Python application runtimes via environment variables, requiring zero invasive code modifications for basic LangChain workflows.

1. Directory Structure Setup

BASH

# Terminal setup commands
mkdir langsmith-masterclass
cd langsmith-masterclass
python -m venv myenv
source myenv/bin/activate  # On Windows: myenv\Scripts\activate

2. Dependency Installation

Create a requirements.txt file containing the core GenAI stack:

TEXT

langchain==0.2.11
langchain-community==0.2.10
langchain-openai==0.1.19
langgraph==0.1.19
langsmith==0.1.93
pypdf==4.3.1
faiss-cpu==1.8.0.post1
python-dotenv==1.0.1
duckduckgo-search==6.2.7

Install via pip:

BASH

pip install -r requirements.txt

3. Environment Variable Configuration (`.env`)

Create a .env file in your root project directory:

BASH

# OpenAI Key
OPENAI_API_KEY=sk-proj-xxxx...

# LangSmith Tracing Configuration
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=lsv2_pt_xxxx...
LANGCHAIN_PROJECT=LangSmith-Demo

Note: Setting LANGCHAIN_TRACING_V2=true instructs the underlying LangChain primitives to automatically dispatch telemetry payloads asynchronously to the LangSmith backend using the specified API key and endpoint.

5. Tracing Simple & Sequential Chains in LangChain

1. Basic LLM Call Tracing

When executing standard LangChain chains (PromptTemplate | ChatOpenAI | StrOutputParser), LangSmith hooks into the invocation without needing custom tracing logic.

PYTHON

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

# Load environment configuration
load_dotenv()

# Define chain components
prompt = ChatPromptTemplate.from_template("What is the capital of {country}?")
model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
parser = StrOutputParser()

# Construct Expression Language (LCEL) chain
chain = prompt | model | parser

# Execute chain
response = chain.invoke({"country": "Peru"})
print("Response:", response)

LangSmith UI Trace Output:

Project: LangSmith-Demo
Trace Name: RunnableSequence
Child Runs:
1. ChatPromptTemplate (Inputs: country='Peru', Output: Formatted Prompt)
2. ChatOpenAI (Inputs: Formatted System/Human Messages, Output: AI Message object)
3. StrOutputParser (Inputs: AI Message object, Output: "The capital of Peru is Lima.")

2. Sequential Chains with Custom Tags, Metadata, and Dynamic Run Names

To maintain organized trace repositories in production, you can inject explicit tags, runtime metadata, model configuration, and custom run names using the config parameter during chain invocation.

PYTHON

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

load_dotenv()

# Override default project name programmatically
os.environ["LANGCHAIN_PROJECT"] = "Sequential-Report-App"

# Initialize distinct models for distinct workflow stages
stage1_model = ChatOpenAI(model="gpt-4o-mini", temperature=0.7)
stage2_model = ChatOpenAI(model="gpt-4o", temperature=0.2)
parser = StrOutputParser()

# Stage 1: Generate comprehensive report
prompt1 = ChatPromptTemplate.from_template("Generate a detailed report on {topic}.")
chain1 = prompt1 | stage1_model | parser

# Stage 2: Summarize report into key takeaways
prompt2 = ChatPromptTemplate.from_template("Generate a 5-point summary from the following report:\n{report}")
chain2 = prompt2 | stage2_model | parser

# Combined Sequential Workflow
topic = "Unemployment in India"
report_output = chain1.invoke({"topic": topic})
final_summary = chain2.invoke({"report": report_output})

# Invocation with Custom Configuration & Metadata Logging
config = {
    "run_name": "Sequential_Report_and_Summary_Chain",
    "tags": ["Production", "Report-Generator", "HR-Analytics"],
    "metadata": {
        "stage1_model": "gpt-4o-mini",
        "stage2_model": "gpt-4o",
        "input_topic": topic,
        "environment": "staging"
    }
}

# Executing with metadata injection
res = (prompt1 | stage1_model | parser | prompt2 | stage2_model | parser).invoke(
    {"topic": topic}, 
    config=config
)

print(res)

6. Case Study 1: Tracing Enterprise Retrieval-Augmented Generation (RAG)

The Architectural Problem

Standard LangChain tracing captures explicit LCEL chain components. However, pre-chain operations—such as PDF loading, document chunking, and vector index generation—execute as raw Python code. By default, these remain uncaptured in LangSmith, leaving developers blind to data pipeline performance and storage bottlenecks.

To achieve complete observability across custom Python code, LangSmith provides the @traceable decorator.

Structure & Concept Flow

Raw PDF File

[ @traceable: load_pdf() ] ------------> Captures page count, load latency

[ @traceable: split_documents() ] ------> Captures chunk sizes, total chunks

[ @traceable: build_vector_store() ] ---> Captures embedding model, index runtime

[ RAG LCEL Chain Ingestion ] -----------> Captures retriever, prompt, LLM output

Complete Implementation: Optimizing Latency with Faiss Index Caching

The following implementation builds a PDF RAG system over the machine learning textbook An Introduction to Statistical Learning. It uses @traceable decorators for non-LCEL functions and implements local index persistence to prevent redundant PDF parsing and vector generation on subsequent executions.

PYTHON

import os
import time
from dotenv import load_dotenv
from langsmith import traceable

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser

load_dotenv()
os.environ["LANGCHAIN_PROJECT"] = "RAG-Enterprise-System"

PDF_PATH = "Introduction_to_Statistical_Learning.pdf"
INDEX_PATH = "faiss_index_cache"

# --- TRACEABLE CUSTOM FUNCTIONS ---

@traceable(name="Load_PDF_Document", tags=["ETL", "PDF-Loader"])
def load_pdf(path: str):
    loader = PyPDFLoader(path)
    docs = loader.load()
    return docs

@traceable(name="Split_Documents_Into_Chunks", tags=["ETL", "Text-Splitter"])
def split_documents(docs, chunk_size=1000, chunk_overlap=150):
    splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    split_docs = splitter.split_documents(docs)
    return split_docs

@traceable(name="Build_or_Load_Vector_Index", tags=["Indexing", "FAISS"])
def get_vector_store(split_docs=None):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    
    # Cache optimization: Check if index exists on disk
    if os.path.exists(INDEX_PATH):
        print("Loading cached FAISS index from disk...")
        vectorstore = FAISS.load_local(INDEX_PATH, embeddings, allow_dangerous_deserialization=True)
    else:
        print("Generating new embeddings and building FAISS index...")
        vectorstore = FAISS.from_documents(split_docs, embeddings)
        vectorstore.save_local(INDEX_PATH)
        
    return vectorstore

@traceable(name="Format_Retrieved_Documents", tags=["RAG", "Formatter"])
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# --- PIPELINE INITIALIZATION ---

@traceable(name="Setup_RAG_Pipeline", tags=["Pipeline-Setup"])
def setup_pipeline():
    if not os.path.exists(INDEX_PATH):
        docs = load_pdf(PDF_PATH)
        split_docs = split_documents(docs)
        vectorstore = get_vector_store(split_docs)
    else:
        vectorstore = get_vector_store()
    return vectorstore.as_retriever(search_kwargs={"k": 3})

# Initialize Retriever
retriever = setup_pipeline()

# --- RAG CHAIN EXECUTION ---

prompt = ChatPromptTemplate.from_template("""
Answer the question using ONLY the provided context below.
If the answer cannot be derived from the context, state "I don't know".

Context:
{context}

Question:
{question}
""")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Parallel retrieval & query pass-through chain
rag_chain = (
    RunnableParallel({
        "context": retriever | format_docs,
        "question": RunnablePassthrough()
    })
    | prompt
    | llm
    | StrOutputParser()
)

# Execute RAG Query
query = "What is the difference between classification and regression?"
response = rag_chain.invoke(
    query, 
    config={"run_name": "RAG_Query_Execution", "tags": ["User-Query", "RAG"]}
)

print("\n--- Response ---")
print(response)

Trace Analysis & Latency Metrics

When executing the pipeline for the first time vs. subsequent runs, LangSmith records the latency improvement achieved through index caching:

Structure & Concept Flow

First Run Execution (No Index Cache):

RAG-Enterprise-System (Trace) [Latency: ~28.5s]

|-- Setup_RAG_Pipeline [Latency: ~26.2s]

| |-- Load_PDF_Document [Latency: 14.1s] (Loaded 441 pages)

| |-- Split_Documents_Into_Chunks [Latency: 1.2s] (Generated 2,100 chunks)

| |-- Build_or_Load_Vector_Index [Latency: 10.9s] (Created FAISS embeddings)

|-- RAG_Query_Execution [Latency: ~2.3s]

|-- RunnableParallel [Latency: ~0.8s] (Retrieved top 3 docs)

|-- ChatOpenAI [Latency: ~1.5s] (Generated answer)

Subsequent Run Execution (With Disk Cache):

RAG-Enterprise-System (Trace) [Latency: ~1.8s]

|-- Setup_RAG_Pipeline [Latency: ~0.2s]

| |-- Build_or_Load_Vector_Index [Latency: 0.2s] (Loaded index from cache)

|-- RAG_Query_Execution [Latency: ~1.6s]

|-- ChatOpenAI [Latency: ~1.4s]

7. Case Study 2: Tracing Autonomous React Agents

The ReAct (Reasoning + Action) Loop

Autonomous agents evaluate user input, determine required tool actions, observe tool execution outputs, and loop through reasoning cycles until reaching a final answer.

Structure & Concept Flow

User Query

[ Prompt + History ] ---> [ LLM Reasoning ] ---> Final Answer? (Yes) -> Return

^ |

| v (No)

+--- [ Scratchpad ] <--- [ Execute Tool ]

Because agent execution is non-deterministic and multi-iterative, tracing the internal state of the agent's Scratchpad across every loop iteration is vital to prevent runaway execution costs.

Complete React Agent Implementation with Custom & Community Tools

The following code implements a ReAct agent using LangChain, integrated with a DuckDuckGo Search tool and a custom Python weather fetcher tool.

PYTHON

import os
import requests
from dotenv import load_dotenv
from langsmith import traceable

from langchain_openai import ChatOpenAI
from langchain_community.tools import DuckDuckGoSearchRun
from langchain.agents import create_react_agent, AgentExecutor
from langchain_core.tools import tool
from langchain import hub

load_dotenv()
os.environ["LANGCHAIN_PROJECT"] = "Autonomous-ReAct-Agents"

# --- TOOL DEFINITIONS ---

search_tool = DuckDuckGoSearchRun()

@tool
@traceable(name="Weather_API_Tool", tags=["Tool", "Weather"])
def get_weather_data(city: str) -> str:
    """Fetch real-time weather information for a specified city."""
    # Open-Meteo Public API (No API Key Required)
    geocoding_url = f"https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1"
    geo_res = requests.get(geocoding_url).json()
    
    if not geo_res.get("results"):
        return f"Could not find coordinates for city: {city}"
        
    lat = geo_res["results"][0]["latitude"]
    lon = geo_res["results"][0]["longitude"]
    
    weather_url = f"https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lon}&current_weather=true"
    w_res = requests.get(weather_url).json()
    
    current = w_res.get("current_weather", {})
    temp = current.get("temperature")
    wind = current.get("windspeed")
    
    return f"Current temperature in {city}: {temp}°C, Wind Speed: {wind} km/h."

tools = [search_tool, get_weather_data]

# --- AGENT INITIALIZATION ---

# Pull canonical ReAct prompt template from LangChain Hub
prompt = hub.pull("hwchase17/react")
llm = ChatOpenAI(model="gpt-4o", temperature=0)

agent = create_react_agent(llm, tools, prompt)
agent_executor = AgentExecutor(
    agent=agent, 
    tools=tools, 
    verbose=True, 
    handle_parsing_errors=True
)

# --- EXECUTION WITH MULTI-TOOL HOPPING ---

query = "Find the birthplace of Kalpana Chawla and tell me the current weather there."

response = agent_executor.invoke(
    {"input": query},
    config={"run_name": "MultiTool_ReAct_Execution", "tags": ["Agent", "ReAct", "Weather-Search"]}
)

print("\n--- Final Agent Output ---")
print(response["output"])

Trace Breakdown in LangSmith

LangSmith maps the agent's iterative execution lifecycle:

Structure & Concept Flow

MultiTool_ReAct_Execution (Trace)

+-- Iteration 1: Reasoning Stage

| |-- Prompt formatting (Input: "Find birthplace of Kalpana Chawla...")

| |-- ChatOpenAI Invocation -> Output Thought: "I need to search for Kalpana Chawla's birthplace."

| |-- Tool Call: duckduckgo_search("Kalpana Chawla birthplace") -> Observation: "Karnal, Haryana, India"

+-- Iteration 2: Reasoning Stage (Scratchpad updated with Iteration 1 Observation)

| |-- ChatOpenAI Invocation -> Output Thought: "Birthplace is Karnal. Now fetch weather for Karnal."

| |-- Tool Call: Weather_API_Tool("Karnal") -> Observation: "Current temperature in Karnal: 27°C..."

+-- Iteration 3: Final Response Generation

|-- ChatOpenAI Invocation -> Output Thought: "I have all information."

|-- Final Output: "Kalpana Chawla was born in Karnal. The current weather in Karnal is 27°C."

8. Case Study 3: Observability in Complex Workflows with LangGraph

LangGraph Mental Model

LangGraph models LLM applications as State Graphs, where:

Nodes represent computational steps or LLM invocations.
Edges define execution flow and conditional branching logic based on state updates.
State is a shared data structure mutated across node operations.

LangSmith Mapping Mapping:

Entire Graph Execution = $1\text{ Trace}$

Individual Graph Node Execution = $1\text{ Run}$ within the Trace.

Structure & Concept Flow

[ Input Essay ]

+----------------+----------------+

| | |

v v v

[ Node: Lang ] [ Node: Anal ] [ Node: Thought ]

| | |

+----------------+----------------+

[ Node: Final Synthesis ]

[ Output ]

Complete Implementation: Parallel Criteria Essay Evaluator

The following application evaluates an essay across three parallel dimension nodes (evaluate_language, evaluate_analysis, and evaluate_thought) before aggregating scores in a final node.

PYTHON

import os
from typing import TypedDict, Annotated, List
from dotenv import load_dotenv
from pydantic import BaseModel, Field
from langsmith import traceable

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langgraph.graph import StateGraph, START, END

load_dotenv()
os.environ["LANGCHAIN_PROJECT"] = "LangGraph-Essay-Evaluator"

# --- PYDANTIC SCHEMAS FOR STRUCTURED OUTPUT ---

class DimensionEvaluation(BaseModel):
    feedback: str = Field(description="Detailed feedback on this specific evaluation dimension.")
    score: int = Field(description="Score out of 5 for this dimension.")

# --- GRAPH STATE DEFINITION ---

class EssayState(TypedDict):
    essay: str
    language_feedback: str
    language_score: int
    analysis_feedback: str
    analysis_score: int
    thought_feedback: str
    thought_score: int
    final_synthesis: str
    average_score: float

# --- MODEL INITIALIZATION WITH STRUCTURED OUTPUT ---

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
structured_evaluator = llm.with_structured_output(DimensionEvaluation)

# --- NODE FUNCTIONS ---

@traceable(name="Evaluate_Language_Node", tags=["Graph-Node", "Language"])
def evaluate_language(state: EssayState):
    prompt = ChatPromptTemplate.from_template("Evaluate the vocabulary and grammar of this essay:\n\n{essay}")
    res = (prompt | structured_evaluator).invoke({"essay": state["essay"]})
    return {"language_feedback": res.feedback, "language_score": res.score}

@traceable(name="Evaluate_Analysis_Node", tags=["Graph-Node", "Analysis"])
def evaluate_analysis(state: EssayState):
    prompt = ChatPromptTemplate.from_template("Evaluate the depth of argument and evidence in this essay:\n\n{essay}")
    res = (prompt | structured_evaluator).invoke({"essay": state["essay"]})
    return {"analysis_feedback": res.feedback, "analysis_score": res.score}

@traceable(name="Evaluate_Thought_Node", tags=["Graph-Node", "Thought-Process"])
def evaluate_thought(state: EssayState):
    prompt = ChatPromptTemplate.from_template("Evaluate the logical structure and flow of this essay:\n\n{essay}")
    res = (prompt | structured_evaluator).invoke({"essay": state["essay"]})
    return {"thought_feedback": res.feedback, "thought_score": res.score}

@traceable(name="Final_Synthesis_Node", tags=["Graph-Node", "Aggregator"])
def final_synthesis(state: EssayState):
    summary_prompt = ChatPromptTemplate.from_template("""
Synthesize the following evaluation dimension feedbacks into an overall evaluation summary.

Language Feedback ({lang_score}/5): {lang_fb}
Analysis Feedback ({anal_score}/5): {anal_fb}
Thought Process Feedback ({thought_score}/5): {thought_fb}
""")
    res = (summary_prompt | llm).invoke({
        "lang_score": state["language_score"], "lang_fb": state["language_feedback"],
        "anal_score": state["analysis_score"], "anal_fb": state["analysis_feedback"],
        "thought_score": state["thought_score"], "thought_fb": state["thought_feedback"]
    })
    
    avg_score = (state["language_score"] + state["analysis_score"] + state["thought_score"]) / 3.0
    return {"final_synthesis": res.content, "average_score": avg_score}

# --- BUILD STATE GRAPH ---

builder = StateGraph(EssayState)

# Add Nodes
builder.add_node("evaluate_language", evaluate_language)
builder.add_node("evaluate_analysis", evaluate_analysis)
builder.add_node("evaluate_thought", evaluate_thought)
builder.add_node("final_synthesis", final_synthesis)

# Add Parallel Edges from START
builder.add_edge(START, "evaluate_language")
builder.add_edge(START, "evaluate_analysis")
builder.add_edge(START, "evaluate_thought")

# Join Parallel Nodes to Aggregator
builder.add_edge("evaluate_language", "final_synthesis")
builder.add_edge("evaluate_analysis", "final_synthesis")
builder.add_edge("evaluate_thought", "final_synthesis")

# Add Terminal Edge
builder.add_edge("final_synthesis", END)

# Compile Graph
graph = builder.compile()

# --- EXECUTE GRAPH ---

sample_essay = """
Artificial Intelligence is rapidly reshaping human society. While it provides high efficiency in data processing, 
it introduces labor displacement challenges. Modern governance frameworks must evolve to balance innovation with ethical safeguards.
"""

output = graph.invoke(
    {"essay": sample_essay},
    config={"run_name": "Parallel_Essay_Evaluation_Graph", "tags": ["LangGraph", "Essay-Evaluator"]}
)

print("--- FINAL EVALUATION SUMMARY ---")
print(output["final_synthesis"])
print(f"Average Score: {output['average_score']:.2f}/5.0")

9. Beyond Tracing: The Broader LLM Ops Ecosystem in LangSmith

Tracing single application runs provides immediate debugging capabilities. However, production systems require continuous maintenance across the broader LLM Ops (Large Language Model Operations) lifecycle. LangSmith offers five additional modules to support production deployments:

Structure & Concept Flow

+-----------------------------------+

| LangSmith LLM Ops Framework |

+-----------------------------------+

+-----------------+---------------+-----------------+-----------------+

| | | | |

v v v v v

[ Production [ System [ Prompt [ Dataset [ User Feedback

Monitoring ] Evaluation ] Playground ] Curation ] Integration ]

1. Production Monitoring & Custom Alerting

While tracing analyzes single executions, Monitoring aggregates telemetry across thousands of daily runs.

Key Aggregated Metrics: Latency distribution ( $p_{50}, p_{90}, p_{99}$ ), daily token spend, error frequencies, and tool invocation counts.
Proactive Alerting: Developers can set up automated alerts (e.g., webhook notifications sent to Slack) triggered when metrics breach defined operational limits:

\text{Trigger Alert IF } \text{Latency}_{p_{90}} > 5.0\text{ seconds OR Error Rate} > 2.5\%

2. System Evaluation & "LLM as a Judge"

To deploy prompt or architecture changes safely, applications must be evaluated against standardized benchmark datasets.

Evaluation Paradigms:
1. Heuristic Evaluation: Regex pattern matching, exact string match, JSON schema validation.
2. Semantic Similarity: Vector cosine similarity between generated output and ground-truth text.
3. LLM as a Judge: Utilizing an advanced model (e.g., GPT-4o) with explicit rubrics to grade generated responses on metrics like Faithfulness, Conciseness, and Hallucination Score.

3. Prompt Engineering, Playgrounds, & Versioning

Prompt Playground: An interactive UI environment in LangSmith allowing developers to test prompt variations side-by-side on logged production inputs without changing application code.
Prompt Hub & Versioning: Version-control prompt templates (e.g., hr-assistant-prompt:v2.1). Changes can be rolled back or tested in isolation, eliminating unversioned prompt modifications.

4. Dataset Curation & Annotation Queues

Trace-to-Dataset Export: Production traces containing failed runs or novel user queries can be converted into curated evaluation datasets with one click.
Annotation Queues: Human reviewers can manually label logged traces, assign quality scores, or correct output text to create high-quality ground-truth datasets for model fine-tuning.

5. User Feedback Integration & Collaboration

Feedback Ingestion: Connect front-end application feedback (e.g., thumbs-up/thumbs-down buttons) directly to individual LangSmith Trace IDs via API endpoints.
Team Collaboration: Shared trace URLs enable engineers, prompt designers, and product owners to inspect execution failures without needing local development environments.

10. Comparative Analysis of Debugging Strategies

The following table summarizes common LLM application failure modes, traditional debugging challenges, and the resolution provided by LangSmith:

Failure Scenario	Root Cause	Traditional Debugging Obstacle	LangSmith Resolution Workflow
Silent RAG Hallucination	Context returned by retriever lacked the requested information.	System output appears normal; no runtime exceptions logged.	Inspect the `Retriever` Run payload in the trace to confirm whether the required facts were present in the retrieved chunks.
Runaway Agent Cost	Agent entered a non-terminating reasoning loop due to an ambiguous prompt.	High API bills observed hours or days after execution.	Examine trace iteration counts, inspect intermediate scratchpad tokens, and apply loop termination conditions.
Late-Stage Output Parsing Error	Model output failed Pydantic schema validation.	Generic execution crash trace logged at the end of execution.	Isolate the `ChatOpenAI` Run directly preceding the parser to inspect raw response text vs. required JSON schema.
RAG Pipeline Latency Spike	Document loading and vector indexing ran synchronously on every query.	High overall HTTP request runtime logged in server APM.	Compare component run durations within the trace to identify slow PDF parsing vs. fast vector database lookup steps.

11. Summary and Key Takeaways

The Observability Imperative: Non-deterministic LLM applications require fine-grained execution tracing to diagnose unpredictable latency, high token costs, and logical hallucinations that traditional APM tools miss.
Core Hierarchy: LangSmith structures telemetry into Projects (the application container), Traces (single end-to-end request lifecycles), and Runs (individual execution steps within a trace).
Zero-Invasive Tracing: LangChain LCEL workflows automatically capture execution telemetry simply by exporting environment variables (LANGCHAIN_TRACING_V2=true).
Custom Code Telemetry: Non-LCEL operations—such as custom ETL scripts, PDF loaders, or raw Python utility functions—can be brought into execution traces using the @traceable decorator.
Production Optimization: RAG pipelines can significantly reduce execution latency (e.g., from ~28s down to ~1.8s) by caching persistent vector index stores on disk and using LangSmith trace metrics to verify performance.
Agent Transparency: Complex, multi-iteration ReAct loops are rendered fully visible in LangSmith, enabling developers to inspect thought-action-observation cycles and prevent infinite tool-use loops.
LangGraph Integration: State Graph executions map natively to LangSmith, where the full graph run represents the Trace and individual Nodes represent discrete Runs, simplifying the debugging of complex parallel and cyclic workflows.
Full-Lifecycle LLM Ops: Beyond tracing, LangSmith provides a complete ecosystem encompassing Production Monitoring, Automated System Evaluation (using LLM-as-a-Judge), Prompt Engineering Playgrounds, Dataset Curation, and User Feedback Integration.

LangSmith Crash Course | LangSmith Tutorial for Beginners | Observability in GenAI | CampusX

Observability in GenAI: Masterclass on LangSmith and LLM Ops

1. The Production Crisis of Non-Deterministic AI

Scenario 1: Unexplained Latency Spikes in Multi-Stage Workflows

Scenario 2: Exponential Cost Explosion in Autonomous Agents

Scenario 3: Hallucinations in Enterprise RAG Systems

2. Demystifying Observability in GenAI Systems

Definition of Observability

Core Execution Metrics Tracked by LangSmith

3. Core Architecture & Conceptual Hierarchy

4. Setting Up the Environment & API Configurations

1. Directory Structure Setup

2. Dependency Installation

3. Environment Variable Configuration (.env)

5. Tracing Simple & Sequential Chains in LangChain

1. Basic LLM Call Tracing

2. Sequential Chains with Custom Tags, Metadata, and Dynamic Run Names

6. Case Study 1: Tracing Enterprise Retrieval-Augmented Generation (RAG)

The Architectural Problem

Complete Implementation: Optimizing Latency with Faiss Index Caching

Trace Analysis & Latency Metrics

7. Case Study 2: Tracing Autonomous React Agents

The ReAct (Reasoning + Action) Loop

Complete React Agent Implementation with Custom & Community Tools

Trace Breakdown in LangSmith

8. Case Study 3: Observability in Complex Workflows with LangGraph

LangGraph Mental Model

Complete Implementation: Parallel Criteria Essay Evaluator

9. Beyond Tracing: The Broader LLM Ops Ecosystem in LangSmith

1. Production Monitoring & Custom Alerting

2. System Evaluation & "LLM as a Judge"

3. Prompt Engineering, Playgrounds, & Versioning

4. Dataset Curation & Annotation Queues

5. User Feedback Integration & Collaboration

10. Comparative Analysis of Debugging Strategies

11. Summary and Key Takeaways

3. Environment Variable Configuration (`.env`)