Building an Ultra-Fast Q&A Chatbot with Streamlit, Groq, and LangChain

This lesson walks step-by-step through engineering a production-grade, interactive Q&A chatbot using LangChain, the Groq API (running open-source models at high-throughput inference speeds exceeding typical web chat interfaces), Google SerpAPI for real-time external data grounding, and Streamlit for the frontend user interface.

The guide addresses core engineering hurdles: maintaining multi-turn context across UI redraws using LangGraph's MemorySaver and Streamlit's session_state, preventing state loss during reruns, rendering chat histories cleanly, and token-by-token real-time response streaming.

1. Architectural Overview & Component Pipeline

Modern conversational agents require more than just a large language model (LLM). To avoid hallucinations, retain context, and deliver smooth human-computer interaction, five foundational systems work in tandem:

The System Pipeline Breakdown

1. Inference Engine (Groq LPU): Utilizes custom hardware (Language Processing Units) running through langchain-groq to deliver high-speed token generation with minimal First-Token Latency (FTL).
2. Dynamic Tooling (SerpAPI): Integrates GoogleSerpAPIWrapper as a callable agent tool, allowing the model to bridge its knowledge cutoff and ground answers in real-time search data.
3. Execution Layer (LangChain Agents & LangGraph): Combines models, instructions, and tools via create_agent alongside a checkpointer that manages dialog state.
4. State Persistence (st.session_state): Bridges the stateless execution model of Streamlit by anchoring thread references, checkpointers, and chat transcripts across app reruns.
5. Streaming Interface (st.empty): Progressively unpacks token chunks via a context manager to provide an interactive, real-time typing effect.

2. Environment Setup & Core Dependencies

The application relies on python-dotenv for managing sensitive credentials, langchain-groq for inference, langchain-community for search tools, langgraph for memory management, and streamlit for the presentation layer.

Required Environment Variables

Create a .env file in the root directory:

GROQ_API_KEY="gsk_your_groq_api_key_here"
SERPAPI_API_KEY="your_serpapi_api_key_here"

Core Imports

import os
from dotenv import load_dotenv

# Streamlit UI
import streamlit as st

# LLM & Tools
from langchain_groq import ChatGroq
from langchain_community.utilities import GoogleSerpAPIWrapper

# Agent & Memory Architecture
from langchain.agents import create_agent
from langgraph.checkpoint.memory import MemorySaver

# Load API credentials from .env
load_dotenv()

3. Naive Agent Construction vs. Stateful Agent Construction

The Naive Stateless Agent (Baseline Architecture)

The naive pattern configures an LLM, attaches tools, registers an in-memory checkpointer, and invokes the pipeline in a synchronous, one-off execution:

# 1. Initialize Groq LLM
llm = ChatGroq(
    model_name="openai/gpt-oss-20b", # Open-source high-throughput model variant on Groq
    streaming=False
)

# 2. Configure SerpAPI Tool
search = GoogleSerpAPIWrapper()
tools = [search.run]

# 3. Instantiate LangGraph In-Memory Checkpointer
memory = MemorySaver()

# 4. Create Agent
agent = create_agent(
    model=llm,
    tools=tools,
    checkpointer=memory,
    system_prompt="You are an amazing AI agent and can search on Google as well."
)

# 5. Baseline Invocation
response = agent.invoke(
    {
        "messages": [
            {"role": "user", "content": "Who is the PM of India?"}
        ]
    },
    config={"configurable": {"thread_id": "session_user_01"}}
)

# Output extraction
print(response["messages"][-1].content)

The Streamlit Lifecycle Dilemma

Streamlit operates on an immediate-mode execution model. Every time a user types in a chat box, clicks a button, or submits an input, the Python file runs entirely from line 1 to the end:

The In-Memory Object Re-Instantiation Trap

If memory = MemorySaver() sits directly in the script execution path, its memory address shifts on every rerun:

• Rerun 1: Memory object allocated at address 0x10d41c60.
• Rerun 2 (Follow-up question): A brand-new MemorySaver allocates at address 0x10d52b90.

The old checkpointer is garbage-collected, and context is lost. Asking "Who is the PM of India?" followed by "What is his age?" yields:

"I am not sure which person you are referring to."

4. Solving State Degradation with Streamlit Session State

To retain conversational threads and prevent duplicate object initialization across page reruns, store both the Checkpointer and the Message History Array inside st.session_state.

Session State Initialization Code

# Guard pattern: instantiate the MemorySaver checkpointer exactly once
if "memory" not in st.session_state:
    st.session_state.memory = MemorySaver()

# Guard pattern: instantiate message transcript store exactly once
if "history" not in st.session_state:
    st.session_state.history = []

Persisting Chat History in the Interface

To keep messages visible across reruns, render the persistent history before processing any new prompt:

# Render historical messages from the transcript state
for message in st.session_state.history:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

5. Token-by-Token Streaming Architecture

Synchronous invocations (agent.invoke()) block execution until the LLM generates the entire answer. For longer explanations, this introduces a noticeable delay.

To match modern conversational interfaces, transition from static invocation to chunked streaming:

Core Streaming Mechanics

1. Configure the LLM for streaming: ChatGroq(..., streaming=True).
2. Swap the invocation engine from .invoke() to .stream().
3. Set the agent streaming mode: stream_mode="messages".
4. Create an empty UI container: container = st.chat_message("ai") with space = container.empty().
5. Iterate over the yielded message chunks, concatenate them, and continuously update the container via space.markdown().

# Dynamic streaming implementation
with st.chat_message("ai"):
    space = st.empty()
    accumulated_response = ""
    
    # stream returns an iterator of token/message chunks
    for chunk in agent.stream(
        {"messages": [{"role": "user", "content": user_query}]},
        config={"configurable": {"thread_id": "global_user_thread"}},
        stream_mode="messages"
    ):
        # Extract chunk content safely
        if hasattr(chunk[0], "content"):
            accumulated_response += chunk[0].content
            space.markdown(accumulated_response)

6. End-to-End Production Code Walkthrough

Below is the complete, self-contained application (app.py), integrating the model setup, search tool, stateful checkpointer, Streamlit UI, and real-time streaming:

import os
from dotenv import load_dotenv
import streamlit as st

from langchain_groq import ChatGroq
from langchain_community.utilities import GoogleSerpAPIWrapper
from langchain.agents import create_agent
from langgraph.checkpoint.memory import MemorySaver

# 1. Environment & API Configuration
load_dotenv()

# Verify that essential API keys exist
if not os.getenv("GROQ_API_KEY"):
    st.error("Missing GROQ_API_KEY in environment variables.")
    st.stop()

# 2. UI Header Configuration
# st.subheader is used over st.title to provide a balanced interface layout
st.subheader("⚡ Quick Answer: Answer at the Speed of Thought")

# 3. Session State & Memory Preservation
if "memory" not in st.session_state:
    st.session_state.memory = MemorySaver()

if "history" not in st.session_state:
    st.session_state.history = []

# 4. Agent Tooling and Model Orchestration
# Groq provides high-speed token generation
llm = ChatGroq(
    model_name="openai/gpt-oss-20b",
    streaming=True
)

# SerpAPI setup for real-time web retrieval
search_wrapper = GoogleSerpAPIWrapper()
tools = [search_wrapper.run]

# Create the LangGraph-backed conversational agent
agent = create_agent(
    model=llm,
    tools=tools,
    checkpointer=st.session_state.memory,
    system_prompt="You are a helpful, lightning-fast AI assistant equipped with real-time web search capabilities."
)

# 5. Render Historical Conversations
for message in st.session_state.history:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

# 6. Capture User Input
query = st.chat_input("Ask anything...")

if query:
    # Immediately render the user's message
    with st.chat_message("user"):
        st.markdown(query)
    
    # Record user message to persistent transcript history
    st.session_state.history.append({"role": "user", "content": query})

    # Prepare streaming response container
    with st.chat_message("ai"):
        space = st.empty()
        full_response = ""

        # Invoke the streaming execution pipeline
        response_stream = agent.stream(
            {
                "messages": [
                    {"role": "user", "content": query}
                ]
            },
            config={"configurable": {"thread_id": "unified_user_session"}},
            stream_mode="messages"
        )

        for chunk in response_stream:
            # LangGraph messages stream yields a tuple: (BaseMessageChunk, metadata)
            message_chunk = chunk[0]
            if hasattr(message_chunk, "content") and message_chunk.content:
                full_response += message_chunk.content
                space.markdown(full_response)

    # Record AI's final accumulated response to persistent transcript history
    st.session_state.history.append({"role": "ai", "content": full_response})

7. Comparative Technical Syntheses

Streaming vs. Non-Streaming Mechanics

Memory Management Trade-Offs

8. Real-World Pitfalls & Debugging Workflows

1. The Disappearing Chat History Bug

• Symptom: Asking a second question causes the first prompt and answer to vanish from the screen.
• Root Cause: Messages were rendered only inside the if query: block. On the subsequent rerun, that block evaluates to False until a new query is submitted, leaving historical messages unrendered.
• Fix: Separate history rendering from input processing. Run a top-level for message in st.session_state.history loop before reading st.chat_input.

2. Context Window Amnesia

• Symptom: The agent replies "I don't know who you are talking about" to a direct follow-up question.
• Root Cause: MemorySaver() was assigned directly to a local variable instead of st.session_state.memory. Each Streamlit execution pass created a new checkpointer with an empty state.
• Fix: Use the guard pattern:
  PYTHON

  Copy

  if "memory" not in st.session_state:
      st.session_state.memory = MemorySaver()
  

3. Tool Stream Chunk Structural Mismatch

• Symptom: TypeError: 'ChatGenerationChunk' object is not subscriptable or unexpected chunk shapes during streaming.
• Root Cause: When tools are executed (such as GoogleSerpAPIWrapper), the agent emits intermediate tool call chunks before emitting text tokens. Reading chunk[0].content without checking for the attribute will raise errors.
• Fix: Validate that chunk[0] has a content attribute and that it contains a non-empty string:
  PYTHON

  Copy

  if hasattr(chunk[0], "content") and chunk[0].content:
      full_response += chunk[0].content
  

9. Summary and Key Takeaways

• High-Throughput Inference with Groq: Using langchain-groq offloads compute to Groq LPUs, yielding the low first-token latency required for responsive web chat applications.
• Dynamic Tool Integration: Adding tools like GoogleSerpAPIWrapper allows agents to retrieve current, real-time facts beyond their training cutoff dates.
• Streamlit's Execution Model: Streamlit scripts re-execute in their entirety on every user interaction. Stateful items must be anchored within st.session_state.
• Decoupled Memory Checkpointing: Agent checkpointers (such as LangGraph's MemorySaver) will reset on every rerun unless held inside st.session_state.
• Dual-Layer Conversation Persistence: A stateful UI requires tracking state at two levels: the LangGraph checkpointer (for model context) and a message history list (for UI re-rendering).
• Chunked Dynamic Rendering: Using .stream(..., stream_mode="messages") alongside st.empty() allows real-time token streaming that matches modern conversational interfaces.