Integrating LLMs with Applications: From Raw Models to GenAI Software

This chapter explores how to transition from manual, playground-based interactions with Large Language Models (LLMs) to programmatic integration within software applications. We will demystify the architectural differences between a raw LLM and a complete Generative AI (GenAI) application, analyze how models leverage external tools to overcome inherent reasoning limitations, and walk through a step-by-step implementation of a production-grade support ticket summarizer using Postman and Java Spring Boot.

1. The Core Misconception: Raw LLM vs. GenAI Application

A common misconception among developers and end-users is that applications like ChatGPT, Claude, or Gemini are simply "raw" LLM models. In reality, when you interact with a web interface, you are not communicating directly with a raw transformer model.

A raw LLM is fundamentally a next-token prediction engine. It takes a sequence of tokens as input and mathematically predicts the most probable next token. It does not natively understand web browsing, user authentication, security guardrails, or state management.

As illustrated above, the raw LLM is merely one component of a much larger, highly orchestrated software ecosystem.

2. The Engine and the Car: A Mental Model for GenAI Architecture

To understand how to build software around artificial intelligence, it is helpful to use a real-world analogy.

Real-Life Analogy: The Engine vs. The Car

• The Raw LLM is the Engine: The engine provides the core capability—the raw horsepower and rotational force. However, you cannot sit on a bare engine and drive it down the highway. An engine on its own cannot steer, brake, or protect you from the elements.
• The GenAI Application is the Car: To make the engine useful, engineers build a car around it. They add a chassis (infrastructure), wheels (integrations), a steering wheel (user interface), brakes and seatbelts (guardrails and safety filters), and a dashboard (monitoring and analytics).

Similarly, in software engineering:

• Raw LLM (The Engine): Predicts text based on patterns learned during training.
• GenAI Application (The Car): Wraps the model with tokenizers, database connectors, external APIs, memory management, and security layers to deliver a safe, deterministic, and valuable user experience.

3. Limitations of Raw LLMs (Weather, Math, and Code Execution)

Raw LLMs have strict operational boundaries. Because they are static neural networks frozen after their training phase, they suffer from three primary limitations:

1. Information Cutoff Dates: A raw model cannot answer questions about real-time events. For example, if a model's training data cutoff is June 2024, it cannot natively know the weather in Delhi today.
2. Lack of Computational Capabilities: LLMs do not calculate math; they predict the most likely sequence of digits. When asked to multiply two large, unique numbers (e.g., $87,458,733 \times 56$), a raw LLM will often output a plausible-looking but mathematically incorrect number.
3. Inability to Execute Code: If asked to count the exact number of asterisks (*) in a highly repetitive string, a raw LLM will guess based on visual patterns (often failing significantly, such as predicting 16 instead of 32) because it cannot run a loop to count them.

Note: To test this yourself, you can bypass the consumer ChatGPT interface and use the raw developer playground at platform.openai.com. When you ask a raw model without tools about current weather or complex arithmetic, it will openly admit its inability to assist or provide an inaccurate, hallucinated guess.

4. How GenAI Applications Leverage External Tools

To overcome these limitations, modern GenAI applications equip the LLM with external tools. The LLM acts as the "brain" or "reasoning engine," deciding when and how to use these tools, while deterministic software programs execute the actual tasks.

Common Tools Integrated with LLMs

• Web Search / Browsing APIs: Allows the model to query the live internet for real-time information (e.g., current weather, stock prices).
• Deterministic Calculators: Intercepts mathematical queries and processes them via standard arithmetic libraries rather than text prediction.
• Code Compilers / Interpreters: Executes model-generated code (such as Python or JavaScript) in a secure sandbox to solve complex data processing, counting, or algorithmic problems.

5. Programmatic Access: API Keys, Authentication, and Pricing Dynamics

When integrating an LLM into your own application, you must bypass the consumer web UI and communicate directly with the provider's API servers. This requires an API Key for authentication and billing.

Why Do LLM APIs Cost Money?

Training and running inference on LLMs requires massive computational power, specifically high-end GPUs (Graphics Processing Units). Providers charge developers based on Tokens (chunks of characters, where 1 token is roughly 4 characters or 0.75 words).

OpenAI Model Pricing Comparison (Conceptual)

Note on Free Alternatives: If you want to experiment with free API keys, services like Open Router or Groq provide free access to open-source models (such as Llama or Mistral). However, for production-grade reliability and advanced reasoning, a paid tier (such as adding a minimum $5 credit to an OpenAI developer account) is highly recommended.

6. Step-by-Step API Integration via Postman

Before writing code, it is best practice to test your API connection using a client like Postman. This isolates network, authentication, and payload issues from your application logic.

Step 1: Configure the Request

• HTTP Method: POST
• URL: https://api.openai.com/v1/chat/completions

Step 2: Set Authentication Headers

LLM APIs use Bearer Token authentication. In Postman:

1. Navigate to the Authorization tab.
2. Select Bearer Token from the Type dropdown.
3. Paste your secret API Key into the Token field.

Headers:
Authorization: Bearer sk-proj-xxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json

Step 3: Construct the JSON Payload

In the Body tab, select raw and set the format to JSON. Paste the following payload:

{
  "model": "gpt-5.6-luna",
  "messages": [
    {
      "role": "user",
      "content": "Explain Docker in two lines."
    }
  ]
}

Step 4: Analyze the Response

Upon hitting Send, the API returns a JSON response containing metadata, token usage statistics, and the generated text:

{
  "id": "chatcmpl-A1B2C3...",
  "object": "chat.completion",
  "created": 1719234567,
  "model": "gpt-5.6-luna",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Docker packages applications and their dependencies into lightweight, portable containers. This ensures software runs consistently across any development or production environment."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 29,
    "total_tokens": 42
  }
}

Debugging Tip: Handling 404 Errors

If you receive a 404 Not Found error, double-check your endpoint URL. A common mistake is using incorrect paths (e.g., /v1/responses instead of /v1/chat/completions). Ensure your URL matches the provider's official API documentation exactly.

7. The Statelessness Trap: Why the API "Forgets" Your Name

A critical architectural reality of working with raw LLM APIs is that HTTP requests are completely stateless.

If you send a request telling the model your name, and then send a separate, subsequent request asking "What is my name?", the model will respond with "I do not know your name."

Why does ChatGPT web UI remember, but the API does not?

The consumer ChatGPT application maintains a database of your chat history. Every time you send a new message, the application's back-end server retrieves your past conversation history, appends your new message to it, and sends the entire conversation history back to the LLM API.

When building your own application, you are responsible for managing this state and sending the historical context with each API call.

8. Building a Production-Grade Support Ticket Summarizer in Spring Boot

Let us build a practical, real-world application. Imagine you are a Forward Deployed Engineer at a food delivery company (like Zomato). Customer support agents are overwhelmed by long, angry customer messages. We will build an API endpoint /api/summarize that takes a long support ticket and uses gpt-5.6-luna to summarize it into two concise lines.

Step 1: Add Dependencies (pom.xml)

We will use the official Spring AI starter dependency to simplify communication with OpenAI.

<dependencies>
    <!-- Spring Boot Web Starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <!-- Spring AI OpenAI Starter -->
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-openai-starter</artifactId>
        <version>1.0.0-M1</version>
    </dependency>
</dependencies>

Step 2: Configure Application Properties (application.properties)

Store your API credentials and model preferences securely in your configuration file.

# Server Configuration
server.port=8080

# Spring AI OpenAI Configurations
spring.ai.openai.api-key=sk-proj-yourActualSecretAPIKeyHere
spring.ai.openai.chat.options.model=gpt-5.6-luna

Step 3: Create the Service Layer (SummarizeService.java)

This class handles the business logic, constructing the prompt and communicating with the LLM via Spring AI's fluent ChatClient.

package com.coderarmy.fde.service;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;

@Service
public class SummarizeService {

    private final ChatClient chatClient;

    // Constructor injection of ChatClient.Builder
    public SummarizeService(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    /**
     * Summarizes a raw customer support ticket into exactly two lines.
     * @param ticket The raw text of the customer complaint.
     * @return A concise two-line summary.
     */
    public String summarizeTicket(String ticket) {
        String systemInstruction = "Summarize this support ticket in exactly two lines:\n\n";
        String fullPrompt = systemInstruction + ticket;

        return this.chatClient.prompt()
                .user(fullPrompt)
                .call()
                .content(); // Extracts only the text content, ignoring metadata
    }
}

Step 4: Create the Controller Layer (SummarizeController.java)

Expose a REST endpoint to accept incoming customer support tickets.

package com.coderarmy.fde.controller;

import com.coderarmy.fde.service.SummarizeService;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api")
public class SummarizeController {

    private final SummarizeService summarizeService;

    // Constructor Injection
    public SummarizeController(SummarizeService summarizeService) {
        this.summarizeService = summarizeService;
    }

    @PostMapping("/summarize")
    public String summarize(@RequestBody String ticket) {
        return summarizeService.summarizeTicket(ticket);
    }
}

Step 5: Testing the Application

Start your Spring Boot application and send a POST request to http://localhost:8080/api/summarize with a long customer complaint in the request body:

Raw Input Payload:

"I am extremely disappointed and frustrated with my recent order from your restaurant. I ordered a large chicken biryani, butter chicken, garlic naan, and a coke. I waited for over 50 minutes, and when the delivery driver finally arrived, he handed me a bag containing veg fried rice and spring rolls! This is completely unacceptable. I tried calling the restaurant multiple times but no one picked up. I had guests waiting at my house and this ruined our dinner. I want an immediate refund and someone from customer support to contact me right away!"

Response Output:

"Customer received veg fried rice instead of chicken biryani and butter chicken after a 50-minute wait. They are requesting an immediate refund and urgent contact from support after failing to reach the restaurant."

9. Real-World Pitfalls: Prompt Hijacking and Frivolous Queries

While our support ticket summarizer works perfectly for legitimate complaints, it suffers from a major security vulnerability: Prompt Hijacking (or Prompt Injection).

Because we naively concatenated the system instructions with raw user input (systemInstruction + ticket), a malicious or playful user can bypass the application's intended purpose.

The "What is 2 + 2?" Vulnerability

If a user submits the query "What is 2 + 2?" or "Write a Python script to scrape a website" instead of a support ticket, our code executes the following prompt:

Summarize this support ticket in exactly two lines:

What is 2 + 2?

Because the LLM is a reasoning engine, it tries to satisfy the user's intent. Instead of rejecting the input as an invalid support ticket, it might respond with:

The customer is asking for the mathematical sum of 2 and 2. The answer to their query is 4.

Why is this a major issue in production?

1. Financial Drain: Users can abuse your API endpoint to perform free, general-purpose LLM tasks (like writing code or answering homework questions), running up your API bill.
2. Brand Damage: A food delivery app's support bot answering questions about software engineering or political topics looks highly unprofessional.
3. Security Risks: Attackers can inject instructions that force the model to output malicious payloads, bypass safety filters, or leak system prompts.

To build robust GenAI applications, we must separate developer instructions from user inputs using System Prompts, User Prompts, and Guardrails—concepts we will explore in depth in the upcoming chapters.

10. Summary and Key Takeaways

• GenAI Apps are Ecosystems: An LLM is just a next-token prediction engine (the engine). A GenAI application wraps that engine with user interfaces, security guardrails, tokenizers, and database integrations (the car).
• Raw LLMs are Limited: Raw models cannot natively perform accurate math, execute code, or access real-time data without being integrated with external tools like compilers, calculators, and web search APIs.
• APIs are Stateless: Unlike consumer web interfaces, raw LLM APIs do not remember past interactions. Developers must manually manage and send conversation history with each request to maintain context.
• Cost Optimization is Key: Always choose the right model for the job. For standard text processing and summarization, lightweight models like gpt-5.6-luna offer incredible speed and cost savings ($0.20 per 1M input tokens) compared to premium models.
• Prompt Injection is a Real Threat: Naive string concatenation of system instructions and user inputs allows users to hijack the model's behavior, leading to financial waste and security vulnerabilities. Separate system and user roles to mitigate this risk.