How AI Understands Images? Build a Multimodal AI Application

Most early Large Language Model (LLM) applications operated strictly on textual input and output. However, real-world information exists across multiple channels—including invoice images, code error screenshots, diagrams, photos, and scanned documents. Modern multimodal AI models extend basic text capabilities by natively processing visual inputs and generating contextual textual answers.

This guide explores the underlying mechanics of Vision-Language Models (VLMs). We cover how images are tokenized into spatial patches, transformed into visual embeddings via Vision Encoders, aligned into language representation spaces using Projection Adapters, and integrated into a full-stack Spring Boot multimodal chat system.

1. Evolution of AI Inputs: Single-Modality vs. Multimodal vs. Multi-Model

To understand multimodal applications, we must first establish clear distinctions between different system architectures:

• Single-Modality Systems: Traditional LLMs that process text inputs and return text outputs (e.g., standard text completion or early chat interfaces).
• Multimodal Systems: Applications powered by models that natively process multiple input types (such as text, images, audio, or video) to produce responses (typically text).
• Multi-Model Architectures: Backend systems that route requests across multiple distinct AI models (e.g., combining OpenAI, Anthropic, and Google Gemini within a single server infrastructure).

Modality vs. Architecture Breakdown

2. Real-World Analogy: The Expert Blind Reader with a Specialized Scanner

Real-Life Analogy

Think of a pure Text LLM as a brilliant scholar who is completely blind and can only read and communicate via Braille text vectors.

• OCR (Optical Character Recognition) acts like a helper who reads out individual letters written on a page letter-by-letter without understanding what the full context or code means.
• Vision Multimodality acts like a specialized tactile scanner. It cuts an image into uniform square tiles (patches), analyzes the visual features within each tile (textures, shapes, spatial locations), and converts those visual features into the exact vector language that the blind scholar already uses to think.

The scholar never "sees" the picture with human eyes; instead, they receive a sequence of spatial mathematical vectors that describe the contents of the picture in high-dimensional context.

3. Traditional OCR vs. Vision LLMs

A common misconception is that multimodal models simply run an Optical Character Recognition (OCR) script behind the scenes to extract text from an image. Vision LLMs perform a fundamentally richer task.

Feature Comparison

4. The Core Technical Problem: How LLMs Process Input

Text Tokenization Pipeline Recap

Before exploring image processing, recall how standard text is supplied to an LLM:

1. Raw Input: "Hi, my name is Aditya"
2. Tokenization: Split into token chunks: ["Hi", ",", "my", "name", "is", "Aditya"]
3. Token IDs: Mapped to numerical dictionary indices: [23, 41, 56, 102, 88, 512]
4. Embeddings: Converted into dense vectors ($v_1, v_2, v_3, \dots, v_n$) in $D$-dimensional space.
5. Self-Attention: Passed through Query ($Q$), Key ($K$), and Value ($V$) transformations to establish relationships between tokens.

The Raw Pixel Trap

An image is a matrix of pixels. A high-resolution photo ($1920 \times 1080$ pixels) contains $2,073,600$ individual pixels. Each pixel consists of 3 primary color channels: Red, Green, and Blue ($RGB$), where each intensity ranges from $0$ to $255$:

Passing over 6 million raw numerical values directly into an LLM's transformer network causes massive performance issues. Transformer self-attention complexity scales quadratically with input token length ($O(N^2)$). Dumping millions of raw pixel numbers directly into the attention mechanism is computationally infeasible.

5. Mechanics of Visual Tokenization: Patches, Encoders, and Adapters

To handle visual inputs efficiently, modern Vision-Language Models convert continuous image matrices into structured Visual Embeddings.

Step 1: Image Patching ("Visual Tokens")

Instead of reading pixel-by-pixel, the system divides the image matrix into a uniform grid of larger regions called patches.

• A standard patch size is $16 \times 16$ pixels.
• Each patch acts as the visual equivalent of a word token in natural language processing.

Step 2: Spatial Position Encoding

Just as word order alters sentence meaning, the spatial position of a visual patch determines image layout.

• A patch depicting a cat's ear at the top-left of an image represents a completely different layout than the same patch at the bottom-right.
• Spatial positional vectors are added to each patch representation to retain geographic information across the grid.

Step 3: Vision Encoder

The patched regions pass through a specialized neural network—typically a Vision Transformer (ViT) or a Convolutional Neural Network (CNN) backbone.

• The Vision Encoder inspects pixel patterns across patches (textures, edges, shapes, colors).
• It computes attention between patches (e.g., relating a patch containing fur to an adjacent patch containing an eye or an ear).
• It outputs a set of dense Visual Embeddings.

Step 4: Projection / Adapter Layer

Visual embeddings produced by a Vision Encoder live in a visual representation vector space. However, the core LLM operates strictly in a text representation vector space.

To bridge this gap, a Projection Layer (also known as a Linear Adapter) mathematically maps visual embeddings directly into the LLM's text embedding dimension:

This aligns visual features with text semantics, allowing the LLM to process visual tokens as if they were standard text tokens.

6. End-to-End Multimodal Data Flow

The diagram below details the dual-path input processing workflow used in multimodal LLM architectures:

7. Building a Multimodal Vision Application in Java Spring Boot

Let's build a production-grade multimodal chat service in Java using Spring AI. This application accepts both user text messages and optional uploaded image files (MultipartFile), maintains conversation history, and communicates with a multimodal endpoint (such as gpt-4o-mini).

Configuration (application.properties)

spring.application.name=vision-chat
spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o-mini
spring.ai.openai.chat.options.temperature=0.7

# File Upload Configuration
spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=10MB

Controller Layer: ChatController.java

The controller accepts requests containing a mandatory textual message and an optional image file payload.

package com.coderarmy.visionchat.controller;

import com.coderarmy.visionchat.service.ChatService;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;

@RestController
@RequestMapping("/api")
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @PostMapping("/chat")
    public String chat(
            @RequestParam("message") String message,
            @RequestParam(value = "image", required = false) MultipartFile image) {
        return chatService.chat(message, image);
    }

    @DeleteMapping("/chat")
    public String clearHistory() {
        chatService.clearHistory();
        return "Conversation history cleared successfully.";
    }
}

Service Layer: ChatService.java

The service layer converts uploaded image files into Spring AI Media representations, formats user messages, manages conversation context history, and interacts with the LLM API via ChatClient.

package com.coderarmy.visionchat.service;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.model.Media;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.stereotype.Service;
import org.springframework.util.MimeTypeUtils;
import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

@Service
public class ChatService {

    private final ChatClient chatClient;
    private final List<Message> history = new ArrayList<>();

    public ChatService(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    public String chat(String messageText, MultipartFile imageFile) {
        UserMessage userMessage;

        // Check if an image payload is attached
        if (imageFile != null && !imageFile.isEmpty()) {
            try {
                // Convert MultipartFile to byte array resource
                byte[] imageBytes = imageFile.getBytes();
                ByteArrayResource imageResource = new ByteArrayResource(imageBytes);

                // Determine image MIME type (e.g., image/png, image/jpeg)
                String contentType = imageFile.getContentType();
                var mimeType = (contentType != null) ? 
                        MimeTypeUtils.parseMimeType(contentType) : MimeTypeUtils.IMAGE_JPEG;

                // Create Spring AI Media object
                Media media = new Media(mimeType, imageResource);

                // Build UserMessage with both text content and media content
                userMessage = UserMessage.builder()
                        .text(messageText)
                        .media(media)
                        .build();

            } catch (IOException e) {
                throw new RuntimeException("Failed to process uploaded image file", e);
            }
        } else {
            // Standard text-only message construction
            userMessage = new UserMessage(messageText);
        }

        // Store User Message in Conversation History
        history.add(userMessage);

        String systemPrompt = "You are a helpful AI assistant. Answer questions clearly and simply based on text and image inputs provided.";

        // Invoke Multimodal LLM Endpoint
        String response = chatClient.prompt()
                .system(systemPrompt)
                .messages(history)
                .call()
                .content();

        // Store Assistant Response in Conversation History
        history.add(new AssistantMessage(response));

        return response;
    }

    public void clearHistory() {
        this.history.clear();
    }
}

8. Practical Edge Cases & Production Considerations

1. Payload Sizes and Resolution Token Scaling

Large images consume significant bandwidth and token budgets. Modern vision models scale token consumption based on image size and detail level (e.g., standard low-res detail vs. high-res tiled detail).

• Optimization: Compress raw client-side inputs (e.g., converting 4K images down to $1080p$) before transmitting them to the backend server.

2. Multi-Page Documents vs. Direct Vision

While small image files can be sent directly to Vision APIs, processing multi-page PDF documents via raw visual streams can quickly exceed token context windows and API limits.

• Best Practice: For multi-page text documents, use Retrieval-Augmented Generation (RAG). Parse pages using text extraction tools, store text vectors in a Vector Database, and route only specific document image chunks to the VLM when structural layout parsing is strictly required.

3. Unified Embedding Space Expansion

The visual adapter pattern used for vision extends directly to other modalities:

• Audio: Split into short spectrogram time-frequency frames $\rightarrow$ Audio Encoder $\rightarrow$ Audio Projection Adapter $\rightarrow$ Text Embeddings.
• Video: Sample continuous frame matrices + temporal difference vectors $\rightarrow$ Video Encoder $\rightarrow$ Adapter $\rightarrow$ Text Embeddings.

9. Summary and Key Takeaways

• Multimodal AI Definitions: Multimodal models natively accept distinct input types (text, images, audio) to generate answers, whereas multi-model architectures route requests across different model providers.
• Vision LLMs vs. OCR: OCR strictly extracts raw character strings from images without semantic comprehension. Vision LLMs analyze spatial layout and visual context to reason, diagnose code errors, and answer complex queries.
• The Inefficiency of Raw Pixels: Passing raw RGB pixel values directly into transformer attention layers causes severe computational bottlenecks due to quadratic scaling ($O(N^2)$).
• Visual Tokenization (Patches): Vision systems divide images into uniform grids (e.g., $16 \times 16$ pixel patches) that act as visual word equivalents.
• Spatial Awareness: Positional encodings are appended to image patches to preserve spatial relationships (such as layout, coordinate order, and UI element positioning).
• Vision Encoders & Projection Adapters: Vision Encoders (like ViT) transform image patches into visual embeddings, while Linear Adapters project these vectors into the LLM's unified text embedding space.
• Spring Boot Multimodal Integration: Spring AI simplifies visual chat development by wrapping raw image bytes into Media objects passed directly along with UserMessage instances to a vision-capable ChatClient.