Forward Deployed Engineer Full Course

Chapter 2: How LLMs Actually Work? | Tokens, Attention & Transformers

This chapter demystifies the inner workings of Large Language Models (LLMs) such as ChatGPT, Claude, and DeepSeek. We explore the progression from simple next-word prediction to sub-word tokenization, multi-dimensional vector embeddings, the Query-Key-Value (QKV) self-attention mechanism, and multi-layered Transformer architectures. Finally, we examine how decoding strategies like sampling and temperature control model output creativity and determinism.

1. The Next-Word Prediction Game: How Context Shapes Meaning

At their core, Large Language Models operate on a deceptively simple game: Next-Word Prediction. However, predicting the next word accurately is not merely a function of looking at the immediately preceding word; it is entirely dependent on the surrounding context.

Real-Life Analogy: Context Expansion

Consider how your own mind predicts the next word as context is progressively added:

• Scenario 1 (Minimal Context):

*   *Possible Predictions:* `morning`, `evening`, `day`, `night`, `weekend`.
*   *Analysis:* The options are vast because the context is highly open-ended.

• Scenario 2 (Moderate Context):

*   *Possible Predictions:* `office`, `gym`, `college`, `school`.
*   *Analysis:* The search space of words shrinks. We know the destination must be a physical place typically visited in the morning.

• Scenario 3 (Rich Context):

*   *Possible Predictions:* `office`, `work`.
*   *Analysis:* Because a software engineer is unlikely to go to school or college, those options are eliminated. The context of the profession filters out irrelevant destinations.

• Scenario 4 (Hyper-Specific Context):

*   *Possible Predictions:* `my desk`, `workstation`.
*   *Analysis:* The introduction of "remotely" eliminates a physical corporate office. The prediction collapses onto a highly specific, highly accurate target.

This progression demonstrates that context acts as a filter that restricts the probability distribution of the vocabulary. LLMs leverage this exact mechanism to generate coherent text.

2. The Core Mental Model: LLMs as Next-Token Predictors

While we intuitively think of LLMs as conversational agents, they are fundamentally autoregressive next-token predictors.

Once the model predicts the next token, it appends that token to the original input and feeds the entire sequence back into itself to predict the subsequent token:

This feedback loop continues until the model generates a special stop token (often called <|endoftext|> or EOS) or reaches a predefined limit.

3. From Text to Numbers: The Role of the Tokenizer

Computers, neural networks, and mathematical models do not understand characters, syllables, or words. They only understand numbers. Therefore, a text input must be converted into a numerical format before an LLM can process it.

Naive Approach: Word-to-ID Mapping

A simple way to convert text to numbers is to assign a unique integer to every word in a dictionary:

Warning on Arbitrary IDs: These assigned numbers are purely arbitrary identifiers (similar to database primary keys or student roll numbers). They carry no mathematical meaning. A "Car" with ID 4 is not "four times greater" than a "Cat" with ID 1. There is no inherent relationship or scale implied by these IDs.

4. Tokens vs. Token IDs: Sub-Word Tokenization & Trade-Offs

In production systems, we do not map entire words to IDs. Instead, we use sub-word tokenization. A Token is a set of characters (which can be a whole word, a part of a word, or even punctuation/spaces), and a Token ID is its corresponding integer index in the model's vocabulary.

Why Not Tokenize Whole Words?

If we mapped every unique word to a unique token, our vocabulary size would explode. Words like automatic, automatically, automate, and automation would require four separate entries. By breaking words into sub-words (e.g., auto + matic), we can reuse common sub-components across different words, keeping the vocabulary size manageable.

Why Not Tokenize Individual Characters?

If we mapped only individual characters ($A \rightarrow 1, B \rightarrow 2, \dots, Z \rightarrow 26$), our vocabulary would be tiny (only 26 tokens for English). However, the sequence length would explode. A single paragraph would turn into thousands of tokens, requiring massive computational resources to process because the model's memory and compute scale quadratically with sequence length.

The Tokenizer Trade-Off

Tokenizers strike a balance between vocabulary size and sequence length:

Model Tokenization Differences

Different models use different tokenization algorithms and vocabularies. As models evolve, their vocabularies become richer, allowing them to compress text into fewer tokens.

• Example 1: "Elephants are huge"
  • GPT-3 (Legacy): Splits into 5 tokens: [E, le, ph, ants, are, huge]
  • GPT-4: Splits into 4 tokens: [Eli, fants, are, huge]
  • GPT-5: Compresses it further due to a larger vocabulary.
• Example 2: Complex Words ("inevitable")
  • Often split into fragments: [in, ev, itable] so that these fragments can be reused in other contexts.
• Example 3: Emojis (🔥, ❤️)
  • GPT-3 / GPT-4: Often fail to recognize emojis as single tokens, splitting them into multiple raw unicode character tokens.
  • GPT-5: Possesses a highly rich vocabulary where common emojis are mapped directly to a single token.

Heuristic Rule of Thumb: In English text, 1 token is approximately equal to 4 characters (or $0.75$ words). This is a statistical estimation used for billing and rate-limiting, not a strict mathematical rule.

5. The Importance of Sequence Order and Context (Polysemy)

Order Matters

Consider these two sentences:

1. "Dog bites man"
2. "Man bites dog"

Both sentences contain the exact same tokens and token IDs, but their meanings are completely opposite. Therefore, an LLM cannot treat text as a "bag of words." It must preserve positional information (achieved via Positional Encodings added to the token representations).

Polysemy (Contextual Meaning)

Consider the word "bank" in two different contexts:

• "I deposited money at the bank." (Financial Institution)
• "I sat on the bank of a river." (River Edge)

The tokenizer will assign the exact same Token ID (e.g., 1001) to the word "bank" in both sentences. The tokenizer itself is static and context-blind. It is the job of the downstream neural network layers—specifically Vector Embeddings and Self-Attention—to dynamically compute the correct contextual meaning of the token based on its surrounding sequence.

6. Vector Embeddings: Mapping Words to Multi-Dimensional Space

Once tokens are converted to Token IDs, they are mapped to dense, continuous vectors called Vector Embeddings.

Human-Understandable Dimensions (Mental Model)

Imagine we want to represent words across four conceptual dimensions: Royalty, Food, Living, and Technology. We can assign scores between $-1.0$ and $+1.0$ (or $0.0$ and $1.0$) for each word along these dimensions:

By looking at these vectors, a mathematical model can easily determine that King and Queen are close to each other in vector space, while Banana and Laptop reside in completely different neighborhoods.

Latent Dimensions in LLMs

In production LLMs, we do not manually label dimensions like "Royalty" or "Food." Instead, the model learns latent dimensions (often $1024$, $4096$, or $12288$ dimensions) during training on raw internet text. These dimensions capture highly abstract linguistic, syntactic, and semantic patterns that are often impossible for humans to label or describe.

7. The Self-Attention Mechanism: Resolving Contextual Relationships

The breakthrough of the Transformer architecture (from the landmark paper "Attention Is All You Need") is the Self-Attention mechanism. Self-attention allows tokens to dynamically update their vector representations by "looking at" and extracting information from other relevant tokens in the sequence.

The Pronoun Resolution Problem

Consider the sentence:

How does the model resolve which "his" refers to whom?

• For the first "his", the self-attention mechanism calculates a high attention score between "his" and "Rohit".
• For the second "his", the self-attention mechanism calculates a high attention score between "his" and "Aditya", because it is followed by "laptop was broken", and Aditya was the recipient in need.

8. The QKV (Query, Key, Value) Model Under the Hood

To calculate self-attention, the Transformer uses a database-inspired retrieval mechanism called Query-Key-Value (QKV).

Google Search Analogy

• Query ($Q$): The search term you type into Google (e.g., "best Java course"). This represents "What information am I looking for?"
• Keys ($K$): The titles, metadata, and index terms of all web pages on the internet. This represents "What kind of information do I offer?"
• Values ($V$): The actual content of those web pages. This represents "Here is the rich information I contain."

Google compares your Query against all indexed Keys to find the best matches. Once the best matches are found, it returns the corresponding Values.

QKV in Self-Attention

For every token in the input sequence, the model projects its embedding into three distinct vectors using trained weight matrices ($W_Q, W_K, W_V$):

1. Query Vector ($Q$): What context does this token need from other tokens?
2. Key Vector ($K$): What kind of context can this token offer to other tokens?
3. Value Vector ($V$): What is the actual semantic content of this token?

Step-by-Step Attention Calculation

Let's trace how the token "on" is processed in the sentence: "The cat sat on the mat".

Step 1: Query-Key Comparison (Dot Product)

We take the Query vector of "on" ($Q_{on}$) and calculate its similarity (dot product) with the Key vectors ($K$) of every token in the sentence:

Step 2: Weight Normalization (Softmax)

These raw similarity scores are scaled and passed through a Softmax function to convert them into a probability distribution (weights that sum to $1.0$ or $100\%$):

• $W_{on \rightarrow mat} = 0.70$ (High relevance: "on" is physically related to "mat")
• $W_{on \rightarrow cat} = 0.20$ (Medium relevance: the cat is the entity performing the action)
• $W_{on \rightarrow the} = 0.05$ (Low relevance)

Step 3: Weighted Sum of Values

We multiply these normalized weights by the Value vectors ($V$) of each corresponding token to produce the final, context-aware output vector ($Z_{on}$):

This output vector $Z_{on}$ is no longer a static representation of the word "on"; it is now heavily infused with the concepts of "cat" and "mat".

Why Three Separate Vectors ($Q, K, V$)?

Just as a single human can be described differently depending on the context (e.g., professional skills, physical location, moral values), a single token plays different roles during attention calculation.

Using three separate vectors allows the model to decouple:

• The search for context ($Q$)
• The advertising of its own context ($K$)
• The actual content payload ($V$)

9. The Transformer Architecture: Multi-Layered Processing

A Transformer does not perform self-attention just once. It consists of multiple stacked Transformer Layers (often called blocks).

Real-World Analogy: Photo Editing Pipeline

Think of the layers of a Transformer like a professional photo editing workflow:

1. Stage 1 (Raw Import): Adjust basic brightness and contrast (Layer 1: simple syntax).
2. Stage 2 (Color Grading): Apply specific color filters and balance tones (Layer 2: local semantics).
3. Stage 3 (Compositing): Add complex background elements and blend layers (Deep Layers: long-range context and abstract reasoning).

Accumulating Context at the Final Token

When we feed "The capital of India is" into a multi-layered Transformer, the self-attention mechanism runs at every layer. By the time the vectors reach the final layer, the embedding for the last token ("is") has pulled in information from "capital", "India", and "The".

The final vector of the last token ($H_{final}$) acts as a bottleneck that contains the distilled semantic meaning of the entire input sequence.

10. The Generation Phase: Logits, Softmax, and Output Prediction

Once we have the final context-rich vector of the last token ($H_{final}$), we need to map it back to our vocabulary to select the next word.

1. Logits Generation

We compare $H_{final}$ against the static embeddings of every single token in our vocabulary (e.g., $100,000$ words). This comparison yields a raw, unnormalized score for each word, known as Logits:

2. Softmax Normalization

To turn these raw logits into clean percentages, we apply the Softmax function:

This converts the logits into a probability distribution:

11. Controlling Creativity: Greedy Decoding vs. Sampling

Once we have the probability distribution, how do we select the next token?

1. Greedy Decoding (Argmax)

The model always selects the token with the absolute highest probability.

• Behavior: Highly deterministic.
• Problem: If you ask the model "Hi, how are you?", it will output the exact same response every single time. It can also get stuck in repetitive loops.

2. Sampling (The Lottery Ticket Analogy)

Instead of always picking the top choice, we treat the probabilities as lottery tickets.

If we draw a random number between $1$ and $100$:

• If we draw $34$, we output "I am fine".
• If we draw $72$, we output "I am doing well".
• If we draw $93$, we output "Great".

This introduces creativity and variety. Even if "I am fine" is the most likely answer, the model has a $40\%$ chance of choosing "I am doing well", explaining why ChatGPT generates different responses to the same prompt.

12. The Temperature Parameter: Tuning Determinism and Randomness

Temperature ($T$) is a hyperparameter applied to the raw logits before they are passed to the Softmax function. It scales the logits to control how flat or sharp the probability distribution becomes.

Tabular Synthesis of Temperature Effects

13. Summary and Key Takeaways

• Next-Token Prediction: LLMs do not think or plan globally; they iteratively predict the next token based on the mathematical probability distribution of preceding context.
• Sub-word Tokenization: Modern models process text as sub-word tokens rather than whole words or individual characters, balancing vocabulary size against sequence length.
• Arbitrary Token IDs: Tokenizers map text to arbitrary integer IDs that carry no mathematical meaning until they are converted into dense vector embeddings.
• Vector Embeddings: Embeddings map tokens into high-dimensional space (often thousands of dimensions), allowing the model to represent semantic concepts mathematically.
• Self-Attention (QKV): The Query-Key-Value mechanism allows tokens to dynamically update their vector representations by calculating relevance weights with all other tokens in the sequence.
• Transformer Layers: Deep stacks of Transformer layers allow the model to build highly complex, abstract, and contextual relationships from raw input text.
• Logits and Softmax: The final layer compares the context vector of the last token against the entire vocabulary to produce raw scores (logits), which Softmax converts into percentages.
• Sampling & Temperature: Sampling introduces creativity by treating probabilities as lottery tickets, while Temperature scales logits to make the model either highly deterministic ($T < 1$) or highly creative ($T > 1$).