How Large Language Models Work: GPT Architecture, Tokenization, and Context Windows

Large Language Models (LLMs) represent a paradigm shift in artificial intelligence, transitioning machine learning from task-specific discriminative models to general-purpose generative engines. At their core, modern LLMs leverage the Transformer architecture to process, contextualize, and generate human language through statistical next-token prediction over high-dimensional vector spaces. Understanding their internal mechanics requires unpacking the interplay between the Generative Pre-trained Transformer (GPT) framework, subword tokenization, attention mechanisms, and context window constraints.

1. Understanding Large Language Models (LLMs)

To understand modern GenAI, we must decompose the foundational term Large Language Model (LLM) into its three constituent elements:

• Large: Refers to two dimensions—the scale of the training dataset (billions to trillions of tokens harvested from public web crawls, technical documentation, digitized books, and code repositories) and the parameter scale of the neural networks (ranging from billions to hundreds of billions of trainable weights).
• Language: Specifies the operational domain. The model processes syntax, semantics, idiom, and logical structure inherent in human communications, domain-specific vernaculars, and formal programming languages.
• Model: Represents the underlying mathematical artifact—a high-dimensional non-linear function composed of multi-layer neural networks optimized to map inputs to predictive outputs.

Raw Multimodal / Text Corpus (Web, Books, Code)
   |
   | massive-scale distributed pre-training
   v
Billions of Weight Parameters (Mathematical Approximation)
   |
   | inference / prompt input
   v
Generated Natural Language Response

At an engineering level, an LLM does not "think" or query a conventional relational database. Instead, it approximates a joint probability distribution over sequential data, retrieving contextual representations embedded inside its trained parameters during the pre-training phase.

2. Deconstructing GPT: Generative, Pre-trained, Transformer

The term GPT serves as the blueprint for modern state-of-the-art auto-regressive language models. Each term denotes a specific stage of the machine learning lifecycle:

Generative

Unlike discriminative models that assign labels to input data (e.g., classifying an email as Spam vs. Ham), generative models synthesize new data instances. Given an input sequence, the model evaluates the statistical probability of all known words in its vocabulary to produce an appropriate continuation.

Pre-trained

Before an LLM can perform downstream tasks (summarization, zero-shot question answering, or code generation), it undergoes self-supervised pre-training across vast corpora without explicit human annotations.

• The training objective is simple: given a truncated sequence of words, predict the immediate next token.
• By optimizing this objective across trillions of tokens, the network implicitly internalizes factual data, syntactic rules, reasoning patterns, and software syntax.

Transformer

Introduced by Google researchers in the seminal 2017 paper "Attention Is All You Need", the Transformer replaced recurrent neural networks (RNNs) and Long Short-Term Memory (LSTM) networks as the default backbone for natural language processing. Unlike recurrent networks, which process tokens sequentially and suffer from vanishing gradient bottlenecks over long sequences, Transformers process all input tokens concurrently using attention mechanisms.

Note: Models such as Google Gemini, Anthropic Claude, and Meta Llama differ in licensing, tokenizers, and post-training alignments, but fundamentally adhere to the same underlying Generative Pre-trained Transformer paradigm.

3. The Transformer Architecture and Attention Mechanism

The primary innovation powering Transformers is the Self-Attention Mechanism. Attention allows the model to compute dynamic, context-aware relationships between every pair of words within an input sequence, regardless of their distance from one another.

Disambiguating Polysemy via Attention

Natural language features polysemy—words with identical spellings but radically divergent meanings based on contextual framing. Traditional static embeddings (like early Word2Vec models) struggled to distinguish these contextual shifts.

Consider the sentence:

"The bank was too steep to climb."

The word bank carries two primary semantic interpretations:

1. A financial institution holding monetary capital.
2. The sloping land bordering a body of water (riverbank).

The Attention Mechanism evaluates contextual relationships across the entire sentence simultaneously:

1. When parsing the token bank, the model assigns a high mathematical attention weight between bank and the tokens steep and climb.
2. The mathematical dot product between the query vector of bank and the key vectors of steep and climb produces a strong relevance score.
3. This shifts the internal vector representation of bank toward its topographical definition, entirely suppressing its financial definition before selecting subsequent tokens.

4. Autoregressive Next-Token Prediction Engine

At inference time, an LLM behaves as an autoregressive sequence generator. It outputs text one discrete token at a time, continually appending its own outputs back into its input sequence for the subsequent inference cycle.

The Autoregressive Loop

Given the prompt: "We finally sat"

Step 1:
Input:  "We finally sat"
Prediction: "down" (Highest probability candidate)

Step 2:
Input:  "We finally sat down"
Prediction: "on"

Step 3:
Input:  "We finally sat down on"
Prediction: "the"

Step 4:
Input:  "We finally sat down on the"
Possible Candidates: ["bench", "chair", "floor", "beach"]
Selected: "bench" (Probability: P("bench" | Context) is maximal)

Final Result:
"We finally sat down on the bench."

Mathematical Probability Formulation

For a vocabulary $V$, the model computes a raw score (logit) $z_i$ for each candidate token $w_i \in V$. The conditional probability $P(w_t \mid w_1, w_2, \dots, w_{t-1})$ is calculated via the Softmax function:

Where $T$ represents the Temperature hyperparameter controlling randomness:

• Low $T$ ($T \to 0$): Forces greedy deterministic selection, prioritizing only the top-probability token.
• High $T$ ($T \ge 1.0$): Flattens the probability curve, introducing diversity and creative variation across tokens.

The autoregressive loop terminates once the model emits a designated End-of-Sequence (<|endoftext|> or [EOS]) token, signaling that no further tokens are required.

5. Tokens, Token IDs, and the Tokenization Pipeline

Digital computers run on binary architecture and digital arithmetic; neural networks compute operations over tensors containing floating-point numbers. They cannot directly process raw alphanumeric strings.

To bridge human-readable text and neural computation, models employ a three-tier pipeline: Raw Text $\to$ Tokens $\to$ Token IDs.

Raw Text: "Who is PM of India?"
   |
   | Step 1: Lexical segmentation (Tokenizer splitting)
   v
Tokens: ['Who', ' is', ' PM', ' of', ' India', '?']
   |
   | Step 2: Vocabulary lookup (Static dictionary lookup)
   v
Token IDs: [15463, 374, 10243, 315, 3797, 30]
   |
   | Step 3: Numerical processing
   v
Transformer Embedding Layer (Vectors)

Key Terminology

• Token: The atomic linguistic unit processed by the model. A token does not map strictly 1:1 to an entire English word. Instead, modern tokenizers use subword algorithms (e.g., Byte-Pair Encoding or BPE) where common words become single tokens, while rare words, code syntax, or compound terms are broken into smaller morphological fragments or whitespace-prefixed units.
• Token ID: A unique integer assigned to every specific token within the model's static vocabulary dictionary. If an LLM has a vocabulary size of $\vert V\vert = 100,000$, its token IDs span from 0 to 99,999.
• Tokenizer: A deterministic compiler that manages the conversion back and forth between textual strings and integer arrays using two core functions: encode() (text to IDs) and decode() (IDs to text).

6. Multi-Model Tokenization Nuances & Visualizers

Because tokenizers are trained separately prior to model pre-training, vocabulary mappings are strictly model-dependent. A single prompt yields differing token counts and entirely distinct Token IDs when passed to different model families or versions.

Comparing Tokenization Behaviors

Given the input query: "Who is PM of India?"

The subword token breakdown splits whitespace and punctuation into distinct elements:

• Token 1: Who
• Token 2: is (includes leading whitespace)
• Token 3: P
• Token 4: M
• Token 5: of
• Token 6: India
• Token 7: ?

Note: Passing Token IDs generated by an older tokenizer (e.g., p50k_base) into a model expecting a newer vocabulary (e.g., cl100k_base) will produce catastrophic semantic corruption, as the integers reference completely different words or characters.

7. Hands-on Python Lab: Building an Encoder/Decoder with Tiktoken

OpenAI maintains tiktoken, an optimized, open-source Byte-Pair Encoding tokenizer implemented in Rust with Python bindings, designed for sub-millisecond execution speeds.

Step 1: Environment Setup

To install the official library in a Python or Google Colab environment:

pip install tiktoken

Step 2: Implementation of Encoding and Decoding Pipelines

The following script demonstrates:

1. Instantiating an encoding scheme mapped to a target LLM.
2. Converting a natural language string into a vector of Token IDs (encode).
3. Reconstructing human text from raw integer IDs (decode).

import tiktoken

# 1. Define input text
input_text = "Who is PM of India?"

# 2. Instantiate tokenizer matching specific model architecture
# 'gpt-4' uses the 'cl100k_base' encoding schema
tokenizer = tiktoken.encoding_for_model("gpt-4")

# 3. Encode: Convert human text into integer Token IDs
token_ids = tokenizer.encode(input_text)

print("--- ENCODING PROCESS ---")
print(f"Original Text : '{input_text}'")
print(f"Token Count   : {len(token_ids)}")
print(f"Token IDs     : {token_ids}")

# Verify exact token chunks generated under the hood
token_chunks = [tokenizer.decode([tid]) for tid in token_ids]
print(f"Token Chunks  : {token_chunks}")

# 4. Simulating Model Generation (Simulated output Token IDs)
# Assume model processes input and autoregressively generates these token IDs:
generated_ids = tokenizer.encode("PM of India is Modi ji.")

# 5. Decode: Reconvert generated integer IDs back to human-readable string
decoded_output = tokenizer.decode(generated_ids)

print("\n--- DECODING PROCESS ---")
print(f"Generated IDs : {generated_ids}")
print(f"Decoded Output: '{decoded_output}'")

Execution Output

--- ENCODING PROCESS ---
Original Text : 'Who is PM of India?'
Token Count   : 7
Token IDs     : [15463, 374, 10243, 315, 3797, 30]
Token Chunks  : ['Who', ' is', ' PM', ' of', ' India', '?']

--- DECODING PROCESS ---
Generated IDs : [10243, 315, 3797, 374, 27623, 2191, 13]
Decoded Output: 'PM of India is Modi ji.'

Implementation Note: Notice that punctuation (?, .) and casing boundaries are explicitly preserved. The decoding process is lossless: tokenizer.decode(tokenizer.encode(text)) == text strictly holds true across arbitrary unicode strings.

8. Context Windows: Constraints and Memory Mechanics

Every Large Language Model is bounded by a fundamental architectural limit known as the Context Window.

Real-Life Analogy: The Dual-Gate Venue

Think of an LLM as a secured venue with two doors:

• An Entrance Gate (Input Prompt Window)
• An Exit Gate (Generated Output Window)

A venue has a strict structural maximum capacity. If 1,000 visitors crowd through the entrance gate, only a tiny sliver of capacity remains for the exit queue before exceeding structural limits. Conversely, if you want a massive group to exit together, you must limit how many enter upfront.

Operational Characteristics

1. Combined Threshold: The context window is the absolute upper ceiling on the sum of input tokens plus generated output tokens processed during a single inference call:

2. Stateless Inference: LLMs retain zero persistent internal memory between API invocations. To sustain multi-turn conversational experiences, client applications must bundle the entire previous chat history and feed it back into the model as input on every round-trip.
3. Context Exhaustion: When long-running conversations exceed this window:
   • Older models crash or return hard runtime exceptions (e.g., "Memory limit reached / context exceeded").
   • Production applications must implement token-pruning algorithms, sliding windows, or vector database retrieval (RAG) to truncate conversational history safely.

9. LLM Cost Architecture and the max_tokens Parameter

In enterprise environments, interactions with frontier model APIs (OpenAI, Anthropic, Google) are metered and billed on a strictly consumption-based utility model pegged to Token Quantities, not raw wall-clock compute time or server uptime.

Token Cost Breakdown

Pricing models split traffic across three main categories:

1. Input Tokens: Charged for parsing user prompts, injected system instructions, and chat history.
2. Cached Input Tokens: Discounted input tokens that the provider has already loaded and kept warm in memory from recent calls.
3. Output Tokens: Charged for model-synthesized response text.

Output tokens command a massive premium over input tokens (often $4\times$ to $8\times$ more expensive). Generating tokens requires sequential auto-regressive decoding loops across hardware clusters, preventing the parallel tensor speedups possible when processing input tokens concurrently.

Cost Metric Comparison

Cost Engineering via the max_tokens Parameter

Because production workloads cannot afford unpredictable generation bills, developers configure the max_tokens parameter during API calls:

# Conceptual client API configuration
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Draft an overview of transformer attention."}],
    max_tokens=300  # Hard ceiling on output sequence generation
)

Setting max_tokens:

• Protects engineering budgets by establishing a hard stop on auto-regressive generation loops.
• Prevents runaway outputs caused by prompt injection or model hallucination loops.
• Signals the inference engine to force-terminate sequence generation immediately if token consumption hits the specified boundary, even if an [EOS] token has not yet been emitted.

10. Summary and Key Takeaways

• LLMs as Statistical Approximations: Modern language models do not access an underlying factual database during generation; they are vast mathematical functions executing statistical inference derived from trillions of pre-trained parameters.
• The GPT Framework: Translates directly to Generative (synthesizes new content sequentially), Pre-trained (self-supervised learning across web-scale data), and Transformer (deep learning neural network relying on the Attention mechanism).
• Self-Attention Resolves Ambiguity: The Attention Mechanism dynamically calculates relevance scores across distinct words in a sequence, disambiguating multi-meaning words (such as determining whether bank refers to finance or topography via contextual words like steep).
• Autoregressive Generation: Text is generated one token at a time. Each predicted token is appended to the input sequence, forming the context for the subsequent probability pass over the model's vocabulary.
• Text to Token Mapping: Computers process numbers, not text. Tokenizers segment sentences into subword tokens, which map to unique integer Token IDs via a static dictionary lookup before passing to the model.
• Tokenizer Volatility: Token mappings and vocabulary configurations differ significantly across model families (cl100k_base vs. o200k_base); token IDs cannot be mixed between incompatible models.
• Reversible Encoding: Subword tokenization using libraries like tiktoken allows deterministic, lossless translation between strings and vectors of integers via symmetric .encode() and .decode() operations.
• Context Window Ceilings: An LLM’s context window defines the absolute combined capacity of input prompt tokens and generated output tokens within a single forward pass.
• Economic Optimization: Production API utilization is billed strictly per token. Because output tokens cost orders of magnitude more than input tokens, defining programmatic constraints using max_tokens is essential for application stability and financial control.