AI Engineering: Understanding Foundation Models

Foundation models serve as the structural backbone of modern artificial intelligence engineering. To architect, optimize, and deploy systems powered by these models, an engineer must dissect them across four interconnected lifecycle pillars: training data, core modeling architecture, post-training alignment, and inference-time sampling dynamics.

1. The Four Pillars of Foundation Models

A foundation model is not merely a black-box function; its behavior, failure modes, cost profile, and capabilities are direct consequences of decisions made across four distinct engineering stages:

1. Training Data: Defines the operational boundaries, inherent biases, factual boundaries, and token efficiency of the model.
2. Modeling Architecture: Dictates memory footprints, computational complexity ($O(N^2)$ vs. $O(N)$), context window scalability, and parallelization limits.
3. Post-Training Alignment: Transforms an unconstrained auto-complete statistical engine into an instruction-following assistant aligned with human intent and safety constraints.
4. Sampling and Inference: Governs runtime decoding, balancing determinism versus creativity, enforcing structured schema adherence, and controlling stochastic variability.

2. Pre-Training Data Dynamics and Token Economics

A neural network is fundamentally bounded by the quality and distribution of its pre-training corpus: a smaller, highly curated dataset frequently produces superior downstream performance compared to an uncurated, noisy, ultra-large dataset.

Web Scraping and Data Cleansing

The primary raw ingredient for foundational language models is massive web scrape data (such as the Common Crawl dataset). However, raw web crawls contain toxicity, duplications, adversarial spam, machine-generated gibberish, and factual hallucinations.

Google introduced the C4 (Colossal Clean Crawled Corpus) dataset by applying aggressive heuristic filters to Common Crawl:

• Deduplication of documents and boilerplate text.
• Removal of code snippets, placeholder strings, and lorem ipsum text.
• Filtering of offensive, toxic, or sexually explicit content.
• Removal of pages where sentences did not end in valid terminal punctuation marks.

High-Resource vs. Low-Resource Languages

Modern foundational models exhibit substantial behavioral disparities between high-resource languages (e.g., English, Spanish) and low-resource languages (e.g., Telugu, Vietnamese, Hindi).

• Data Imbalance: Low-resource languages comprise a fraction of a percent of raw Common Crawl data. Even when trained with multilingual objectives, translation representations remain bounded by the quality of translation pairs.
• Token Fragmentation: Tokenizers (such as Byte-Pair Encoding or WordPiece) are optimized primarily on high-resource corpora. A single concept expressed in English might consume 1 token, whereas expressing the identical concept in Hindi or Vietnamese may require 4 to 8 subword tokens.

Token Economics, Latency, and Cost Asymmetry

Because cloud API providers (OpenAI, Anthropic, Google) charge monetarily on a per-token basis (both for prompt ingestion and autoregressive completion), operating in low-resource languages incurs a direct financial penalty:

Because autoregressive generation is computed strictly token by token, an increased token count directly inflates end-to-end inference latency.

3. Modeling Architectures: From Recurrent Bottlenecks to Self-Attention

Before the Transformer architecture, Sequence-to-Sequence (Seq2Seq) tasks relied almost exclusively on Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs).

The Recurrent Bottleneck

RNNs processed inputs sequentially:

This design had two fundamental architectural flaws:

1. The Context Vector Bottleneck: The decoder generated target tokens relying strictly on the final hidden state ($h_{\text{final}}$). The network was forced to compress an entire document, sequence, or book into a single fixed-size mathematical vector.
2. Vanishing and Exploding Gradients: Unrolling recurrent computations over long temporal horizons caused gradients during backpropagation to iteratively shrink toward zero or explode exponentially, preventing the retention of long-range dependencies.
3. Sequential Compute: Processing token $t$ strictly required the completed computation of token $t-1$, completely preventing GPU parallelization during encoding.

The Transformer Breakthrough

The Transformer removed recurrent state transitions, swapping temporal recurrence for parallelized self-attention.

Real-Life Analogy

An RNN is like an analyst reading a 500-page book who must throw away every page as they read it, attempting to write a full report solely from the mental memory left behind on the final page. The Transformer is like an analyst sitting with all 500 pages laid out across a large conference table simultaneously, using highlighters to trace and cross-reference connections across every page instantly.

4. Dissecting the Transformer Layer and Scaled Dot-Product Attention

Input Embedding and Positional Encoding

Because self-attention operations are permutation-invariant (they compute unordered set operations across inputs), positional information must be injected explicitly:

where $\mathbf{X}$ is the one-hot encoded token matrix, $\mathbf{W}_{\text{embed}} \in \mathbb{R}^{V \times d_{\text{model}}}$, and $\mathbf{P} \in \mathbb{R}^{S \times d_{\text{model}}}$ represents positional embeddings (learned or sinusoidal).

The Attention Mechanism

Input representations are projected into three distinct spaces via learned weight matrices:

• Queries ($\mathbf{Q}$): $\mathbf{Q} = \mathbf{Z}\mathbf{W}_Q \in \mathbb{R}^{S \times d_k}$
• Keys ($\mathbf{K}$): $\mathbf{K} = \mathbf{Z}\mathbf{W}_K \in \mathbb{R}^{S \times d_k}$
• Values ($\mathbf{V}$): $\mathbf{V} = \mathbf{Z}\mathbf{W}_V \in \mathbb{R}^{S \times d_v}$

The attention weights are computed via scaled dot-product operations:

• $\sqrt{d_k}$ serves as a variance-scaling factor. When inner products grow large in high-dimensional vector spaces, the softmax function pushes activations into regions with near-zero gradients. Scaling stabilizes variance to $1$.
• $\mathbf{M}$ is an optional causal mask matrix containing $0$ for valid attention indices and $-\infty$ for future token positions, preventing autoregressive decoders from attending to future context tokens.

Multi-Head Attention (MHA)

Rather than computing a single set of attention patterns, Multi-Head Attention splits the model dimension across $h$ distinct attention heads, allowing the network to simultaneously focus on syntax, factual references, and conversational tone:

In a baseline 7B parameter architecture (such as LLaMA-7B):

• Hidden dimension $d_{\text{model}} = 4096$
• Number of heads $h = 32$
• Per-head dimension $d_k = \frac{4096}{32} = 128$

Feed-Forward Networks (MLP) and Normalization

Following attention projection, representations pass through a Multi-Layer Perceptron (MLP) with non-linear activations:

where $\sigma$ represents non-linear activations such as GeLU or SwiGLU. Modern models utilize Root Mean Square Normalization (RMSNorm) and residual connections to stabilize deep forward/backward passes:

5. Architectural Frontiers: $O(N^2)$ Complexity, Mamba, and Mixture of Experts (MoE)

The Quadratic Bottleneck

Standard self-attention computes an $S \times S$ attention matrix, where $S$ is the sequence length. As sequences increase, memory and computational demands scale quadratically ($O(S^2)$).

Sequence Length (Tokens)     Attention Operations (Relative Scale)
1,000                        1,000,000       (1M)
8,000                        64,000,000      (64M)
32,000                       1,024,000,000   (1B)
128,000                      16,384,000,000  (16.4B)

State Space Models: Mamba

To overcome the quadratic wall, non-transformer alternatives such as Mamba leverage continuous State Space Models (SSMs).

• Linear Complexity: Transforms the compute curve from $O(S^2)$ to $O(S)$ relative to sequence length.
• Selective State Retention: Selectively propagates or drops input tokens along a temporal hidden state trajectory, matching Transformer reasoning efficiency on dense tasks while operating with a minimal constant memory footprint per generation step.

Sparse Mixture of Experts (MoE)

Standard foundational models are "dense": every single parameter is activated for every incoming token. A 70-billion parameter model executes 70 billion parameter computations for a single punctuation token.

Mixture of Experts (MoE) replaces the dense MLP layer with multiple parallel "expert" networks coordinated by a lightweight parametric router:

                 Input Token Vector
                         |
                 [ Softmax Router ]
                    /          \
        Route to Expert 2   Route to Expert 7
                  |              |
             [ Expert 2 ]   [ Expert 7 ]
                  \              /
                Weighted Aggregation
                         |
                    Next Layer

• A model may contain 50B total parameters across 8 individual expert blocks.
• The router conditionally activates only the top-2 experts per token.
• Inference Footprint: The model processes the compute load of a ~10B parameter model while retaining the expressiveness, factual recall, and pre-training capacity of a 50B parameter network.

6. Scaling Laws and Compute Budgets

Foundation model pre-training is bounded by hardware capacity, raw FLOPs (Floating Point Operations), and budget constraints.

The Chinchilla Optimal Scaling Laws

Historically (e.g., GPT-3), models were scaled aggressively in parameter count while keeping tokens relatively constant, resulting in severely undertrained models. The DeepMind Chinchilla research showed that parameters and training tokens should scale in equal proportion:

Note: Production engineering favors inference-optimized models. Models such as LLaMA-3 are intentionally trained far beyond the compute-optimal Chinchilla frontier. While this increases initial training FLOPs, it reduces parameter size, driving down per-token serving costs and latency across billions of subsequent inference requests.

Real-World Training Financials

Consider training an early-generation 175B model requiring $3.14 \times 10^{23}$ total FLOPs:

• Cluster size: 256 NVIDIA H100 GPUs.
• Theoretical performance: $\approx 5.2 \times 10^{18}$ FLOPs/day per GPU.
• Achieved Model Flops Utilization (MFU): ~70% (due to communication, thermal throttling, and pipeline bubbles).
• Training duration: $\approx 236 \text{ days}$.
• At an infrastructure hosting rate of $2.00 per H100 hour:

When including pipeline retries, checkpoint rewinds, CPU nodes, interconnects, and cooling, base cluster runs easily exceed $4,000,000.

Scaling Bottlenecks

• Exhaustion of Human Data: High-quality public text is projected to be exhausted, pushing labs toward synthetic data loops.
• Model Collapse: Recursively training models on synthetic data from unverified model outputs causes the network to lose tail distributions and experience semantic drift.
• Power and Grid Infrastructure: Training frontier runs demands dedicated data center power grids running at hundreds of megawatts.

7. The Post-Training Pipeline: SFT, Reward Modeling, and RLHF

Raw pre-trained foundation models are pure text completion engines. Asked "How do I bake a pizza?", an unaligned base model might complete the prompt with "How do I bake a calzone? Can anyone help me with these recipes?" because it models document continuation rather than instruction execution.

Supervised Fine-Tuning (SFT)

SFT curates tens of thousands of high-quality demonstration prompts paired with human-written expert responses. The objective function remains cross-entropy token prediction, but executed purely across supervised target tokens:

Reward Modeling via Pairwise Comparison

Humans struggle to provide uniform, calibrated scalar grades (e.g., assigning a 7.5 vs. an 8.2 out of 10) to open-ended model answers. However, humans readily identify the superior response in a head-to-head comparison.

Pairwise reward datasets format records containing a prompt $x$, a winning response $y_w$, and a losing response $y_l$. The learned reward model $r_\psi(x, y)$ assigns scalar scores to outputs optimized via negative log-sigmoid loss:

Maximizing this objective trains the reward model parameters $\psi$ to separate positive completions from subpar, toxic, or incorrect completions.

Policy Optimization (RLHF vs. DPO)

• RLHF (via Proximal Policy Optimization - PPO): The SFT model acts as a policy producing responses to random queries. The generated output is scored by the frozen Reward Model. PPO adjusts policy weights to maximize the scalar reward while applying a Kullback-Leibler (KL) divergence penalty to prevent the model from drifting too far from the initial SFT baseline.
• Direct Preference Optimization (DPO): Eliminates the complex machinery of training a standalone reward model and managing multi-model GPU allocations. DPO algebraically expresses the reward function directly through the policy model, optimizing preference pairs via an implicit, stable cross-entropy loss.

8. Inference Dynamics and Decoding Strategies

Foundation models output raw unnormalized vectors of real numbers representing logits $\mathbf{z} \in \mathbb{R}^{V}$ across the vocabulary. Converting logits to actual tokens relies on configurable decoding dynamics.

       Model Backbone Forward Pass
                   |
           Raw Logits Vector (z)
                   |
           [ Temperature Scaling: z_i / T ]
                   |
       [ Top-K / Top-P Logit Truncation ]
                   |
           [ Softmax Exponentiation ]
                   |
         Probability Distribution P(y)
                   |
              Sampling Engine
                   |
             Emitted Token

Logits, Log-Probabilities, and Numerical Stability

Calculating raw joint probabilities across output sequences causes numerical arithmetic underflows:

Repeated multiplication of fractional values ($0.003 \times 0.04 \times \dots$) inevitably rounds to $0.000000$ in floating-point memory. Models therefore operate internally in log-space:

This transforms products into sums, maintaining floating-point stability across thousands of sequence steps.

Temperature Scaling

Temperature ($T$) re-scales the logit vector before calculating the softmax probability distribution:

• $T \to 0$ (Greedy Decoding): Logit differences are magnified exponentially. The highest logit collapses probability mass strictly toward $1.0$, rendering generation fully deterministic.
• $T = 1.0$: Unaltered model distribution based on learned training cross-entropy.
• $T > 1.0$ (High Temperature): Logit values flatten toward uniformity. Low-probability, unconventional tokens gain non-trivial probabilities, yielding diverse completions at the expense of structural degradation.

Top-$k$ vs. Top-$p$ (Nucleus) Sampling

• Top-$k$: Limits candidates strictly to the top $k$ tokens. This fixed threshold can be problematic: in flat distributions, it prunes viable tokens; in peaked distributions, it retains irrelevant tokens.
• Top-$p$ (Nucleus): Adaptively selects candidates from the smallest set of tokens whose cumulative probability exceeds the threshold $p \in (0, 1]$. In highly certain contexts, the nucleus shrinks to 1 or 2 tokens; in open-ended contexts, it dynamically expands to include a wider set of valid candidates.

9. Test-Time Compute, Reasoning Scaling, and Structured Outputs

A central frontier in AI systems engineering is moving beyond fixed per-token training parameters toward scalable test-time compute—allocating dynamic computing power during inference to improve output quality.

Test-Time Compute Strategies

1. Best-of-$N$ Sampling: The system generates $N$ independent candidate trajectories for a single prompt at $T > 0.7$. A trained verifier or reward model evaluates each candidate and selects the trajectory with the highest scalar score.
2. Self-Consistency: When generating solutions for deterministic tasks (e.g., mathematics, coding), the model samples dozens of reasoning paths. The final response is determined by majority vote across output answers.
3. Beam Search vs. Dynamic Search Trees: Exploration of high-probability paths across step-by-step reasoning tokens before final emission.

Note on Adversarial Degradation: OpenAI empirical studies revealed that scaling candidate trajectories arbitrarily does not produce monotonic accuracy improvements. As $N$ expands to high volumes, the probability of sampling adversarial completions or reward-hacking sequences increases, leading to an eventual drop in output accuracy.

Enforcing Structured Outputs

Production downstream pipelines frequently require valid JSON schemas, SQL statements, or structured function calls. Unconstrained generative sampling frequently breaks these formats.

Engineers resolve this via four primary patterns:

Constrained Decoding Mechanisms: Modern inference engines enforce valid syntax by modifying generation directly at the logit level. If an API contract requires an open JSON object ({"name": ...}), the engine applies an absolute causal mask ($-\infty$) to every token in the vocabulary that violates the JSON grammar at that specific generation step. This guarantees that invalid tokens are mathematically impossible to sample.

10. The Stochastic Nature of Foundation Models: Hallucination and Inconsistency

Foundation models are probabilistic systems. Operating them in mission-critical software pipelines introduces challenges around inconsistency and hallucination.

The Inconsistency Dilemma

Because autoregressive sampling can be non-deterministic, identical prompts may yield divergent responses across separate runs. Minor perturbations—such as changing a single character's capitalization or altering whitespace—can significantly shift generation trajectories.

Engineering Mitigations:

• Deterministic Sampling: Set temperature strictly to $0$ and utilize top-$p = 1.0$.
• Retrieval-Augmented Generation (RAG): Anchor generation by injecting definitive context into the prompt, reducing the model's reliance on parametric weights.

Root Causes of Hallucination

Hallucination refers to the generation of plausible-sounding statements unsupported by factual reality.

1. Pre-Training Factual Inconsistencies: The underlying web corpus contains millions of contradictions, speculative claims, satire, and uncorrected historical errors.
2. Autoregressive Self-Deception: A model conditions strictly on preceding tokens:

If the model emits a minor inaccuracy early in a sequence due to stochastic variance, it treats that hallucinated token as ground truth for all subsequent generation steps. To preserve linguistic coherence, it progressively fabricates additional claims around the original error. 3. Knowledge Misalignment in Alignment Tuning: During SFT and RLHF, preference raters may score a response positively simply because it sounds confident, articulate, and helpful, even when it contains subtle factual errors. This teaches the model to prioritize conversational confidence over factual uncertainty.

11. Complete System Walkthrough: From Raw Web to Structured Query Production

The following end-to-end trace illustrates how these disparate systems interact across a complete production pipeline:

12. Summary and Key Takeaways

• Data Determines Performance: A foundation model's capabilities are primarily bounded by the quality and domain relevance of its pre-training corpus. Small, highly curated datasets often outperform larger, noisier web scrapes.
• The Tokenization Penalty: Tokenizers are optimized primarily for high-resource languages. Low-resource languages require significantly more tokens to convey equivalent meaning, which directly drives up operational costs and latency.
• Attention Overcomes the Recurrent Bottleneck: Traditional RNNs bottlenecked information through a single context vector and suffered from vanishing gradients. The Transformer resolves this by enabling parallel self-attention across an entire sequence.
• Quadratic vs. Linear Scaling: Standard attention scales quadratically ($O(S^2)$) with context length. Alternative architectures such as Mamba offer linear scaling ($O(S)$), while Mixture of Experts (MoE) maintains lower runtime compute by selectively activating subsets of parameters per token.
• Compute-Optimal Training: Chinchilla scaling laws state that parameter counts and training token volumes should scale in equal proportion, with roughly 20 training tokens per parameter. However, production deployments often deliberately over-train smaller models to minimize serving latency.
• Alignment Refines Base Capabilities: Unaligned base models are simple text completion engines. Supervised Fine-Tuning (SFT) and preference optimization (RLHF/DPO) align them to follow instructions safely and reliably.
• Decoding Balances Novelty and Precision: Temperature re-scales logit variances to trade off creativity and predictability. Nucleus sampling (top-$p$) dynamically adjusts candidate pools based on cumulative probability, adapting to context more effectively than fixed top-$k$ truncation.
• Inference Scaling via Test-Time Compute: Increasing inference-time computation through mechanisms like Best-of-$N$ sampling, verifier models, and self-consistency voting can yield significant performance gains without expanding pre-trained model weights.
• Mitigating Stochastic Errors: Hallucinations stem from pre-training noise, autoregressive error propagation, and misaligned preference tuning. Production architectures manage these behaviors through deterministic decoding, grammar-constrained logit masking, and retrieval-augmented generation.