Chapter 7: Vectors & Embeddings Explained | How AI Understands Meaning

In Generative AI and Large Language Models (LLMs), vectors and embeddings form the mathematical bedrock that allows machines to process human language. While computers natively operate on numbers, human communication relies on nuanced, context-dependent semantics. This chapter explores how we transition from naive discrete computing to continuous vector spaces, examining how modern AI models convert text, images, and audio into high-dimensional numerical representations that capture real-world meaning.

1. The Core Problem: How Computers Process Meaning

When a human hears the word "Batman", a vast network of associations is immediately triggered:

• Concepts & Categories: Superhero, DC Comics, Gotham City, Vigilante.
• Attributes & Contexts: Wealthy (Bruce Wayne), Dark, Martial Arts, Action, Justice.

Computers do not inherently understand human concepts or textual semantics. At their core, digital systems store data as binary strings ($0$s and $1$s).

To process text, early language systems assigned arbitrary integer IDs to words via Tokenization:

While tokenization converts raw strings into numeric format, integer assignments carry zero semantic value:

• The integer 23 does not inherently inform the computer that Batman is a superhero.
• Token 23 is as abstract to the machine as the raw character string "Batman".
• Without context or positional relationships, isolated tokens cannot convey meaning to an LLM or traditional program.

2. The Limits of Naive Categorization & String Matching

To understand why simple numerical mappings fail, consider how computer programs historically evaluated semantic relationships.

Phase 1: Exact Identity Matching (Character Matching)

Early software relied on exact identity comparisons:

# Exact String Comparison
string_1 = "Batman"
string_2 = "Batman"

if string_1 == string_2:
    print("Same Meaning") # Output: True

This logic evaluates character codes sequentially. However, real-world human intent breaks character-level matching:

• Sentence 1: "I forgot my password."
• Sentence 2: "I lost my credentials."

To a human, both sentences communicate identical intent. To a machine performing string matching, character evaluations diverge at the second word (forgot $\neq$ lost), returning False.

Phase 2: Scalar Numerical Mapping (Categorical IDs)

To attempt mathematical relationships, one might assign sequential scalar numbers to concepts:

If we evaluate these assignments using standard arithmetic operations:

This ordering introduces nonsensical constraints:

• It implies that a Banana is mathematically "smaller" than Batman, which in turn is "smaller" than Superman.
• It suggests Banana is closer in value to Batman ($\vert 12 - 23\vert = 11$) than Batman is to Superman ($\vert 23 - 43\vert = 20$).

Scalar assignments fail because language concepts are multidimensional; single numbers cannot capture domain differences such as Fruit vs. Superhero.

3. The Mental Model: The Netflix Recommendation Engine

To construct a mathematical system that accurately represents real-world entities, consider a classic engineering problem: Building a Movie Recommendation System.

Suppose a streaming platform wants to recommend movies based on genre similarity without relying on advanced neural networks.

Draft 1: The 1-Dimensional Feature Space

We assign every movie a score from $0$ to $10$ along a single attribute axis: Action.

We measure similarity using the absolute scalar distance: $d = \vert x_1 - x_2\vert$.

• $\text{Distance}(\text{Avengers}, \text{Batman}) = \vert 10 - 9\vert = 1$ $\rightarrow$ High Similarity
• $\text{Distance}(\text{Avengers}, \text{Hera Pheri}) = \vert 10 - 2\vert = 8$ $\rightarrow$ Low Similarity

Critical Flaw in 1D Space:

Both Titanic (a romantic drama) and Hera Pheri (a slapstick comedy) receive an Action score of $2$. The 1D distance between them is $\vert 2 - 2\vert = 0$. The system evaluates Titanic and Hera Pheri as identical movies, leading to invalid recommendations.

Draft 2: The 2-Dimensional Feature Space

To resolve this collision, we introduce a second axis: Comedy.

Each movie is now represented by a 2D ordered tuple: $\mathbf{v} = (x, y) = (\text{Action}, \text{Comedy})$.

Plotting these coordinates on a 2-dimensional Cartesian plane separates Titanic from Hera Pheri:

• Titanic Coordinates: $(2, 1)$
• Hera Pheri Coordinates: $(2, 10)$
• The $y$-axis distance separates the two movies by $9$ units, preventing incorrect recommendation matches.

Definition — Vector: An ordered collection or array of numbers where each position corresponds to a specific dimensional axis in space. Order is strictly preserved: Vector $(2, 10) \neq (10, 2)$.

Draft 3: Scaling to Higher $N$-Dimensional Spaces

While a 2D space improves recommendations, two dimensions remain insufficient to capture all movie characteristics. Titanic is primarily a romantic drama, but neither Action nor Comedy explicitly represents Romance.

Adding more feature dimensions allows us to capture richer semantics:

• 3D Space: $(\text{Action}, \text{Comedy}, \text{Romance})$
• 6D Space: $(\text{Action}, \text{Comedy}, \text{Romance}, \text{Darkness}, \text{Violence}, \text{Mystery})$

6D Feature Vector Profile for "Batman":

Higher Dimensions = Richer Semantic Precision
Human visual limits: Max 3 Dimensions
Mathematical limits: Arbitrary N-Dimensions ($d = 768, 1536, 4096$)

Although humans cannot visualize geometry beyond $3$ dimensions, mathematical operations (such as distances and angles) generalize identically across $N$-dimensional spaces.

4. Mathematical Similarity Metrics in Vector Spaces

To evaluate how close two vectors are within an $N$-dimensional space, machine learning systems rely on three primary distance metrics.

1. Euclidean Distance

Calculates the straight-line distance between two point coordinates in $N$-dimensional space.

The Scaling Limitation of Euclidean Distance:

Consider two users scoring their movie preferences for $(\text{Action}, \text{Comedy})$:

• User A: Scores $(1, 10)$ on a $1\text{--}10$ scale.
• User B: Scores $(10, 100)$ on a $1\text{--}100$ scale.

Both users share the exact preference ratio ($1:10$), indicating identical relative tastes. However, the Euclidean distance between $(1, 10)$ and $(10, 100)$ is large:

Euclidean distance penalizes these vectors due to differences in magnitude, even though their relative direction is identical.

2. Cosine Similarity

Cosine similarity evaluates the angle $\theta$ between two vectors, ignoring difference in raw magnitude.

Metric Scale & Interpretation:

• $\mathbf{+1.0}$ ($\theta = 0^\circ$): Identical direction (Maximum similarity).
• $\mathbf{0.0}$ ($\theta = 90^\circ$): Orthogonal / Unrelated concepts.
• $\mathbf{-1.0}$ ($\theta = 180^\circ$): Diametrically opposite direction.

Because User A $(1, 10)$ and User B $(10, 100)$ point along the exact same ray ($\theta = 0^\circ$), their Cosine Similarity is $\cos(0^\circ) = 1.0$.

3. Dot Product

Combines vector magnitude and directional alignment:

If vectors are normalized to unit length ($\|\mathbf{u}\| = \|\mathbf{v}\| = 1$), the Dot Product equals Cosine Similarity. When magnitudes carry semantic importance (e.g., term frequency or intensity), the Dot Product incorporates both intensity and direction.

Summary Comparison of Distance Metrics

5. The Breakdown of Handcrafted Feature Engineering

Manually defining feature dimensions (e.g., Action, Comedy, Romance) is known as Handcrafted Feature Engineering. While intuitive, manual vector creation fails at scale due to four primary constraints:

1. Arbitrary Dimension Selection: There is no objective rule for how many dimensions fully represent an entity. Stopping at $6$ dimensions risks missing key attributes like Historical Context, VFX Intensity, or Pacing.

2. Domain & Task Specificity: Features defined for movie recommendations (genres) are useless for predicting box-office revenue (which requires Budget, Cast Popularity, and Release Window).

3. Natural Language Ambiguity (Polysemy): A single word can have multiple meanings based on context:

   • "I deposited money at the bank." $\rightarrow$ Financial Institution
   • "He sat by the river bank." $\rightarrow$ Geographical Shoreline Handcrafted features cannot dynamically adapt to context shifts.

4. Incompatible Feature Spaces: Comparing King, Queen, and Banana:

   • King/Queen Dimensions: Royalty, Power, Leadership.
   • Banana Dimensions: Fruit, Nutrition, Agriculture.

   Forcing every word into a shared manually engineered space creates sparse, bloated vectors with excessive zero-padding:

6. Representation Learning: How Neural Networks Learn Embeddings

Rather than manually constructing feature axes, modern Machine Learning utilizes Representation Learning. We task neural networks with discovering high-dimensional vector spaces automatically.

The Linguistic Intuition: Distributional Semantics

Machine-learned embeddings rely on John Rupert Firth’s Distributional Hypothesis (1957):

"You shall know a word by the company it keeps."

Consider the unfamiliar word "laden" in context:

"The tree was laden with apples."

Even without a dictionary definition, the surrounding words (tree, apples, with) allow humans—and algorithms—to infer that "laden" means heavily loaded or filled.

The Iterative Optimization Process

To learn vector representations automatically, an embedding model undergoes self-supervised training using context prediction tasks.

Training Step Breakdown:

1. Initialization: Assign small, random vector weights to all words in the vocabulary.
2. Objective Task: Pass a target word (e.g., King) and demand the model predict its surrounding context words.
3. Error Calculation: If the model predicts Banana as a context word for King, the loss function flags a high margin of error.
4. Parameter Update: Optimization algorithms (e.g., Gradient Descent) adjust the numeric vector values of King slightly away from Banana and closer to Queen.
5. Convergence: Across billions of text samples, words appearing in similar contexts naturally cluster together in the vector space.

7. Vectors vs. Embeddings vs. Latent Space

Understanding the subtle distinctions between these core AI terms is essential:

Key Conceptual Distinctions:

• Vector: Any ordered numerical array. Vectors can be handcrafted, randomly generated, or machine-learned.
• Embedding: A specific type of vector produced by a trained neural network model. Raw inputs (words, sentences, images) are embedded into a continuous, dense numerical space.
• Latent Space / Latent Dimensions: The underlying hidden feature space constructed by the model. Unlike handcrafted dimensions ($\text{Index } 0 = \text{Action}$), latent dimensions carry no isolated human-interpretable labels.

Distributed Representation: In a 1536-dimensional OpenAI embedding, concept traits like Royalty or Gender are not isolated to a single dimension. Instead, semantic concepts are distributed patterns spanning hundreds of latent axes simultaneously.

Structural Comparison

8. Code Implementation: Evaluating Similarity Metrics

The following Python script demonstrates how vector representations are evaluated using Euclidean Distance, Cosine Similarity, and Dot Product calculations with numpy.

import numpy as np

def euclidean_distance(u: np.ndarray, v: np.ndarray) -> float:
    """Calculates the straight-line Euclidean distance between two vectors."""
    return float(np.linalg.norm(u - v))

def cosine_similarity(u: np.ndarray, v: np.ndarray) -> float:
    """Calculates the cosine of the angle between two vectors."""
    dot_prod = np.dot(u, v)
    norm_u = np.linalg.norm(u)
    norm_v = np.linalg.norm(v)
    return float(dot_prod / (norm_u * norm_v))

def dot_product(u: np.ndarray, v: np.ndarray) -> float:
    """Calculates the raw dot product between two vectors."""
    return float(np.dot(u, v))

# Example 2D Vectors (Action, Comedy)
v_avengers = np.array([10.0, 7.0])
v_batman   = np.array([9.0, 1.0])
v_hera_pheri = np.array([2.0, 10.0])

print("--- Vector Similarity Evaluations ---")

# 1. Avengers vs Batman (Action Heavy)
euc_ab = euclidean_distance(v_avengers, v_batman)
cos_ab = cosine_similarity(v_avengers, v_batman)
dot_ab = dot_product(v_avengers, v_batman)

print(f"Avengers vs Batman:")
print(f"  - Euclidean Distance : {euc_ab:.4f} (Lower = Closer)")
print(f"  - Cosine Similarity  : {cos_ab:.4f} (Higher = Closer)")
print(f"  - Dot Product        : {dot_ab:.4f}\n")

# 2. Avengers vs Hera Pheri (Action vs Comedy)
euc_ah = euclidean_distance(v_avengers, v_hera_pheri)
cos_ah = cosine_similarity(v_avengers, v_hera_pheri)
dot_ah = dot_product(v_avengers, v_hera_pheri)

print(f"Avengers vs Hera Pheri:")
print(f"  - Euclidean Distance : {euc_ah:.4f}")
print(f"  - Cosine Similarity  : {cos_ah:.4f}")
print(f"  - Dot Product        : {dot_ah:.4f}")

Summary and Key Takeaways

• Computers Require Numeric Representations: Discrete character strings and token IDs lack intrinsic semantic meaning; models require vector-space representations to perform semantic reasoning.
• Vectors as Spatial Coordinates: A vector is an ordered collection of numbers mapping an entity to coordinates within an $N$-dimensional geometric space.
• Dimensions Capture Nuance: Higher-dimensional spaces resolve semantic collisions by separating distinct features along independent axes.
• Cosine Similarity Overcomes Scale Discrepancies: While Euclidean distance measures spatial distance, Cosine Similarity evaluates directional alignment, making it ideal for text and document matching regardless of magnitude.
• Handcrafted Feature Engineering Fails at Scale: Manual feature creation cannot handle language ambiguity, polysemy, domain variance, or non-overlapping feature spaces across different concept types.
• Embeddings are Learned Representations: Embeddings are dense, continuous vectors generated automatically by neural networks trained on distributional context tasks.
• Latent Dimensions are Distributed: Dimensions in neural embeddings do not map to explicit human labels; semantic traits are distributed across hundreds or thousands of latent axes.
• Fundamental AI Primitives: Embedding functions $f(\text{Input}) \to \mathbb{R}^d$ serve as foundational components across modern LLMs, vector databases, and semantic search systems.