Building a Character-Level Language Model from Scratch

An end-to-end guide to designing, training, and sampling from a character-level language model using PyTorch. Inspired by Andrej Karpathy's makemore project, this chapter demonstrates how to build a Multi-Layer Perceptron (MLP) that learns the underlying structure of text data—specifically Indian male names—to generate entirely new, plausible names from scratch.

1. High-Level Problem Statement & System Design

The core objective of a character-level language model is to predict the next character in a sequence given a fixed window of preceding characters. Unlike large token-level language models, a character-level model operates directly on individual letters and symbols.

The system processing lifecycle is divided into two distinct primary phases:

2. Technical Prerequisites & Environment Setup

To construct this architecture from first principles, four core competencies and tools are required:

1. Python Proficiency: Working knowledge of data structures, list comprehensions, dictionary mappings, and data manipulation.
2. Neural Network Fundamentals: Intuitive grasp of forward passes, loss calculation, backpropagation (gradient propagation), weight updates, and hyperparameter tuning.
3. PyTorch Framework: Hands-on familiarity with torch.Tensor, torch.nn modules, parameter tensors, and gradient management (loss.backward()).
4. Data Science Libraries: pandas for tabular dataset ingestion and matplotlib for training loss curves visualization.

3. Feature Engineering I: Data Cleaning & Vocabulary Extraction

Real-World Analogy: The LEGO Brick Vocabulary

Think of the extracted unique character vocabulary as a set of distinct LEGO bricks. The raw dataset provides all available brick shapes (letters and symbols). The trained neural network acts as a master builder that receives these individual LEGO bricks and learns the probability rules of how to snap them together to construct coherent, recognizable structures (names).

Dataset Ingestion and Cleaning

The dataset contains 14,845 records of Indian male names along with metadata columns. Before feeding data into a model, non-string entries and invalid values must be stripped out.

import torch
import pandas as pd
import matplotlib.pyplot as plt

# 1. Load dataset
df = pd.read_csv("indian_male_names.csv")

# 2. Extract raw name column and inspect count
raw_names = df["name"].tolist()
print(f"Initial raw record count: {len(raw_names)}")

# 3. Clean dataset by removing invalid non-string entries
valid_names = []
corrupted_count = 0

for name in raw_names:
    if isinstance(name, str) and name.strip():
        valid_names.append(name.strip())
    else:
        corrupted_count += 1

print(f"Corrupted records removed: {corrupted_count}")
print(f"Cleaned valid name count: {len(valid_names)}")

Dataset Output Summary:

• Original Count: $14,845$ records
• Invalid/Corrupted Records Removed: $24$ records
• Cleaned Record Total: $14,821$ valid names

Unique Vocabulary Construction & Token Mappings

Neural networks cannot process raw strings; they require numerical identifiers. We construct a unique set of characters (the vocabulary) from all cleaned names and add a special padding/stop token.

# Extract all unique characters across all names
unique_chars = sorted(list(set("".join(valid_names))))
print(f"Unique characters found: {len(unique_chars)}")

# Define special stop/start token
SPECIAL_TOKEN = "."

# Vocabulary size: unique characters + 1 special token
vocab = unique_chars + [SPECIAL_TOKEN]
vocab_size = len(vocab)
print(f"Total Vocabulary Size (V): {vocab_size}")

# Build bidirectional lookup maps
char_to_index = {ch: i for i, ch in enumerate(vocab)}
index_to_char = {i: ch for i, ch in enumerate(vocab)}

Note: The unique character set contains 87 characters (comprising Latin characters and Hindi script characters present in the dataset) plus 1 special token (.), yielding a total vocabulary size $V = 88$.

4. Feature Engineering II: Context-Target Dataset Construction

To train a predictive language model, text strings must be sliced into input-target pairs using a sliding context window.

Sliding-Window Concept

For a given context block size $N = 3$, the model uses 3 continuous characters to predict the 4th target character. Names are padded at the start with special stop tokens so the network learns valid starting letters.

Example: Processing the name "Emma" (using . as padding/stop token):

import random

def build_dataset(names, block_size=3):
    X, Y = [], []
    stop_idx = char_to_index["."]
    
    for name in names:
        # Initialize context buffer padded with special stop tokens
        context = [stop_idx] * block_size
        
        # Iterate over characters including an implicit end token
        for ch in name + ".":
            target_idx = char_to_index[ch]
            X.append(context)
            Y.append(target_idx)
            # Slide window to the left
            context = context[1:] + [target_idx]
            
    return torch.tensor(X), torch.tensor(Y)

# Shuffle dataset for uniform distribution
random.seed(42)
random.shuffle(valid_names)

# Generate dataset tensors
X_all, Y_all = build_dataset(valid_names, block_size=3)

# 80/20 Train/Test Split
n_train = int(0.8 * len(X_all))
X_train, Y_train = X_all[:n_train], Y_all[:n_train]
X_test, Y_test = X_all[n_train:], Y_all[n_train:]

print(f"Total dataset samples: {X_all.shape[0]}")
print(f"Training set shape (X_train): {X_train.shape}, (Y_train): {Y_train.shape}")
print(f"Test set shape     (X_test):  {X_test.shape},  (Y_test):  {Y_test.shape}")

Dataset Transformations:

• Original Names Count: $14,821$
• Generated Context-Target Pairs: $163,500$ samples
• Training Subset ($80\%$): $130,800$ samples
• Testing Subset ($20\%$): $32,700$ samples

5. Vector Embeddings: Mapping Characters to Continuous Space

Passing raw integer indices ($0$ to $87$) directly into a neural network creates artificial ordinal relationships (e.g., character index $80$ is not mathematically "80 times greater" than character index $1$).

Instead, each integer index is projected into a $d$-dimensional continuous space ($d = 5$).

# Embedding Parameters
vocab_size = 88
n_embd = 5     # 5-dimensional embedding space
block_size = 3 # 3 context characters

# PyTorch Embedding Layer
C = torch.nn.Embedding(num_embeddings=vocab_size, embedding_dim=n_embd)

# Lookup demonstration for a context block of 3 characters
example_context = torch.tensor([0, 0, 5]) # Indices representing ['.', '.', 'E']
embedded_context = C(example_context)

print(f"Raw Input Index Tensor Shape: {example_context.shape}")
print(f"Embedded Tensor Shape:        {embedded_context.shape}") 
# Shape: (3, 5) -> 3 context characters, each mapped to 5 dimensions

When concatenating $3$ context tokens each represented by a $5$-dimensional vector, the effective input length to the first hidden layer becomes $3 \times 5 = 15$ continuous dimensions.

6. Neural Network Architecture: Multi-Layer Perceptron (MLP)

The model relies on a dense multi-layer perceptron enhanced with Batch Normalization and Tanh Activations.

Architectural Topology

Mathematical Foundations

1. Hyperbolic Tangent Activation ($\tanh$): Pushes values into a stable $[-1, 1]$ range to prevent exploding activations across deeper layers.

2. Batch Normalization: Stabilizes training by zero-centering and scaling activations across the mini-batch:

where $\mu_{\text{batch}}$ is the mean, $\sigma^2_{\text{batch}}$ is variance, $\epsilon$ is numerical stability constant, and $\gamma, \beta$ are learnable parameters.

3. Softmax Output Transformation: Converts raw network outputs ($\text{logits}$) into a valid probability distribution over all 88 possible characters:

Network Definition in PyTorch

import torch.nn as nn
import torch.nn.functional as F

class CharLM(nn.Module):
    def __init__(self, vocab_size, n_embd, block_size, n_hidden):
        super().__init__()
        self.block_size = block_size
        self.n_embd = n_embd
        
        # Embedding Matrix
        self.emb = nn.Embedding(vocab_size, n_embd)
        
        # Layer 1: Flattened input (block_size * n_embd) -> n_hidden
        self.fc1 = nn.Linear(block_size * n_embd, n_hidden)
        self.bn1 = nn.BatchNorm1d(n_hidden)
        self.act1 = nn.Tanh()
        
        # Layer 2: n_hidden -> n_hidden
        self.fc2 = nn.Linear(n_hidden, n_hidden)
        self.bn2 = nn.BatchNorm1d(n_hidden)
        self.act2 = nn.Tanh()
        
        # Output Layer: n_hidden -> vocab_size
        self.fc3 = nn.Linear(n_hidden, vocab_size)
        self.bn3 = nn.BatchNorm1d(vocab_size)

    def forward(self, x):
        # x shape: (batch_size, block_size)
        embeds = self.emb(x) # (batch_size, block_size, n_embd)
        
        # Flatten embeddings for dense layer: (batch_size, block_size * n_embd)
        x_flat = embeds.view(embeds.size(0), -1)
        
        # Hidden Layer 1
        h1 = self.act1(self.bn1(self.fc1(x_flat)))
        
        # Hidden Layer 2
        h2 = self.act2(self.bn2(self.fc2(h1)))
        
        # Output Logits
        logits = self.bn3(self.fc3(h2))
        return logits

# Instantiate model
model = CharLM(vocab_size=88, n_embd=5, block_size=3, n_hidden=100)

7. Training Pipeline & Batch Normalization Mechanics

Mini-Batch Optimization

Given a dataset of 130,800 training samples, computing gradients over the entire dataset per step is computationally expensive. Instead, mini-batches of size $512$ are randomly sampled at each iteration.

# Hyperparameters
batch_size = 512
learning_rate = 0.1
iterations = 100000

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
loss_history = []

model.train()
for i in range(iterations):
    # 1. Sample mini-batch indices randomly
    ix = torch.randint(0, X_train.shape[0], (batch_size,))
    Xb, Yb = X_train[ix], Y_train[ix]
    
    # 2. Forward pass
    logits = model(Xb)
    loss = F.cross_entropy(logits, Yb)
    
    # 3. Backward pass & parameter update
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    # Track metrics
    loss_history.append(loss.item())
    
    if (i + 1) % 10000 == 0 or i == 0:
        print(f"Iteration {i+1:6d} / {iterations} | Loss: {loss.item():.4f}")

Note: F.cross_entropy combines log_softmax and Negative Log-Likelihood (nll_loss) into a single numerically stable function. It handles converting raw logits into target probabilities internally.

Training Loss Evolution

Due to mini-batch sampling, the loss curve displays local oscillations while maintaining a consistent overall downward trajectory.

8. Inference & Categorical Sampling Workflow

Once trained, generating new text requires sampling characters iteratively from the network's output probability distribution.

@torch.no_grad()
def generate_names(model, num_samples=10):
    model.eval()
    stop_idx = char_to_index["."]
    generated_names = []
    
    for _ in range(num_samples):
        out = []
        # Start context initialized to ['.', '.', '.']
        context = [stop_idx] * model.block_size
        
        while True:
            # Prepare tensor input: shape (1, block_size)
            x_in = torch.tensor([context])
            
            # Forward pass to get logits
            logits = model(x_in)
            
            # Convert logits to probabilities via Softmax
            probs = F.softmax(logits, dim=1)
            
            # Sample next character index from multinomial distribution
            next_idx = torch.multinomial(probs, num_samples=1).item()
            
            # Terminate string generation if stop token is reached
            if next_idx == stop_idx:
                break
                
            out.append(index_to_char[next_idx])
            
            # Slide context window left
            context = context[1:] + [next_idx]
            
        generated_names.append("".join(out))
        
    return generated_names

# Sample generated names from trained model
samples = generate_names(model, num_samples=10)
for idx, name in enumerate(samples, 1):
    print(f"{idx:2d}. {name}")

9. Empirical Results & Underfitting Analysis

Model optimization depth directly influences output coherence. Training for insufficient iterations leaves the network underfitted, outputting unstructured noise.

A/B Experimental Comparison

Diagnostic Observations

1. At Loss $\approx 4.89$: The model assigns uniform probabilities across all vocabulary characters. Categorical sampling acts as uniform random noise.
2. At Loss $\approx 1.56$: The model captures common bi-gram and tri-gram groupings (e.g., sh, an, singh).
3. At Loss $\approx 1.40$: The network accurately encodes long-range phonetic structures, valid name endings, and space-separated surname prefixes (pal singh, kumar, raj).

Summary and Key Takeaways

• Character-Level vs. Token-Level Modeling: Character-level models tokenize text into individual letters rather than full subwords, operating over a small vocabulary ($V = 88$) while requiring longer sequence context windows.
• Feature Cleaning Mandate: Eliminating corrupted or non-string values before vocabulary extraction prevents out-of-vocabulary exceptions during processing.
• Sliding Window Tokenization: Context blocks of size $N=3$ split names into input sequences mapped to the immediate next target character, scaling a dataset of 14,821 names into 163,500 training samples.
• Dense Continuous Embeddings: Integer character indices are mapped into a low-dimensional continuous space ($d=5$), removing arbitrary integer scaling assumptions and allowing the model to learn semantic relationships between characters.
• Batch Normalization Dynamics: Inserting BatchNorm1d layers between linear transformations zero-centers activations across mini-batches, smoothing loss trajectories and preventing vanishing/exploding gradients during backpropagation.
• Cross-Entropy & Softmax Alignment: F.cross_entropy converts raw unnormalized output logits into target probabilities, calculating the loss against ground-truth index targets efficiently.
• Multinomial Sampling Mechanics: Rather than greedily selecting the single highest-probability token, torch.multinomial samples proportional to output probabilities, yielding diverse, creative, and novel text generation.
• Direct Loss-to-Quality Correlation: Training loss serves as a direct proxy for text synthesis quality; high loss ($\approx 4.89$) outputs noise, while optimized loss ($\approx 1.40$) generates structurally sound, domain-accurate output names.