Selecting the Right LLM for Your AI App: Running Custom Model Evals

Model selection is a foundational architectural decision in AI engineering. While public benchmarks offer broad rankings across general reasoning and coding, they fail to answer the critical production question: Which model performs best on my proprietary schema, unique data distributions, and strict latency/cost ceilings?

This guide details the complete end-to-end engineering methodology for running custom model evaluations. Using a production case study—building an automated "Text-to-SQL" feature for a real-time sports portal like ESPN Cricinfo—we demonstrate how to systematically capture engineering constraints, filter candidates via aggregated leaderboards, design a robust golden dataset, execute parallel model queries via an unified gateway, and programmatically validate generated SQL outputs using semantic table-matching heuristics.

1. The Model Selection Problem: Benchmarks vs. Custom Evals

Public leaderboards (such as LMSYS Chatbot Arena, HumanEval, and SWE-bench) evaluate models on standardized, generic datasets. Selecting a production model purely based on public rankings is an anti-pattern.

Public LLM Benchmarks (Generic Datasets)
   |
   | Filters candidate universe down to top performers
   v
Candidate Shortlist (5-10 Models)
   |
   | Evaluated on domain-specific schema, queries, & edge cases
   v
Custom Model Evaluations (Golden Dataset)
   |
   | Produces deterministic accuracy, latency, & ROI metrics
   v
Production Model Selection (Defensible, data-backed decision)

• Generic Benchmarks: High-level proxies for reasoning, coding, and instruction following. They cannot capture domain-specific edge cases, foreign schema conventions, or exact database constraints.
• Custom Model Evaluations (Evals): Running a curated set of internal queries against candidate models, executing their outputs against a target database, and objectively scoring their accuracy against known ground-truth outputs.
• The Core Decision Rule: Benchmarks are for filtering; custom evals are for selection.

2. Production Case Study: Ask Cricinfo (Text-to-SQL)

The Problem Statement

During high-traffic sporting events (e.g., India vs. Pakistan, IPL matches), hundreds of thousands of concurrent users submit statistical queries via text:

• "What is Virat Kohli's batting average against Pakistan in T20s?"
• "Which bowler has the best economy rate in the death overs with a minimum of 500 balls delivered?"

Historically, human data analysts manually wrote SQL queries against production analytical databases to surface answers back to live commentary feeds. This manual workflow collapsed under traffic spikes (e.g., 500 queries in 5 minutes).

The Architectural Evolution: From Manual Analytics to Text-to-SQL

To scale real-time user retention and ad views, the engineering team designed an automated Text-to-SQL engine:

User Query: "Best economy rate with >= 500 legal balls?"
   |
   | [Step 1] Injected into System Prompt with Database Schema
   v
LLM (Text-to-SQL Brain)
   |
   | [Step 2] Generates pure SQLite syntax (no conversational markdown)
   v
Database Engine (Executes generated SQL)
   |
   | [Step 3] Returns raw tabular Result Set
   v
User Interface / Commentary Feed (Renders statistical card)

Note: This architecture is not a traditional Retrieval-Augmented Generation (RAG) pipeline. The system does not retrieve external unstructured documents dynamically; rather, it injects a fixed, known relational schema into the context window for deterministic structured query generation.

3. Step 1: Defining Engineering Constraints & Unit Economics

Before opening leaderboards, an AI engineering team must mathematically establish strict architectural and budgetary guardrails.

A. Token Volume & Query Estimation

• System Prompt & Schema Definition: $\approx 400\text{ tokens}$
• Generated SQL Output: $\approx 100\text{ tokens}$
• Input-to-Output Ratio: $4:1$
• Expected Traffic: $50,000\text{ queries/day}$ (spiky distribution centered on match days).
• Exchange Rate: $1\text{ USD} = 95\text{ INR}$

B. Mathematical Unit Economics: The Uncached Naive Approach

Let $C_{\text{input}}$ be the input cost per million tokens, and $C_{\text{output}}$ be the output cost per million tokens. The per-query cost $C_{\text{query}}$ is:

The monthly cost in INR ($C_{\text{month}}$) across 30 days is:

Evaluating a hypothetical frontier tier (e.g., $C_{\text{input}} = \char36 10/\text{M}$, $C_{\text{output}} = \char36 50/\text{M}$):

Budget Verdict: A cost of $\approx 12.82\text{ Lakhs INR}$ exceeds a $3\text{ Lakh INR}$ monthly ceiling by more than $4\times$. Frontier models at this price tier are instantly disqualified unless aggressive caching is introduced.

C. Prompt Caching Economics (KV Cache Reuse)

Because the database schema ($\approx 350\text{ tokens}$) remains static across all 50,000 queries, providers supporting prefix caching (reusing Key-Value activation vectors from transformer attention layers) drastically alter the cost curve:

• Cache Write / Refresh: Higher initial cost for the initial prompt (e.g., $\char36 12.50/\text{M}$ or $\char36 20/\text{M}$).
• Cache Read (Hit): Up to $90\%$ discount (e.g., $\char36 1.00/\text{M}$) for subsequent queries landing within the cache TTL window (5-minute to 1-hour cycles).
• Under continuous production loads (50,000 requests/day), cache hit rates approach $99\%$, bringing expensive models down from $\text{INR } 12.8\text{ Lakhs}$ to $\text{INR } 6.7\text{ Lakhs}$ or lower.

D. Operational Constraints Matrix

4. Step 2: Filtering Candidates via Aggregated Leaderboards

With constraints codified, we filter a broad pool of candidate models ($> 140\text{ models}$) down to a viable shortlist.

Evaluating SQL-Specific Leaderboards

Direct domain leaderboards often suffer from critical operational defects:

• BIRD-SQL / Spider: Frequently lack modern frontier models, or test complex agentic multi-pass harnesses rather than base model completion capability.
• Outdated Registries: Many benchmarks fail to update within weeks of rapid open-weight and closed-source model releases.

Engineering Solution: Use coding benchmark aggregations (such as LLMstats or LMSYS Coding) as an operational alias for Text-to-SQL proficiency. Writing clean, syntax-valid SQL with complex joins, window functions, and aggregations maps directly to programmatic code synthesis capabilities.

Blended Price Normalization

Many tracking services publish a single blended price calculated on a weighted token ratio. For a $4:1$ blended price:

Because our exact schema-to-SQL token generation profile is $400\text{ input}:100\text{ output}$ (precisely $4:1$), the leaderboard blended price perfectly matches our unit economic footprint.

Candidate Shortlisting Heuristic

To rank models algorithmically within our budgetary cap, apply Min-Max normalization across coding benchmarks and throughput (characters/second), applying a heavy weight to coding quality:

1. Eliminate models exceeding the cost threshold:

2. Normalize Coding Rating and Throughput Score:

3. Composite Weighted Ranking:

Why a 90/10 Split? In Text-to-SQL, output completions are tiny ($\approx 100\text{ tokens}$). Even a slow model generating at 20 tokens/sec will complete output generation in well under 2 seconds. Correctness in syntax and join logic dramatically outweighs raw streaming throughput.

The Evaluated Top Shortlist

From this algorithmic filter, five distinct candidates spanning closed frontier models and open-weight alternatives were selected for empirical testing:

1. GPT-5.6 Tera (OpenAI frontier, high base capability)
2. KMI K3 / Moonshot 2.7T (Open-weight reasoning architecture)
3. Grok 4.5 (xAI high-throughput model)
4. Claude 3.5 / 5 Sonnet (Anthropic coding specialist)
5. MiniMax M3 (Cost-effective open-weight contender)

5. Step 3: Golden Dataset Curation & Database Setup

A custom evaluation is only as robust as its ground truth.

Database Ingestion (db.py)

Historical IPL data (2020–2024) is loaded into an embedded SQLite database featuring two core tables:

• matches: Metadata including season, venue, toss decision, match outcome.
• deliveries: Ball-by-ball tracking ($240\text{ deliveries per match}$) recording batter, bowler, runs off bat, extras, dismissals, and fielding actions.

Schema Extraction

The LLM requires exact schema awareness to avoid hallucinating non-existent column names. An automated utility extracts DDL statements and formats them for the system prompt:

CREATE TABLE matches (
    id INTEGER PRIMARY KEY,
    season INTEGER,
    city TEXT,
    date TEXT,
    team1 TEXT,
    team2 TEXT,
    toss_winner TEXT,
    toss_decision TEXT,
    result TEXT,
    winner TEXT
);

CREATE TABLE deliveries (
    match_id INTEGER,
    inning INTEGER,
    batting_team TEXT,
    bowling_team TEXT,
    over INTEGER,
    ball INTEGER,
    batter TEXT,
    bowler TEXT,
    batsman_runs INTEGER,
    extra_runs INTEGER,
    total_runs INTEGER,
    is_wicket INTEGER,
    dismissal_kind TEXT,
    player_dismissed TEXT,
    FOREIGN KEY (match_id) REFERENCES matches(id)
);

Golden Dataset Design Rules

1. Stratified Complexity: Distribute test cases across simple queries, intermediate aggregations, and brutal queries requiring multi-table JOINs, subqueries, and window functions (DENSE_RANK()).
2. Deterministic SQL Verification: Prior to evaluation, an internal script (golden_dataset_generator.py) runs every ground-truth query against the target database to ensure syntax validity and non-empty result sets.
3. Metadata Enrichment: Track query-level attributes such as:
   • order_sensitive (True/False): Dictates whether an ORDER BY ... LIMIT clause is mandatory.
   • expected_rows: The expected row count of the resulting record set.

6. Step 4: The Evaluation Architecture & Table-Matching Logic

Why can we not simply evaluate SQL queries using exact string matching or Levenshtein distance?

The String-Matching Fallacy: Relational algebra allows dozens of distinct, syntactically disparate SQL queries to yield the exact same relational result set. For instance, WHERE a = 1 AND b = 2 is functionally identical to WHERE b = 2 AND a = 1. A subquery can be equivalently written as an INNER JOIN. Exact string matching causes massive false-negative evaluation failures.

Semantic Table Matching Heuristics (evaluator.py)

The evaluation engine compares DF_gold and DF_cand across multiple deterministic phases:

1. Syntax & Execution Safety:
   • If candidate SQL execution throws an operational syntax or schema exception, the test case instantly records a SQL_ERROR.
2. Dimension Conformance:
   • If $\text{shape}(\text{DF}_{\text{gold}}) \ne \text{shape}(\text{DF}_{\text{cand}})$, fail immediately. If the ground truth expects 5 top players and the candidate returns 3, the query is functionally incorrect.
3. Value Normalization:
   • Cast floating-point values to standard precisions (e.g., matching 2.0 with 2, or rounding float differences below $\epsilon = 10^{-4}$).
   • Strip leading/trailing whitespaces, casing differences, and datetime format variations.
4. Order-Sensitive vs. Set-Based Equality:
   • If order_sensitive == False: Sort both dataframes row-by-row on identical column axes, converting each dataframe into a canonical list of sorted tuples before equality assertions.
   • If order_sensitive == True: Strictly preserve execution sequence; index matching must be $1:1$ without row sorting.

7. Step 5: Implementation: Code Walkthrough

The evaluation harness uses a modular Python architecture driven by LangChain and OpenRouter to hit heterogeneous provider endpoints via a unified client.

A. The Evaluation Runner (main.py)

import os
import sqlite3
import pandas as pd
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from evaluator import evaluate_dataframes

load_dotenv()

OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")

MODELS = [
    ("GPT-5.6-Tera", "openai/gpt-5.6-tera"),
    ("KMI-K3", "moonshot/kmi-k3"),
    ("Grok-4.5", "x-ai/grok-4.5"),
    ("Claude-Sonnet-3.5", "anthropic/claude-3.5-sonnet"),
    ("MiniMax-M3", "minimax/minimax-m3")
]

def load_context():
    with open("schema.sql", "r") as f:
        schema = f.read()
    golden_df = pd.read_csv("golden_dataset.csv")
    return schema, golden_df

def run_evaluation():
    schema, golden_df = load_context()
    db_conn = sqlite3.connect("ipl.db")
    results_summary = {}

    for model_name, model_slug in MODELS:
        print(f"\n================ Running Evaluation: {model_name} ================")
        llm = ChatOpenAI(
            model=model_slug,
            openai_api_key=OPENROUTER_API_KEY,
            openai_api_base=OPENROUTER_BASE_URL,
            temperature=0.0
        )
        
        passes = 0
        total = len(golden_df)

        for idx, row in golden_df.iterrows():
            question = row["question"]
            gold_sql = row["golden_sql"]
            order_sensitive = bool(row["order_sensitive"])

            system_prompt = (
                "You are an expert Text-to-SQL generator for an SQLite database.\n"
                "Return ONLY raw SQL. Do not include markdown blocks, explanations, or quotes.\n"
                f"Database Schema:\n{schema}"
            )
            
            try:
                # Inference
                response = llm.invoke([
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": question}
                ])
                generated_sql = response.content.strip().replace("```sql", "").replace("```", "")

                # Database Executions
                df_gold = pd.read_sql_query(gold_sql, db_conn)
                df_cand = pd.read_sql_query(generated_sql, db_conn)

                # Validation Heuristics
                is_match = evaluate_dataframes(df_gold, df_cand, order_sensitive=order_sensitive)
                
                if is_match:
                    passes += 1
                    print(f"Query {idx + 1:02d}: MATCH")
                else:
                    print(f"Query {idx + 1:02d}: MISMATCH")

            except Exception as e:
                print(f"Query {idx + 1:02d}: SQL_ERROR -> {str(e)[:40]}")

        accuracy = (passes / total) * 100
        results_summary[model_name] = accuracy
        print(f"--- Finished {model_name}: Accuracy = {accuracy:.2f}% ---")

    db_conn.close()
    return results_summary

if __name__ == "__main__":
    summary = run_evaluation()
    print("\nFINAL EVALUATION SUMMARY:", summary)

B. The Semantic Table Comparator (evaluator.py)

import pandas as pd
import numpy as np

def normalize_value(val):
    """Normalize floats, strings, and null representations."""
    if pd.isna(val):
        return None
    if isinstance(val, (float, np.floating)):
        return round(float(val), 3)
    if isinstance(val, (int, np.integer)):
        return float(val)
    return str(val).strip().lower()

def evaluate_dataframes(df_gold: pd.DataFrame, df_cand: pd.DataFrame, order_sensitive: bool = False) -> bool:
    """Compares two dataframes on values rather than query syntax."""
    # Check 1: Dimensional Shape Match
    if df_gold.shape != df_cand.shape:
        return False

    # Check 2: Normalize and Align Data Structures
    gold_clean = df_gold.copy()
    cand_clean = df_cand.copy()

    for col in gold_clean.columns:
        gold_clean[col] = gold_clean[col].apply(normalize_value)
    for col in cand_clean.columns:
        cand_clean[col] = cand_clean[col].apply(normalize_value)

    # Convert records to pure scalar lists
    gold_records = gold_clean.values.tolist()
    cand_records = cand_clean.values.tolist()

    # Check 3: Value Assertions based on Order Sensitivity
    if order_sensitive:
        return gold_records == cand_records
    else:
        # Sort rows canonically to achieve set-based order-independent equality
        gold_records.sort(key=lambda x: [str(i) for i in x])
        cand_records.sort(key=lambda x: [str(i) for i in x])
        return gold_records == cand_records

8. Empirical Results, Analysis & Production Realities

The evaluation suite was executed across the 20 brutal analytical SQL queries on the live IPL SQLite database.

Comprehensive Model Performance Synthesis

Architectural Insights & Production Pitfalls

• The Open-Weight / Reasoning Trap: Despite major market enthusiasm around KMI K3's 2.7 trillion parameters and general coding ranks, it failed catastrophically in production SQL execution (55% accuracy). Its internal reasoning loops inflated latency far beyond our 3.0-second SLA, and its SQLite syntax completions produced consecutive fatal parsing exceptions.
• Syntax Divergence in Regional Models: Non-Western models (KMI, MiniMax) demonstrated recurring syntax errors, suggesting that base pretraining distributions contained fewer standard SQLite dialects and more alternative relational or proprietary variants.
• The Top Contender Tradeoff (Grok 4.5 vs. Claude Sonnet 3.5):
  • Grok 4.5 achieved the highest raw accuracy ($90\%$) and the lowest cost profile ($\text{INR } 2.5\text{ Lakhs}$).
  • Claude Sonnet 3.5 finished closely behind at $85\%$, but achieved significantly superior token streaming speed and boasts historical enterprise API uptime and SDK stability.
• The Decision Framework: In production, raw benchmark numbers are combined with enterprise SLA guarantees. Teams requiring hardened operational stability often favor Anthropic (Sonnet 3.5) by allocating an incremental 10% budget margin, while teams prioritizing raw unit margins and raw correctness lean toward Grok 4.5.

9. Summary and Key Takeaways

• Benchmarks Filter, Evals Select: Never build a production architecture on generic benchmark scores. Use general coding leaderboards only to narrow hundreds of models down to a 5–10 candidate shortlist.
• Codify Hardware and Business Guardrails First: Establish your cost ceilings, latency thresholds, token volume, and privacy constraints mathematically before executing any technical testing.
• Exploit Prompt Caching Economics: For applications with static context (e.g., Text-to-SQL schema injection), KV caching discounts can slash input costs by up to $90\%$, making frontier models financially viable.
• Weight Latency Appropriately: Align evaluation weights with response size. For small completions like single SQL statements ($<100\text{ tokens}$), throughput matters far less than semantic query accuracy; heavily weight your scoring towards precision (e.g., a $90/10$ split).
• Reject String Matching for Code Evals: SQL queries must be evaluated via execution conformance. Compile and run both candidate and golden queries on a real database engine and assert equality on the normalized tabular result sets.
• Ground Truth Requires Stratified Quality: Construct golden datasets covering a realistic difficulty spectrum (simple lookups, multi-table JOINs, aggregations, window functions). Verify ground-truth SQL syntax programmatically before running evals.
• Beware the "Reasoning Model" Latency Penalty: Massive parameter architectures with built-in reasoning mechanisms often introduce severe latency stalls and dial back syntax reliability on lightweight, deterministic queries.
• Iterate via Universal API Gateways: Standardize eval harnesses using multi-provider interfaces (e.g., OpenRouter, LiteLLM) to evaluate multiple proprietary and open-weight models rapidly without writing bespoke API integration clients.