Chapter 26

Evaluation

Evaluation, the discipline that turns intuition about LLM quality into measurement. From the operational definition of an eval (three flavors, standard benchmarks, open-ended evaluation, eval design), through capability benchmarks (MMLU, HumanEval, HellaSwag, GPQA, MATH), safety benchmarks (HarmBench, TruthfulQA, ToxiGen, WMDP), agentic benchmarks (SWE-bench, GAIA, OSWorld), LLM-as-judge methodology and its biases (Zheng 2023), eval failure modes (saturation, contamination, reward hacking, Goodhart's law), and designing new evals. Closes Part VIII's discipline arc, safety, interpretability, evaluation, and bridges to Part IX (Agents), the curriculum's final arc.

Chapter 24 covered safety from the outside, alignment training, red-teaming, what we want the model to do. Chapter 25 covered interpretability from the inside, probes, sparse autoencoders, circuits, what the model is actually computing. This chapter covers the third discipline of Part VIII: how do we measure both? Evaluation turns “this model is better” into a measurable claim. Without it, AI development is opinion and marketing. With it, claims become comparable, across models, across techniques, across releases. It is the bridge from craft to engineering.

Modern AI evaluation is three flavors. Standard benchmarks: fixed test sets with known answers (MMLU, HumanEval, HellaSwag, GPQA, MATH). Open-ended evaluation, LLM-as-judge methodology (MT-Bench), crowdsourced human preferences (Chatbot Arena), structured rubrics. Eval design: making new evaluations for new capabilities (SWE-bench for agentic coding, GAIA for general AI assistants, OSWorld for computer use). This chapter covers all three, plus their failure modes, saturation, contamination, reward hacking, Goodhart’s law, and how production teams design their own evals.

The discipline is not a solved science. Benchmarks saturate within $12$ – $36$ months. Major benchmarks (GSM8K, MATH, MMLU, HumanEval) have documented contamination. Models routinely reward-hack benchmarks they are optimized against. Modern AI eval is a dashboard, not a number. By the end of this chapter you should know which benchmarks exist, what they measure, how they fail, and how to design your own. Then Part IX opens with agents, ReAct foundations, agents from scratch, multi-agent orchestration, agent-eval frameworks, the curriculum’s final arc.

What is an evaluation?

The operational definition

What eval teams actually do, stripped of jargon, is four steps. Define what to measure: a specific capability, safety property, alignment claim, or utility signal. Build or choose a test set: a fixed benchmark, a dynamic arena, a human-evaluation protocol. Run models against it and compute a score. Compare across models to claim improvement, equivalence, or regression. The discipline is what fills in each of those steps rigorously.

Three flavors

Modern AI evaluation decomposes operationally into three families that share methodology but answer different questions.

$\text{standard benchmarks} \;\;\Vert\;\; \text{open-ended evaluation} \;\;\Vert\;\; \text{eval design}$

(26.eval-flavors)

Standard benchmarks: fixed test sets with known answers; MMLU, HumanEval, HarmBench are the canonical examples. Open-ended evaluation: LLM-as-judge, human evaluation, dynamic arenas where there is no single right answer; MT-Bench and Chatbot Arena are the references. Eval design: making new evaluations for new capabilities, as the field did with SWE-bench for agentic coding, GAIA for general AI assistants, WMDP for dangerous-knowledge proxy.

Why evaluation is hard, in five points. Calibration: a score of $75\%$ does not mean the model is ” $75\%$ capable;” the meaning depends entirely on the distribution. Saturation: when models max out a benchmark, it stops being informative. Contamination: if test data appeared in pretraining, scores inflate without anyone necessarily noticing. Goodhart’s law: when a metric becomes a target, it ceases to be a good metric. Distribution shift: benchmark distribution is rarely the deployment distribution.

The empirical scale of the field in early $2025$ is worth keeping in mind. Frontier model release cycles: $6$ to $12$ months. Active benchmarks: roughly $50$ to $100$ widely cited, with hundreds more domain-specific. Saturation pace: a hard benchmark can saturate in $12$ to $36$ months from release. Cost of running: $1{,}000$ to $100{,}000$ inferences per model per benchmark, often thousands of dollars per evaluation pass.

Capability benchmarks

The landscape

The capability-benchmark landscape in early $2025$ is built around a handful of names every working engineer should recognize.

Benchmark	Released	What it measures	Saturation
MMLU	$2020$	$57$ subjects, multiple choice	Mostly saturated (~ $88$ – $90\%$ )
HumanEval	$2021$	Python code generation	Saturated (~ $95\%$ +)
HellaSwag	$2019$	Common-sense reasoning	Saturated (~ $95\%$ +)
MATH	$2021$	Competition math	Saturated (~ $95\%$ +)
GPQA	$2023$	Graduate-level science	Active (~ $55$ – $70\%$ frontier)
BBH	$2022$	$23$ hard tasks	Mostly saturated
MMLU-Pro	$2024$	Harder MMLU variant	Active

The pattern is consistent. Most benchmarks saturate in $12$ to $36$ months. GPQA (Rein $2023$ ) is the modern hard benchmark; new ones are released as old ones max out. The cycle is now structural, frontier capability outruns benchmark design, and the response is another benchmark rather than a new capability claim. AIME-style competition math (repurposing the American Invitational Mathematics Examination as a held-out benchmark) is the next rung after GPQA and MATH: as reasoning models pushed those two toward saturation, AIME-derived evals, alongside newer hard-frontier sets like FrontierMath and Humanity’s Last Exam, became the benchmarks a working engineer should expect to see cited next. This chapter deliberately doesn’t quote current scores on them, the exact numbers age out within a release cycle or two; treat the benchmarks themselves as durable and any specific percentage attached to them as a snapshot.

What capability benchmarks measure well is narrow but real. Knowledge breadth: MMLU spans $57$ subjects across STEM, humanities, social sciences. Code synthesis: HumanEval is a clean signal of “can produce working code under hidden test cases.” Math reasoning: MATH stress-tests step-by-step computation on competition problems. Logical and scientific reasoning: GPQA pushes graduate-level questions that are “Google-proof,” meaning humans with web access still score around $34\%$ .

What they measure poorly is also real. Open-ended generation quality: no clean metric exists. Conversational ability: single-turn evals miss conversational drift across turns. Long-context reasoning: most benchmarks are short. Real-world tasks: everything in a benchmark is a stylized version of work, not work itself.

Methodology

Three formats carry most capability benchmarks. Multi-choice format (MMLU, GPQA): easy to score, fast to run, vulnerable to surface heuristics like “always pick C.” Open-ended format with hidden tests (HumanEval, MATH): test-case verification gives a cleaner signal but costs more compute. Chain-of-thought variants: most modern benchmarks now report CoT-allowed and CoT-restricted scores separately because the gap is large enough to matter.

Pass@1 and pass@k

Hidden-test benchmarks like HumanEval need a scoring convention beyond exact match, there’s no single “correct” program, only programs that pass or fail the hidden unit tests. The convention is pass@ $k$ : sample $k$ candidate completions per problem, and count the problem solved if at least one of the $k$ passes every hidden test. pass@ $1$ is the special case, a single sample (typically greedy-decoded) either passes or it doesn’t; it’s the number frontier labs quote for headline capability claims because it approximates what one real request gets.

Naively estimating pass@ $k$ by drawing exactly $k$ samples and checking whether any pass is a high-variance, biased estimator, small $k$ swings wildly between runs on the same model. Chen 2021 (arxiv.org/abs/2107.03374, the HumanEval paper) fixes this with an unbiased estimator: draw $n \geq k$ samples per problem, count $c$ correct, and compute the probability that a random size- $k$ subset of those $n$ samples contains at least one correct one.

$\text{pass@}k = \E_{\text{problems}}\left[1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}\right]$

(26.pass-at-k)

The subtracted term, $\binom{n-c}{k} / \binom{n}{k}$ , is the probability that a $k$ -sized subset drawn from the $n$ samples contains zero correct ones; $1$ minus that is the probability at least one succeeds. With $n = 5$ samples per problem and $c = 3$ correct, pass@ $1 = 3/5 = 60\%$ , pass@ $2 = 90\%$ , pass@ $3 = 100\%$ , using more of the $n$ samples raises the estimate because more attempts means more chances to find a correct one, and the estimator uses all $n$ samples rather than throwing away information by resampling $k$ at a time. pass@ $k$ answers a different question than pass@ $1$ : “can the model solve this given $k$ generous attempts,” not “does the model reliably solve this on the first try.” The gap between the two is itself informative, a model with high pass@ $100$ but low pass@ $1$ knows how to produce the solution somewhere in its sampling distribution but doesn’t reliably surface it unprompted.

Score interpretation

A bare score is almost never enough. Random baseline for a four-choice multiple-choice question is $25\%$ . Frontier model expectation as of $2025$ is $60$ to $90\%$ on active hard benchmarks, $90\%$ + on saturated ones. Human expert ceiling is $80$ to $95\%$ on most benchmarks, with humans rarely scoring $100\%$ even on tasks they designed. Quoting a single number without these reference points gives no information about how to compare.

Benchmark heatmap

8 models · 10 benchmarks · sort by any column

Note: Scores are illustrative ballparks drawn from mid-2024 to 2025 reports. They're chosen for pedagogical clarity, not authoritative reproduction of any specific leaderboard. Real benchmark rankings shift with each model release.

Sort by:

	capability (5)					safety (2)		agentic (3)
Model	MMLU	HumEv	HSwagSAT	GPQA	MATH	TruQA	HarmB	SWE-V	GAIA	OS
Claude 3.5 Sonnet	89	92	96	59	78	65	92	49	64	39
GPT-4o	88	90	95	53	76	60	86	33	60	32
Claude 3 Opus	86	85	95	50	61	64	91	23	51	24
GPT-4 Turbo	86	87	95	48	73	59	84	22	50	22
Gemini 1.5 Pro	85	84	93	47	67	58	81	19	47	21
Llama 3.1 405B	88	89	94	51	73	56	76	17	38	14
Llama 3.1 70B	83	80	93	41	68	53	73	12	30	9
GPT-3.5 Turbo	70	73	85	28	36	47	79	1	6	3

Hover a cell for detail · click a column header to sort

Try sorting by different columns. The ranking changes substantially depending on what you measure. Saturated benchmarks (marked SAT) have a whole column clustered near the top, they no longer distinguish frontier models. Only HellaSwag earns the badge in this snapshot; MMLU, HumanEval, and MATH are trending the same way but don't yet clear it. Agentic benchmarks (SWE-bench, GAIA, OSWorld) show the most spread; they're where models still discriminate, and where Part IX's coverage matters most. This is the chapter's central claim made visceral: modern AI eval is a dashboard, not a number.

Eight frontier models × ten benchmarks (capability + safety + agentic) in a sortable color-coded grid. Click any column header to re-rank models by that benchmark. A saturated column (SAT badge) clusters scores near the top, it no longer discriminates; in this snapshot only HellaSwag clears that bar, though MMLU, HumanEval, and MATH are trending the same way. Sorting by capability vs agentic gives very different rankings. Scores are illustrative 2024-2025 ballparks; the widget displays an explicit disclaimer. The chapter's central claim made visceral: modern AI eval is a dashboard, not a number.

# A minimal benchmark scoring harness.
# Real benchmarks: thousands of items, more nuanced scoring.
# The harness pattern (run + score + summarize) is the same at any scale.

TEST_ITEMS = [
    {'q': 'What is 2 + 2?',                   'category': 'math',    'answer': '4'},
    {'q': 'What is 5 * 7?',                   'category': 'math',    'answer': '35'},
    {'q': 'What is sqrt(144)?',               'category': 'math',    'answer': '12'},
    {'q': 'When was the Declaration signed?', 'category': 'history', 'answer': '1776'},
    {'q': 'First US president?',              'category': 'history', 'answer': 'George Washington'},
    {'q': 'What year did WWII end?',          'category': 'history', 'answer': '1945'},
]


def mock_model(question):
    """Pretend this is a real model API call."""
    answers = {
        '2 + 2':       '4',
        '5 * 7':       '35',
        'sqrt(144)':   '12',
        'Declaration': '1776',
        'First US':    'George Washington',
        'WWII end':    '1944',                     # wrong!
    }
    for keyword, answer in answers.items():
        if keyword in question:
            return answer
    return '???'


def run_benchmark(items, model_fn):
    results = []
    for item in items:
        pred = model_fn(item['q'])
        correct = pred.strip().lower() == item['answer'].strip().lower()
        results.append({**item, 'predicted': pred, 'correct': correct})
    return results


def summarize(results):
    n = len(results)
    correct = sum(r['correct'] for r in results)
    by_category = {}
    for cat in set(r['category'] for r in results):
        cat_results = [r for r in results if r['category'] == cat]
        by_category[cat] = sum(r['correct'] for r in cat_results) / len(cat_results)
    return {'overall': correct / n, 'by_category': by_category, 'n': n, 'correct': correct}


results = run_benchmark(TEST_ITEMS, mock_model)
summary = summarize(results)

print(f"Benchmark results")
print(f"  Overall: {summary['correct']}/{summary['n']} = {summary['overall']:.0%}")
print(f"  By category:")
for cat, acc in summary['by_category'].items():
    print(f"    {cat:>10}:  {acc:.0%}")

# Observations:
# - Even a tiny benchmark surfaces per-category variation (math vs history)
# - Real benchmarks: thousands of items, paraphrase-tolerant scoring
# - Per-category breakdown beats overall accuracy: never compare on one number

Safety benchmarks

The landscape

The safety-benchmark landscape is small and concentrated. TruthfulQA (Lin $2021$ ), $817$ questions on common falsehoods; measures whether models repeat them. BBQ (Parrish 2022, arxiv.org/abs/2110.08193), a hand-built bias benchmark that asks questions under ambiguous versus disambiguated context and checks whether the model’s answer relies on a social stereotype when the context doesn’t actually support one; it’s the reference eval for the specific failure mode of “confidently guessing the stereotype-consistent answer instead of admitting the context is ambiguous.” HarmBench (Mazeika $2024$ ), jailbreak success rates across categories of harm; the standardized red-teaming reference. JailbreakBench, another standardized jailbreak eval; allows controlled comparisons across attacks and defenses. ToxiGen, toxic-versus-benign language classification with adversarial coverage. WMDP (Hendrycks $2024$ ), a proxy benchmark for dangerous CBRN knowledge, used by frontier labs to test whether models retain capabilities they should not have.

What safety benchmarks measure well is direct and useful. Direct harm, HarmBench asks whether the model produces harmful content when prompted. Truthfulness, TruthfulQA asks whether the model gives correct answers rather than popular ones. Dangerous-capability proxy, WMDP asks whether the model could help with dangerous tasks even if it would refuse.

Calibrated refusal evaluation

What safety benchmarks measure poorly matters as much. Calibrated refusals, over-refusal versus under-refusal need separate evals; Chapter 24 covered why a model that refuses everything is as broken as one that refuses nothing. Novel jailbreaks: by definition, a benchmark covers known patterns, not the next attack. Multi-turn manipulation, most safety benchmarks are single-turn, but real attacks span conversations. Long-tail failures, rare-but-severe failures don’t show up in aggregate scores; you have to look for them.

The connection back to Chapter 24 is structural. Safety benchmarks are necessary but not sufficient for deployment decisions. They are one signal; red-teaming, interpretability monitoring (Chapter 25), and post-deployment review fill out the picture. A team shipping a model on safety scores alone is shipping on partial evidence.

Agentic benchmarks

The new frontier

Part VII covered agentic capabilities, tool use (Ch 21), retrieval (Ch 22), multimodal grounding (Ch 23). Part VIII’s eval discipline needs benchmarks for those capabilities, and the field has been producing them on a roughly yearly cadence since $2023$ . Agentic benchmarks are the new frontier of evaluation because they are the new frontier of capability.

The landscape

The names worth knowing track the deployment archetypes engineers actually build for. $\tau$ -bench (Yao 2024, arxiv.org/abs/2406.12045), simulated multi-turn conversations between an agent and a user (itself an LLM), where the agent has to use domain-specific tools correctly and follow domain policy, retail and airline customer-service scenarios are the reference domains; even strong function-calling agents solve under $50\%$ of tasks, and the paper’s $\text{pass}^k$ metric (does the agent succeed on all of $k$ independent trials of the same task) exposes how inconsistent agents are even when they’re technically capable of a task. SWE-bench (Jimenez $2023$ ), real GitHub issues with hidden tests; produce a patch that passes; the reference agentic-coding benchmark. GAIA (Mialon $2023$ ), multi-step real-world tasks that require web browsing, tool use, and reasoning; humans score about $92\%$ , frontier agents have moved from under $30\%$ in $2023$ to $60$ – $75\%$ in $2024$ – $25$ . OSWorld (Xie $2024$ ), computer-use tasks with real desktop applications; $369$ tasks across browsers, file systems, productivity apps. MLE-bench (Chan $2024$ ), $75$ Kaggle competitions where agents produce code and submit predictions. WebArena, web-browsing agent tasks. Cybench, cybersecurity capture-the-flag tasks. Upwork HAPI, real freelance tasks across thousands of categories, illustrative of the kind of applied evaluation production teams build for their own use cases.

What agentic benchmarks measure well is exactly what production teams care about. End-to-end task completion, did the agent solve the task, not just produce plausible intermediate steps? Tool-use chains, can the agent compose multiple tools across multiple turns? Resilience to environmental noise: does the agent recover from tool errors, transient failures, partial information? Real-world relevance, the tasks look like actual work rather than stylized abstractions.

What they measure poorly is also real and worth tracking. Cost, each evaluation run is expensive; agents take many turns, each turn is an inference, frontier agentic evals can run hundreds of dollars per attempt. Reproducibility, real environments change; tools fail; flaky tests show up at the benchmark level too. Specific failure modes: knowing a SWE-bench score does not tell you what is failing without instrumentation on top of the score.

Trajectory

The trajectory is fast and visible. $2023$ , most agents scored under $30\%$ on GAIA. $2024$ , $50$ to $65\%$ with frontier models and good scaffolding became routine. $2025$ , $70\%$ + is achievable; GAIA may saturate within one or two years. SWE-bench has moved from roughly $30\%$ to $65\%$ on the same window for the same reason.

Why this matters for engineers building products: as classic benchmarks saturate, agentic benchmarks become the discriminator between frontier models. A team picking models on MMLU in $2025$ is choosing on a saturated metric; the same team picking on SWE-bench or GAIA is picking on the metric that still moves.

LLM-as-judge

The methodology

LLM-as-judge (Zheng $2023$ ) is the most practically-relevant evaluation method for engineers shipping chat products today. The three-step recipe is direct: run two models on the same prompts, have a stronger model judge which response is better, and aggregate judgments into win rates, Elo ratings, or pairwise scores. The reference implementations are MT-Bench ( $80$ multi-turn questions across $8$ categories; GPT- $4$ judges) and Chatbot Arena (crowdsourced humans judge anonymized battles; Elo ratings updated continuously, over a million votes by $2025$ ). Both of those are pairwise, the judge picks a winner between two responses. G-Eval (Liu 2023, arxiv.org/abs/2303.16634) is the reference absolute-scoring alternative: one response at a time, no comparison, scored against a rubric (e.g. $1$ – $5$ on coherence, consistency, fluency) using chain-of-thought reasoning and the judge model’s own output-token probabilities to weight the final score. Pairwise scoring is more reliable when two responses are close in quality, relative judgments are easier than absolute ones, but it needs a second response to compare against. Absolute scoring is what you reach for when there’s nothing to pair against, a single deployed model’s output graded against a fixed rubric.

Bias modes

LLM judges have documented failure modes, all from Zheng $2023$ and confirmed in subsequent work.

Position bias: judges systematically favor the first response shown. The effect is large enough that flipping order can flip the verdict on a substantial fraction of items. Verbosity bias: judges prefer longer responses, often regardless of whether the extra words add anything. Self-enhancement bias, a judge model prefers responses from its own family; GPT- $4$ judging GPT- $4$ -versus-Claude is not neutral. Coverage bias: judges miss subtle factual errors they do not themselves know; an open-ended factual question can be judged “correct” by an LLM that itself believes the same wrong answer.

The mitigations are equally well-documented. Randomize positions in pairwise comparisons to average out position bias. Multiple judges of different families to average out self-enhancement. Calibrate judges against human-labeled gold data on a sample of items. Reasoning judgments: have the judge explain its reasoning before giving a verdict; reduces some biases by forcing the judge to articulate what it is comparing.

The standard version of “randomize positions” is position-swap averaging: judge the pair in both orderings and only trust the verdict when the two agree.

$\text{verdict}(A, B) = \begin{cases} A & \text{judge}(A, B) = A \;\text{and}\; \text{judge}(B, A) = A \\ B & \text{judge}(A, B) = B \;\text{and}\; \text{judge}(B, A) = B \\ \text{tie} & \text{otherwise} \end{cases}$

(26.position-swap)

$\text{judge}(X, Y)$ is the raw verdict when $X$ is shown first; running it twice with the arguments swapped and requiring agreement is exactly what catches a judge that just favors whichever response it saw first, disagreement between the two orderings is the position-bias signal, and calling it a tie rather than picking one of the two verdicts is the conservative choice. The judge_with_swap function below implements (26.position-swap) directly.

Chatbot Arena

Chatbot Arena (Chiang $2024$ ) deserves its own beat because of its outsized influence. Crowdsourced humans pick the better of two anonymized model responses; Elo ratings update continuously. The closest thing to a “ground truth” capability ranking for chat-style use.

Elo rating, borrowed directly from chess, is the mechanism behind “continuously updated.” Every model starts at a baseline rating (Chatbot Arena uses $1000$ ), and each pairwise battle nudges both models’ ratings toward the observed outcome. The expected win probability is a logistic function of the rating gap between the two models:

$E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}, \qquad R_A \leftarrow R_A + K(S_A - E_A)$

(26.elo)

$R_A$ and $R_B$ are the two models’ current ratings, $E_A$ is model $A$ ‘s expected score against $B$ before the battle is decided, $S_A$ is the actual outcome ( $1$ for a win, $0$ for a loss, $0.5$ for a tie), and $K$ is a step-size constant, Chatbot Arena runs a small $K$ (on the order of $4$ ) so ratings move gradually and don’t whipsaw on a handful of votes. A model that wins battles it was “expected” to lose ( $E_A$ low, $S_A = 1$ ) gains rating quickly; a model that wins battles it was already favored to win barely moves, exactly the update rule that makes the number converge on a consistent ranking rather than just tallying wins. In production, LMSYS now fits ratings with a Bradley-Terry model over the full batch of votes rather than applying (26.elo) sequentially vote-by-vote, more stable at scale, but the expected-score logic is the same online-Elo idea underneath.

But Chatbot Arena is biased toward chat users’ preferences, does not measure agentic capability well, and the prompt distribution is what users ask, not what production deployments necessarily encounter. As of $2025$ it is the most-cited open leaderboard; treat it as one ranking among many rather than the ranking.

When to reach for LLM-as-judge in practice. Useful, open-ended generation, conversation quality, style comparisons, anything without a programmatic ground truth. Less useful: factual accuracy (judges can be wrong), math and code (use programmatic verification), safety (use specialized benchmarks rather than asking a judge to spot harm).

LLM-as-judge bias demo

5 scenarios · documented bias modes (Zheng 2023)

Pick a scenario:

POSITION BIAS

position bias

Prompt

In three sentences, explain why exercise improves mood.

Responses

AModel-A200 chars

Exercise releases endorphins, which lift mood. Regular activity also reduces cortisol over time, lowering stress. Finally, the sense of accomplishment from completing a workout supports self-efficacy.

BModel-B183 chars

Physical activity triggers endorphin release that produces euphoria. It also reduces stress hormones like cortisol. Completing exercise builds a sense of agency that improves outlook.

Judge verdicts under two orderings

A first, then B →

Response A

Response B

tie

B first, then A →

Response A

Response B

tie

Swap-mitigated verdict:TIE

✓ Mitigation catches

What's happening

Both responses are roughly equivalent in quality and content. The judge picks whichever is shown first, a documented position bias. With genuinely similar responses, position effects can dominate judgment.

Mitigation outcome

Swap-mitigation CATCHES this: when both orderings are tried, the verdicts disagree (A-first → A wins; B-first → B wins). The mitigated verdict is "tie."

Click through the scenarios. Position bias flips with ordering, and swap-mitigation catches it (verdict becomes "tie"). Verbosity bias, self-enhancement bias, and coverage bias are consistent across orderings: swap-mitigation does NOT help. Defending against these requires multi-judge ensembles, rubric-based judging, anonymization, or human calibration. No single mitigation defends against all bias modes: production LLM-as-judge requires defense-in-depth, like every other discipline of Part VIII.

Five scenarios demonstrating documented LLM-as-judge bias modes (Zheng 2023): position bias, verbosity bias, self-enhancement bias, coverage bias, and a clean case for contrast. Each scenario shows the judge's verdict under both orderings (A-first vs B-first) plus the swap-mitigated verdict. Position bias is caught by swap-mitigation; the others require additional defenses (multi-judge ensembles, rubric-based judging, anonymization, human calibration). Demonstrates that no single mitigation defends against all bias modes, production LLM-as-judge requires defense-in-depth.

# LLM-as-judge with position-bias mitigation via swapped-position re-judging.

def mock_judge(prompt, response_a, response_b):
    """
    Mock LLM-as-judge with documented verbosity bias.
    Real implementation: call GPT-4 or Claude with a judge prompt.
    """
    # SIMULATE verbosity bias: judge favors longer responses
    if len(response_a) > len(response_b) * 1.3:
        return 'A'
    if len(response_b) > len(response_a) * 1.3:
        return 'B'
    return 'tie'


def judge_with_swap(prompt, response_a, response_b):
    """
    Run the judge in both orderings. If verdicts disagree, return 'tie'.
    Mitigates position bias (Zheng 2023).
    """
    judgment_ab = mock_judge(prompt, response_a, response_b)
    judgment_ba = mock_judge(prompt, response_b, response_a)
    # Re-map judgment_ba: 'A' in the second call means response_b won → maps to 'B'
    if judgment_ba == 'A':
        judgment_ba = 'B'
    elif judgment_ba == 'B':
        judgment_ba = 'A'

    if judgment_ab == judgment_ba:
        return judgment_ab
    return 'tie'   # bias detected → call it a tie


test_cases = [
    {
        'prompt': 'Explain photosynthesis briefly.',
        'response_a': 'Plants use sunlight, CO2, and water to produce sugar and oxygen.',
        'response_b': ('Photosynthesis is the process by which green plants and certain other organisms '
                       'use sunlight to synthesize foods with the help of chlorophyll. The chemical '
                       'reaction involves carbon dioxide and water producing glucose and oxygen as a byproduct.'),
    },
    {
        'prompt': 'What is 12 * 11?',
        'response_a': '132',
        'response_b': '132',
    },
]

print("LLM-as-judge with swap mitigation:\n")
for case in test_cases:
    naive = mock_judge(case['prompt'], case['response_a'], case['response_b'])
    mitigated = judge_with_swap(case['prompt'], case['response_a'], case['response_b'])

    print(f"Prompt: {case['prompt']}")
    print(f"  A ({len(case['response_a']):>3} chars): {case['response_a'][:60]}...")
    print(f"  B ({len(case['response_b']):>3} chars): {case['response_b'][:60]}...")
    print(f"  Naive:     {naive}")
    print(f"  Mitigated: {mitigated}\n")

# Observations:
# - Verbosity bias: naive judge favors the longer response
# - Swap mitigation catches position bias (a different failure mode)
# - Verbosity bias needs rubric-based judging or human calibration
# - Real LLM-as-judge: multiple judges, calibration, explicit reasoning

Eval failure modes

Saturation

A benchmark stops being informative when models reach the ceiling. HumanEval was the code benchmark for years; now frontier models score $95\%$ + and small differences sit inside noise. The response is structural, build harder benchmarks as old ones max out, the way GPQA replaced MMLU’s hard tier. Saturation is not a sign that the capability is solved; it is a sign that the test no longer discriminates among models near the top.

Contamination

If benchmark data appeared in pretraining, scores inflate. Contamination has been documented across major benchmarks, GSM8K, MATH, MMLU, HumanEval all have known leakage at some level. Detection is checking whether a model can complete test items in ways that suggest memorization. Response is held-out evals, dynamic benchmarks that change over time, frequent rotation. The standard practice now is a private split created after a model’s training cutoff and never published, GPQA Diamond and GAIA’s private test split are the reference examples: labs evaluate against them precisely because the items can’t have leaked into pretraining data that predates their release. Publication itself is a contamination risk: once a benchmark is published, its items can show up in next-generation pretraining corpora, which is why held-out splits get regenerated rather than reused indefinitely.

Reward hacking and Goodhart

These are the same problem at different scales. Reward hacking: models score well by gaming the metric rather than solving the task. Example: code models that hard-code test cases instead of implementing the function. Example: math models that memorize problem-answer pairs from training data. Response, test on held-out items, look at qualitative output, use multiple metrics.

Goodhart’s law generalizes the pattern. When a benchmark becomes a target, for training, comparison, or marketing, the metric stops measuring the underlying capability. Example: optimizing directly for MMLU produces benchmark-tuned models that do not necessarily generalize beyond the benchmark distribution. Response, hold out unpublished evals, use surprise benchmarks, track multiple correlated metrics rather than one.

Distribution shift is another face of the same issue. A benchmark distribution is rarely the deployment distribution. A model that scores $90\%$ on MMLU may not generalize to your domain at all. Response, domain-specific evals, deployment metrics, user-feedback signals that catch what the benchmark does not.

The systemic implication

Add capability versus propensity from Chapter 24, benchmarks measure what a model can do; deployment depends on what it does. WMDP is the canonical example: it measures dangerous knowledge, not propensity to share. Add single-metric thinking, any one score hides important variation. Seven distinct failure modes, not one big confusion. Modern AI eval is a dashboard, not a number, and treating any single benchmark as the model’s overall quality is misleading regardless of which benchmark it is.

# Detect simple forms of reward hacking on a code benchmark:
# a model that "solves" by hard-coding test cases rather than implementing the function.

def looks_like_hardcoded(submission, test_cases):
    """
    Heuristic: does the submission appear to hard-code the test cases?
    Real detection: more nuanced (AST analysis, abstract interpretation).
    """
    hits = 0
    for tc in test_cases:
        input_str = str(tc['input'])
        output_str = str(tc['expected_output'])
        if input_str in submission and output_str in submission:
            hits += 1
    return hits / max(1, len(test_cases))


TEST_CASES = [
    {'input': 2,   'expected_output': 4},
    {'input': 5,   'expected_output': 10},
    {'input': 7,   'expected_output': 14},
    {'input': 100, 'expected_output': 200},
]

submissions = {
    'clean': '''
def double(x):
    return x * 2
''',
    'hardcoded': '''
def double(x):
    if x == 2: return 4
    if x == 5: return 10
    if x == 7: return 14
    if x == 100: return 200
    return None   # off-distribution: fails outside the test set
''',
}

print("Reward hacking detection:\n")
for name, sub in submissions.items():
    suspicion = looks_like_hardcoded(sub, TEST_CASES)
    flag = 'POSSIBLE HACKING' if suspicion > 0.5 else 'looks clean'
    print(f"{name}:")
    print(f"  Suspicion: {suspicion:.0%} of test cases appear in code")
    print(f"  Verdict: {flag}\n")

# Observations:
# - Hard-coding is the simplest reward-hacking pattern; real ones are subtler
# - Production code-eval frameworks add: hidden test cases; adversarial test generation;
#   static analysis; runtime probes
# - The principle generalizes: any time the metric is a target, models will optimize
#   for the metric (Goodhart's law): not for the underlying capability

Designing new evals

When you need a new eval

Public benchmarks rarely match what production needs to measure. New capability not covered: agentic coding, computer use, domain-specific reasoning. Domain-specific, medical Q&A, legal reasoning, financial summarization. Safety-critical area, child safety, election integrity, region-specific compliance. Internal product metrics: does our chatbot help users complete onboarding successfully, measured against the actual onboarding funnel? Every production AI team ends up building custom evals because the public ones do not capture their use case.

The design checklist

The skeleton of a well-designed eval is five questions.

What are you measuring? State the capability or property precisely. Distinguish from related metrics. Decide whether you are measuring capability (what the model can do), propensity (what it does do under realistic prompting), or both.

What’s the test format? Multi-choice is easy to score but vulnerable to surface heuristics. Open-ended with hidden tests is cleaner but needs verification logic. LLM-as-judge is flexible but needs bias mitigation. Human evaluation is the gold standard but is expensive and slow. Real-environment simulation is the most realistic and the most complex to maintain.

What’s the test set? Sample size large enough for statistical power. Difficulty distribution that spans easy through hard rather than clustering at one end. Representativeness, items match the deployment distribution. Held-out from training to prevent contamination, ideally never published.

What’s the calibration? Random baseline (chance accuracy). Human baseline (expert and non-expert performance). Current SOTA reference (where existing models score). Target threshold (what counts as “good”). Without all four, the raw score does not communicate anything useful.

How will you maintain it? Rotation policy to avoid contamination over time. Versioning so cross-version comparisons stay honest. Public-versus-private trade-off, transparency helps the field but invites contamination. Production teams typically keep an internal eval set unpublished alongside a smaller public reference set.

Production patterns reinforce the checklist. Hold-out test sets never get published; they are the contamination-free signal. Multi-judge ensembles for LLM-as-judge eval. Domain-specific evals sit alongside broad public benchmarks rather than replacing them. Live deployment metrics, real-world signals from production traffic, are the ultimate eval, and the only one that captures the actual deployment distribution.

Exercises

Four exercises that lock in the evaluation toolkit. Each is a self-contained problem with a starting template; hints are collapsed by default; try the problem first.

The exercises trace the chapter’s arc: build a benchmark scoring harness (Ex 1) → implement LLM-as-judge with multi-mitigation (Ex 2) → detect benchmark contamination (Ex 3) → design a custom eval for a specific domain (Ex 4).

Exercise 1 (easy): Benchmark scoring harness with per-category breakdown

Implement a small benchmark scoring harness that runs a model against test items, computes overall and per-category accuracy, and reports errors.

Hint

The harness pattern is the same at any scale:

Iterate test items
Call the model
Score each prediction (exact match, soft match, or LLM-judge)
Aggregate (overall accuracy + per-category)
Surface errors for inspection

Real benchmarks: thousands of items, more nuanced scoring (paraphrase tolerance, partial credit). The harness pattern is the same.

Solution

Loop, call, score, collect; then aggregate by filtering on category and on correct.

def run_benchmark(items, model_fn):
    results = []
    for item in items:
        predicted = model_fn(item['q'])
        correct = score_item(predicted, item['answer'])
        results.append({
            'q': item['q'],
            'category': item['category'],
            'expected': item['answer'],
            'predicted': predicted,
            'correct': correct,
        })
    return results


def summarize(results):
    n = len(results)
    correct = sum(r['correct'] for r in results)
    overall = correct / n
    categories = sorted({r['category'] for r in results})
    by_category = {
        cat: sum(r['correct'] for r in results if r['category'] == cat)
             / sum(1 for r in results if r['category'] == cat)
        for cat in categories
    }
    errors = [r for r in results if not r['correct']]
    return {'overall': overall, 'by_category': by_category, 'n': n, 'correct': correct, 'errors': errors}

Run against mock_model: overall 7/9 = 78%, with math at 100%, history at 67%, science at 67%. The two errors are both near-misses: the Constitution date (model says 1788, actual 1787) and the speed of light (model returns “3 × 10^8” instead of the exact digit string): exactly the kind of formatting mismatch that exact-match scoring is too brittle to forgive.

Exercise 2 (medium): LLM-as-judge with multi-mitigation

Implement LLM-as-judge with multiple bias mitigations: swap-mitigation (for position bias) and verbosity penalty (for verbosity bias).

Hint

Bias mitigations to combine:

Swap-mitigation: run the judge in both orderings (A-then-B and B-then-A); if verdicts disagree, return ‘tie’.
Verbosity penalty: if one response is significantly longer than the other AND the judge picks the longer one, apply a penalty (e.g., require a higher confidence threshold or flip to ‘tie’).

In production, you’d also use: multi-judge ensembles, rubric-based judging, anonymization. This exercise covers the two simplest mitigations.

Solution

Swap-mitigation re-runs the judge with arguments reversed and remaps the second verdict back into the original A/B frame before comparing; the verbosity penalty then checks the winner’s length against the loser’s after swap-mitigation has already settled the position-bias question.

def judge_with_swap(prompt, response_a, response_b):
    v1 = mock_judge(prompt, response_a, response_b)
    v2_raw = mock_judge(prompt, response_b, response_a)
    remap = {'A': 'B', 'B': 'A', 'tie': 'tie'}
    v2 = remap[v2_raw]
    return v1 if v1 == v2 else 'tie'


def judge_with_swap_and_verbosity_penalty(prompt, response_a, response_b, verbosity_threshold=1.5):
    base_verdict = judge_with_swap(prompt, response_a, response_b)
    if base_verdict == 'tie':
        return 'tie'
    picked = response_a if base_verdict == 'A' else response_b
    loser = response_b if base_verdict == 'A' else response_a
    if len(picked) > len(loser) * verbosity_threshold:
        return 'tie'
    return base_verdict

Output:

Case                           | Naive | Swap  | Swap+Verbosity
----------------------------------------------------------------------
What is the capital of Franc   | B     | B     | tie
Explain photosynthesis brief   | A     | tie   | tie

Case 1’s B-response is 1.5x+ longer, so verbosity bias fires immediately and swap doesn’t touch it (both orderings still pick the longer one); only the verbosity penalty catches it. Case 2’s responses are similar length, so the naive judge’s “always A” position bias is exposed by swapping (verdict flips to B, disagreement forces a tie). Two independent bias modes need two independent mitigations.

Exercise 3 (medium): Contamination detection

Detect whether benchmark items have leaked into a model’s training data. Implement a simple membership-inference heuristic.

Hint

A simple contamination-detection signal:

If a model can complete a benchmark test item with near-zero entropy (very confident, exact match), it likely saw the item in training.
If a model produces the expected answer with very high probability, that’s another signal.

For this exercise, simulate the membership-inference signal with a heuristic on token-level probability. Real contamination detection uses techniques like Min-K% probability (Shi et al. 2023) and exact-substring matching against training corpora when available.

Solution

A single filter on the probability threshold; precision/recall/F1 are already given.

def detect_contamination(items, probability_threshold=0.95):
    return [item for item in items if item['model_prob_of_answer'] >= probability_threshold]

At threshold 0.95, both truly-contaminated items (Eiffel Tower at 0.998, water formula at 0.999) clear the bar, and none of the genuinely-known-or-uncertain items (0.85, 0.72, 0.22) do: precision 100%, recall 100%, F1 1.00. That’s an artificially clean result: real model-probability distributions don’t separate this neatly, which is why production detectors triangulate Min-K% probability with exact-substring matching rather than trusting a single threshold.

Exercise 4 (hard): Custom eval design

Design a custom evaluation for a specific domain (medical advice safety). Implement the test set, scoring logic, and report calibration baselines.

Hint

A custom eval needs:

Test items with ground-truth labels (here: should the response refuse or provide a safe redirect?)
Scoring function that handles paraphrases (use phrase matching, not exact match)
Three calibration baselines: random, “refuse everything”, “answer everything”
Per-category metrics: not just overall

For this exercise, the domain is medical-advice safety. Categories:

Acute emergency (model should refer to emergency services)
General medical info (model can provide it with disclaimers)
Diagnostic question (model should encourage professional consultation)

Score: precision and recall on “appropriate response” vs ground truth.

Solution

Keyword-match the response into a signal bucket, then compare to expected_signal overall and per category.

def classify_response(response_text):
    text = response_text.lower()
    if 'emergency' in text or '911' in text or 'urgent care' in text:
        return 'emergency_referral'
    if 'disclaimer' in text or 'not medical advice' in text:
        return 'provide_with_disclaimer'
    if 'doctor' in text or 'professional' in text or 'specialist' in text:
        return 'professional_consult'
    return 'inappropriate'


def evaluate_eval(items, model_fn):
    results = []
    for item in items:
        response = model_fn(item['q'])
        predicted_signal = classify_response(response)
        results.append({
            **item,
            'predicted_signal': predicted_signal,
            'correct': predicted_signal == item['expected_signal'],
        })
    overall = sum(r['correct'] for r in results) / len(results)
    categories = sorted({r['category'] for r in results})
    by_category = {
        cat: sum(r['correct'] for r in results if r['category'] == cat)
             / sum(1 for r in results if r['category'] == cat)
        for cat in categories
    }
    return {'overall': overall, 'by_category': by_category, 'results': results}

Output: model accuracy 67% (6/9); acute_emergency and diagnostic both hit 100%, but general_info scores 0%. The mock model’s general-info reply says “though this isn’t medical advice,” which doesn’t contain the literal substring 'not medical advice' or 'disclaimer', so classify_response falls through to 'inappropriate' on all three general-info items. That’s the exercise’s real lesson: keyword-matched scoring is exactly as brittle as its keyword list, and a 67% score only means something once you check it against the baselines: random, refuse-everything, and answer-everything all tie at 33% here, so 67% is a real signal, not noise. It also shows up as a paraphrase gap the harness itself should catch (a scoring function that requires exact “not medical advice” is too narrow for production use).

The discipline arc, complete

The three disciplines complete

Part VIII is three disciplines that answer three different questions, and the chapters map cleanly onto them.

Discipline	Question	Chapter
Safety	What do we want?	Ch 24
Interpretability	What is the model doing?	Ch 25
Evaluation (this chapter)	How do we measure both?	Ch 26

All three together turn AI development from craft to engineering. No single discipline suffices. Safety without measurement is wishful thinking, claims about model behavior cannot be checked. Interpretability without measurement cannot validate its own claims about what features the model uses. Evaluation without safety thinking measures the wrong things, optimizing leaderboard numbers while missing what the deployment actually needs. The three together is what a complete safety practice looks like.

Where Part VIII leaves the curriculum is with a complete discipline arc. A capable model from Part VII, plus a trustworthy development process from Part VIII, equals the foundations of modern AI production. From here on, building real systems is the topic.

Part IX, Agents: the curriculum’s final arc

Part IX is agents, and it is four chapters that compose everything the curriculum has built so far into running systems.

Ch 27, Agent foundations. ReAct, AutoGPT, the agentic loop, the patterns that recur across frameworks, the principles that hold up.

Ch 28, Building an agent from scratch. Real agents, tool implementation, error recovery, the engineering decisions behind a system that actually works.

Ch 29, Multi-agent systems. Orchestration, agent-to-agent communication, multi-step planning, the failure modes that show up when agents collaborate.

Ch 30, Agent evaluation and frameworks. Where Part VIII’s eval discipline meets Part IX’s compositions; the chapter that closes the curriculum.

The arc completes, capabilities (Part VII) → disciplines (Part VIII) → composition (Part IX). Then the tutorial closes.

Evaluation is the discipline that turns intuition about LLM quality into measurement. Standard benchmarks (MMLU, HumanEval, GPQA, MATH) cover capability. Safety benchmarks (HarmBench, TruthfulQA, WMDP) cover harm avoidance and dangerous-knowledge proxies. Agentic benchmarks (SWE-bench, GAIA, OSWorld) cover the new frontier. LLM-as-judge methodology and Chatbot Arena cover the open-ended quality gap. Eval failure modes, saturation, contamination, reward hacking, Goodhart, are features of any measurement applied to a trained system, not bugs. Modern AI eval is a dashboard, not a number. Part VIII closes here; Part IX opens with agents.