Agent evaluation & frameworks

Chapter 26 made evaluation a discipline. Part VIII (Chapters 24–26) closed by establishing that capable models without rigorous evaluation are research demos, not products. Part IX then built the agent stack, foundations (Ch 27), engineering (Ch 28), composition (Ch 29). This chapter closes both arcs. It brings Part VIII’s evaluation discipline to bear on Part IX’s agent systems, and it closes the curriculum.

The chapter’s argument is structural. Agent evaluation is harder than LLM evaluation, harder in kind, not just in degree. Tasks are multi-step; success is multi-dimensional; benchmarks are slow to develop; production-readiness includes observability and deployment patterns the model-eval world rarely confronts. We cover the $2025$ agent benchmark landscape (SWE-bench Verified, GAIA, OSWorld, $\tau$-bench, BrowseComp, WebArena, Upwork HAPI), the methodologies that actually catch failures (pass$^k$ for reliability, cost-quality Pareto for production tradeoffs, a small custom eval when no benchmark fits), the frameworks that build agents (LangGraph, smolagents, the OpenAI Agents SDK, Pydantic AI, Anthropic native) and the frameworks that observe them once they’re live (LangSmith, Helicone, Braintrust, OpenTelemetry), and the deployment patterns plus readiness checklist that mark the boundary between prototype and production.

And then the curriculum closes. Section 8 looks back at the thirty chapters, from numpy primitives in Ch 1 to agent systems in production in this chapter, and takes stock of what the reader can now do. By the end of this chapter you’ll have the full stack of an LLM systems engineer: the discipline to evaluate what you build, and, equally important, the perspective to follow the field as it keeps moving.

Why agent evaluation is harder

Five structural reasons

LLM evaluation (Ch 26) is challenging but bounded. A single prompt, a single response, a labeled dataset, a metric. Agent evaluation is structurally harder for five reasons.

One. Tasks are multi-step. A single LLM eval has one prompt and one response; an agent eval has a multi-turn task, tool calls, retries, intermediate decisions. Evaluation must cover the trajectory, not just the final answer.

Two. Success is multi-dimensional. No single number captures all of these.

• Task success: did the agent complete?
• Cost: LLM calls plus tool calls plus total dollars
• Latency: time to completion
• Safety: no harmful actions taken
• Robustness: pass$^k$, success across $k$ attempts
• Cost-quality tradeoff: the Pareto frontier across configurations

Three. Verifying success is harder. LLM eval has labeled answers; agent eval often requires running the agent’s output (does the produced code compile? do tests pass?), replicating environment state (was the email sent? the file created?), or human judgment (is the produced essay any good?). Verification often requires running the world that the agent acted on.

Four. Reliability matters more. A $95\%$-accurate LLM is great; a $95\%$-reliable agent fails $1$ in $20$ production tasks. pass$^k$ surfaces what single-trial accuracy hides.

Five. Benchmarks are slow to develop. LLM benchmarks (MMLU, HumanEval) can be assembled in months. Agent benchmarks need realistic environments: real GitHub repos, real desktop OS, real web pages, and these take years. SWE-bench took about a year to build; OSWorld required custom VM infrastructure.

The framing

Agent eval is LLM eval $\times$ system complexity $\times$ environmental verification. It’s not just harder in degree, it’s harder in kind. The discipline that made LLM eval rigorous (Ch 26) has to extend to cover trajectories, dimensions, and environments. This chapter is that extension.

Agent benchmarks

The 2025 landscape

Five benchmarks dominate the agent landscape that took shape in $2025$. Each has a clear scope. The state-of-the-art numbers below are the frontier scores as of early $2025$, which is what most vendor system cards and papers still cite as the reference point; by mid-$2026$ every one of these benchmarks had moved substantially, several into saturation territory, and any specific percentage quoted here (including the mid-$2026$ figures) will be stale again by the time you read this. Treat the numbers as a snapshot of relative task difficulty and of how fast the frontier moves, not as today’s leaderboard.

SWE-bench Verified, coding agents. $500$ real GitHub issues from popular Python projects, curated for solvability. Success criterion: the agent’s PR passes the original tests. Frontier agents reached around $50\%$ as of early $2025$; humans hit roughly $85\%$. By mid-$2026$, top systems on the public leaderboard (swebench.com) were reporting scores well into the $80$s and $90$s percent, in some cases above that $85\%$ human baseline, though reported numbers vary widely by agent scaffold (a bare bash-shell ReAct loop scores far lower than a heavily tooled, multi-rollout system) and vendor self-reported numbers aren’t directly comparable to public leaderboard submissions. It remains the de facto coding-agent standard, every coding-agent vendor reports it. The Verified subset (~$500$) was the response to noise in the full $2{,}294$-problem benchmark; cleaner problems, cleaner signal.

GAIA, general AI assistants. $466$ multi-step real-world tasks requiring tool use, browsing, and file reading. Three difficulty levels. Frontier agents hit $60$–$75\%$ on Level 1 as of early $2025$; humans hit roughly $92\%$. By mid-$2026$, scaffolded agent systems on the public leaderboard were reporting Level 1 scores well above that range, while bare-model scores (no tools, no scaffold) remained far lower, so any single current number depends heavily on what exactly is being measured. The full-stack test, planning, tool use, retrieval, for general assistants.

OSWorld, computer-use agents. $369$ desktop tasks executed in Ubuntu and Windows VMs. State-based success criterion: was the file saved? the form submitted? Frontier agents reached roughly $12$–$15\%$ as of early $2025$, significantly harder than text-only benchmarks at the time. By mid-$2026$, leading systems on the public leaderboard were reporting scores in the $80$s percent, closing in on (and by some trackers’ accounting, passing) the roughly $72\%$ human baseline, faster progress than most of the field expected for the embodied regime; as always, scores swing on harness choice, so read any specific number alongside the setup that produced it.

$\tau$-bench (TauBench), tool-use reliability. Customer-service scenarios (retail, airline) with multi-turn tool use. The headline metric is pass$^k$: does the agent succeed across $k$ independent runs of the same task? Frontier pass$^4$ sat around $50\%$ as of early $2025$, surfacing reliability gaps that single-trial accuracy hides entirely. By mid-$2026$, leading systems were reporting pass$^4$ figures well into the $80$s and $90$s percent on the public leaderboard, and a successor benchmark (tau2-bench, from the same group) had broadened the domain set beyond retail and airline; the reliability gap the metric was designed to expose has narrowed considerably for well-engineered systems, though it’s still worth checking directly on any given stack.

BrowseComp, web research agents. Complex web research questions requiring multiple sources. Frontier agents hit $30$–$50\%$ as of early $2025$. By mid-$2026$, leading systems on the public leaderboard were reporting scores in the $80$s and low $90$s percent. Designed to be hard for humans too, questions where the answer requires substantial research, not a single lookup.

Two more benchmarks round out the landscape, both worth knowing even though neither made the widget’s five.

WebArena, realistic web-agent tasks. (Zhou et al. $2023$, arxiv.org/abs/2307.13854) $812$ tasks across real, self-hosted websites, e-commerce, forums, software collaboration, content management, scored on functional correctness: did the agent’s actions actually achieve the stated goal in the live environment, not just produce a plausible-looking answer. The original GPT-4 baseline solved about $14\%$ of tasks against a $78\%$ human baseline; frontier agents have closed much of that gap since, though any current number would be stale by the time you read it. WebArena predates BrowseComp and OSWorld and shaped both, the earlier and still-referenced check for whether an agent can operate a real website end to end, not merely browse it for information.

Upwork HAPI, real freelance-marketplace work. Built on UpBench (Yi et al. $2025$, arxiv.org/abs/2511.12306), a benchmark of jobs sourced directly from Upwork’s labor marketplace: each task is a verified client transaction, not a curated or synthetic problem, and expert freelancers decompose each job into verifiable acceptance criteria that agent submissions are scored against criterion by criterion, closer to this chapter’s partial-credit methodology than a single pass/fail judgment. The Human+Agent Productivity Index (HAPI) study built on this framework found that on simple, well-defined jobs (a deliberately narrow slice, under $6\%$ of Upwork’s marketplace volume), leading agents (GPT-5, Claude Sonnet 4, Gemini $2.5$ Pro, as of late $2025$) working alone completed a minority of projects; pairing the same agents with brief expert feedback, about $20$ minutes per review cycle, raised completion by up to $70$ percentage points, to as high as $93\%$.

What makes it different. Every benchmark above evaluates an agent working alone against a fixed task set. Upwork HAPI does two things differently: its tasks come from an active marketplace of real, paid client work rather than curated problems, and it measures the human-in-the-loop condition explicitly, not just whether the agent solves the task alone, but how much a small amount of expert human feedback changes the outcome. That second axis, solo-agent performance versus human-plus-agent performance, is close to absent from the rest of the benchmark landscape, and it is the more production-relevant number for any team planning to deploy agents alongside people rather than instead of them.

Benchmark hygiene matters. Test problems must not have leaked into training data. Environments need to be realistic, real repos, real OS, real web. Execution should be reproducible where possible. Multiple difficulty levels are useful for tracking progress across the capability range.

What benchmarks don’t capture

No benchmark captures everything. Subjective quality (was the email well-written?) and user satisfaction (did the user like the result?) stay outside the benchmark frame. Long-term impact (did the agent’s action cause downstream problems three days later?) is invisible to any single-task eval. Cost-effectiveness in deployment, where a $50\%$ benchmark score may be entirely acceptable at $1/10$ the cost of alternatives, depends on use case and volume, not the headline number.

Evaluation methodologies

Seven methodologies

Beyond benchmark scores, real agent evaluation uses several methodologies. Each catches a different class of failure; production teams typically run several in combination.

One. Task success rate. Binary, did the agent complete? The most common single metric. Easy to compute; hides everything else.

Two. Partial-credit scoring. Sub-task milestones; score based on how many were completed. More informative than binary success. SWE-bench could score on (files touched correctly, tests modified appropriately, tests passing). More work to design; rubrics drift if unmaintained.

Three. Pass$^k$ (reliability). Run the same task $k$ times; success rate across all $k$ runs (must succeed every time). Surfaces reliability issues hidden by single-trial accuracy. $\tau$-bench’s signature metric.

Four. Cost-quality Pareto frontiers. Plot configurations on (cost, quality) axes; identify which are dominated. This is the production-tradeoff metric.

Five. Human-judged evaluation. Have humans score outputs on a rubric. Necessary for subjective tasks like “write a good essay.” Expensive but irreplaceable for high-stakes decisions.

Six. LLM-as-judge (Ch 26 cross-reference). Use another LLM to score outputs. Cheaper than human eval; catches obvious failures; has known biases, verbosity, position, recency (all covered in Ch 26). Useful at scale; not a replacement for human judgment on important decisions.

Seven. Regression monitoring. Score outputs on a fixed dataset across model and prompt versions; detect regressions when changes degrade quality. The production-eval workhorse, covered in code in section 6.

# Pass^k evaluation: run a task k times; success only if ALL k attempts succeed.

import random

def run_agent_on_task(task, seed=None):
    """Mock agent: ~75% pass^1 rate."""
    if seed is not None:
        random.seed(seed)
    return random.random() < 0.75


def pass_at_k(task, k=4, n_trials=20):
    """pass^k: probability of succeeding in ALL k attempts."""
    successes = 0
    for trial in range(n_trials):
        all_succeeded = all(
            run_agent_on_task(task, seed=trial * k + i)
            for i in range(k)
        )
        if all_succeeded:
            successes += 1
    return successes / n_trials


task = "Handle this customer-service scenario."

print(f"=== Reliability evaluation for: {task} ===\n")
for k in [1, 4, 8]:
    rate = pass_at_k(task, k=k, n_trials=100)
    print(f"  pass^{k}: {rate:.0%}")

# Observations:
# - pass^1 captures single-trial accuracy (~75% in this mock)
# - pass^4 drops to ~32%: the agent fails 2 in 3 production sessions
# - pass^8 drops further: long-running deployments expose reliability issues
# - The gap between pass^1 and pass^k is what production cares about
# - tau-bench uses pass^4 as the headline metric

Methodology selection

Methodologies pair to task type. Coding tasks want benchmark scores (SWE-bench) plus regression monitoring. Customer service wants pass$^k$ plus human judgment on a sample. Research and writing want human judgment plus LLM-as-judge for scale. Production deployment wants all of the above, the cost of running multiple evaluation methodologies is small compared to the cost of shipping an unreliable system.

Building a custom eval

Public benchmarks cover common task shapes; most production agents don’t work in a shape a public benchmark covers. A support bot tied to one product’s return policy, a spreadsheet-automation agent tied to one company’s templates, none of the benchmarks above fit. The discipline is the one Ch 26 taught for LLM eval, applied to agents: build a small eval set with verifiable rewards, an answer checkable by exact match rather than judged, run the agent against it once per task, and compute pass@1 as the baseline before reaching for partial credit or judge-based scoring.

Verifiable rewards beat both alternatives whenever they’re available: no judge bias to correct for (Ch 26), no rubric to maintain. Ten to twenty tasks with one clearly correct answer each is enough to catch obvious regressions and establish a baseline; scale up only once the eval has earned its keep.

# Build a small custom agent eval: verifiable tasks, a mock agent, pass@1.

TASKS = [
    {'id': 1,  'question': 'What is 17 * 23?',                              'answer': '391'},
    {'id': 2,  'question': 'Capital of Australia?',                         'answer': 'canberra'},
    {'id': 3,  'question': 'Reverse the string "agent".',                   'answer': 'tnega'},
    {'id': 4,  'question': 'How many days are in a leap year?',             'answer': '366'},
    {'id': 5,  'question': 'What is the square root of 144?',               'answer': '12'},
    {'id': 6,  'question': 'Python syntax for the first element of list x?', 'answer': 'x[0]'},
    {'id': 7,  'question': 'HTTP status code for "not found"?',             'answer': '404'},
    {'id': 8,  'question': 'Sum of the integers 1 through 10?',             'answer': '55'},
    {'id': 9,  'question': 'Chemical symbol for gold?',                     'answer': 'au'},
    {'id': 10, 'question': 'Largest prime number less than 20?',            'answer': '19'},
]


def verify(task, agent_output):
    """Exact-match verifier: case-insensitive, whitespace-trimmed."""
    return agent_output.strip().lower() == task['answer'].strip().lower()


def mock_agent(task):
    """Mock agent: solid on arithmetic and general knowledge, wrong on
    two tasks (a stand-in for a real agent's blind spots)."""
    wrong_answers = {
        6: 'x(0)',   # plausible but wrong syntax
        9: 'gold',   # confuses the element name with its symbol
    }
    return wrong_answers.get(task['id'], task['answer'])


def pass_at_1(tasks):
    results = [(t['id'], verify(t, mock_agent(t))) for t in tasks]
    rate = sum(ok for _, ok in results) / len(tasks)
    return rate, results


rate, results = pass_at_1(TASKS)
print("=== Custom agent eval: 10 verifiable tasks ===\n")
for task_id, ok in results:
    print(f"  [{'PASS' if ok else 'FAIL'}] task {task_id}")
print(f"\npass@1: {rate:.0%}")

# Observations:
# - Verifiable answers (exact-match after normalization) remove judge subjectivity
# - Ten tasks is enough to see a pattern, not enough to be a production-grade eval
# - Task 6 fails on a plausible-but-wrong near-miss, exactly what a sloppier
#   substring or fuzzy check would let through
# - This is the shape of a custom eval: small, verifiable, and honest about
#   the one thing your system actually needs to be good at

Cost-quality Pareto

The Pareto frontier

Cost matters in production. A $90\%$-accurate agent at \0.50$per task may be worse than an$85%$-accurate agent at$$0.05$ per task, depending on volume and tolerance. The single-metric framing breaks down the moment a real budget enters the picture.

The Pareto frontier is the right tool. Plot configurations on (cost, quality) axes. A configuration is dominated if another exists with lower cost AND higher quality. The non-dominated set, the Pareto frontier, is the meaningful set of choices. Dominated configurations are never the right answer.

Typical cost-quality tradeoffs. The shape is stable even as the names churn: small, fast models sit at the cheap end with lower quality (Claude Haiku, OpenAI’s mini-tier models), frontier models sit at the expensive end with higher quality (Claude Opus and Anthropic’s most capable tier, OpenAI’s flagship reasoning models, Gemini’s Pro/Ultra tier). In $2025$ that split was Claude Haiku and GPT-4o mini on the cheap end against Claude Opus, GPT-4o, and Gemini Pro on the expensive end; by mid-$2026$ GPT-4o itself had been retired from general use and every one of those model names had been superseded at least once. Expect the specific names in this paragraph to look dated within another product cycle; the cheap-tier-versus-frontier-tier structure is the durable part. Hybrid configurations: a cheap model for routing and simple subtasks, a frontier model for the hard parts, usually land somewhere on the interior of the frontier, often dominating naive single-model setups.

Engineering levers

Model choice is the biggest lever; bigger is not always better when cost matters. Caching repeats, common subtasks can hit a cache instead of the model. Prompt compression: tighter prompts cost less per call. Self-routing, a cheap model decides whether the frontier model is needed at all. Parallelism versus sequential: parallel often shortens wall-clock latency without changing cost; sequential conserves cost but takes longer.

Production framing: define your cost budget (per task or per month), define your quality floor (minimum acceptable score), find Pareto-optimal configurations that meet both constraints, and iterate as models, prompts, and tools improve. The SWE-bench cost-quality story as of early $2025$: cheapest viable around \0.10$per task at roughly$30%$success; frontier around$$2$–$5$per task at roughly$50%$success. Both ends of that curve have moved since, frontier success rates on the same benchmark are now far higher (Section 2), and per-task cost has shifted as models got both more capable and more efficient, but a reliably sourced current pair of cost/quality numbers isn't available at time of writing; treat the$2025$ figures as illustrative of the shape of the tradeoff, not its present location. The gap is shrinking as smaller models improve.

# Identify Pareto-optimal configurations on (cost, quality).

CONFIGS = [
    {'name': 'haiku-only',          'cost': 0.02, 'quality': 0.45},
    {'name': 'sonnet-only',         'cost': 0.15, 'quality': 0.68},
    {'name': 'opus-only',           'cost': 0.85, 'quality': 0.78},
    {'name': 'hybrid-haiku-sonnet', 'cost': 0.08, 'quality': 0.62},
    {'name': 'hybrid-sonnet-opus',  'cost': 0.35, 'quality': 0.75},
    {'name': 'opus-bad-prompt',     'cost': 0.95, 'quality': 0.70},  # dominated
]


def pareto_frontier(configs):
    """Return non-dominated configurations."""
    frontier = []
    for c in configs:
        dominated = False
        for other in configs:
            if other is c:
                continue
            if other['cost'] <= c['cost'] and other['quality'] >= c['quality']:
                if other['cost'] < c['cost'] or other['quality'] > c['quality']:
                    dominated = True
                    break
        if not dominated:
            frontier.append(c)
    return frontier


print("=== All configurations ===")
for c in CONFIGS:
    print(f"  {c['name']:<25} cost=${c['cost']:.3f}  quality={c['quality']:.0%}")

print("\n=== Pareto-optimal ===")
for c in sorted(pareto_frontier(CONFIGS), key=lambda x: x['cost']):
    print(f"  v {c['name']:<25} cost=${c['cost']:.3f}  quality={c['quality']:.0%}")

# Observations:
# - 'opus-bad-prompt' is dominated (higher cost AND lower quality than 'opus-only')
# - All other configs are non-dominated: meaningful choices
# - Production picks a point on the frontier based on budget + quality floor
# - Dominated configurations are NEVER the right choice

Agent frameworks: building and observing

Agent-building frameworks

Chapters 27–29 built agents from first principles, loop, tools, schemas, multi-agent composition. Production teams rarely start from that blank slate twice; they reach for a framework. Five options cover most of what teams actually reach for.

LangGraph (LangChain). Graph-based orchestration for a single agent or many: nodes are steps, edges are transitions, and state persists explicitly between them. The same graph-of-nodes model Ch 29 covered for multi-agent orchestration works just as well for one agent with complex control flow, branches, loops, human-in-the-loop interrupts. Strengths: explicit state machine, first-class persistence and checkpointing, a natural fit if a system later grows into multi-agent. Weaknesses: more upfront structure than a simple task needs; a steeper learning curve than writing the loop by hand. When to use: complex state machines, workflows with conditional branches or approval steps, or any agent likely to grow into a multi-agent graph.

smolagents (Hugging Face). A deliberately small agent library built around code-writing agents: instead of emitting a JSON tool call, the agent writes and executes a snippet of Python, and the snippet’s result is the observation. Strengths: a codebase small enough to read end to end in an afternoon; code-as-action is often more expressive than a fixed tool schema for computational tasks. Weaknesses: executing agent-generated code raises the sandboxing stakes past what Ch 28’s tool-safety patterns assume; a smaller ecosystem than LangChain’s. When to use: research, prototyping, or any task where writing code genuinely is the natural action space.

OpenAI Agents SDK. OpenAI’s production agent framework, the successor to the experimental Swarm project (Ch 29). Handoffs between agents, guardrails, and tracing are first-class primitives built around OpenAI’s Responses API. Strengths: minimal boilerplate for an OpenAI-first stack; handoffs and guardrails come built in rather than bolted on. Weaknesses: designed around OpenAI’s API conventions, a poor fit for a multi-provider stack. When to use: OpenAI-first development where the framework and the model vendor are already aligned.

Pydantic AI. An agent framework built on Pydantic’s type system: tool arguments, structured outputs, and dependencies are all typed and validated the same way a Pydantic-based API validates a request. Strengths: malformed tool calls and outputs get caught at the schema level, Ch 28’s tool-schema discipline, enforced by the framework, instead of at runtime; a natural fit for teams already building on Pydantic or FastAPI. Weaknesses: the type discipline is overhead for a quick script or a one-off prototype. When to use: type-heavy production codebases where schema validation is already how the team works.

Anthropic native (no SDK). The API plus a hand-written harness, a loop, a tool-calling schema, your own state management: exactly what Chapters 27–28 built from scratch. Strengths: no framework abstraction between you and the model; every line of control flow is visible; no dependency to track through breaking changes. Weaknesses: you own the plumbing, retries, persistence, multi-agent coordination, that a framework would otherwise provide. When to use: minimal harness needs, or whenever you’d end up writing most of the loop yourself anyway.

Production observability platforms

Building an agent is half the work; operating it is the other half. Whichever framework built it, or none, running it in production needs different tooling: tracing, cost tracking, eval pipelines, regression detection. Five options dominate the $2025$ observability landscape, each with a clear design center.

LangSmith (LangChain). End-to-end LLM application observability. Trace visualization (a direct callback to Ch 28’s flame graph), dataset management, evaluation pipelines, prompt experimentation. Strengths: tight LangChain integration; mature; most-adopted in the LangChain ecosystem. Weaknesses: opinionated; assumes LangChain conventions. When to use: if you’re on LangChain or LangGraph.

Helicone. Drop-in proxy that captures every LLM call. Cost tracking, latency profiling, cache management, user-level analytics. Strengths: minimal integration overhead; open-source; cost-focused. Weaknesses: less feature-rich than LangSmith for complex workflows. When to use: cost-sensitive deployments; lightweight observability needs.

Braintrust. Evaluation-first LLM platform. Strong eval pipelines, regression detection, prompt experimentation, A/B testing. Strengths: rigorous eval-driven development; engineering-team focused. Weaknesses: steeper learning curve. When to use: teams running rigorous evals as part of their development workflow.

Anthropic’s evaluation tooling. Integrated with Anthropic’s Console and Claude apps. Evaluation playgrounds, prompt comparison, safety tooling. Strengths: tight integration with Anthropic models; safety-aware. When to use: Anthropic-native deployments.

OpenTelemetry GenAI conventions. Open standard for LLM trace data, standardized span attributes for model name, token counts, costs. Strengths: future-proof; vendor-neutral. Weaknesses: requires manual instrumentation. When to use: when vendor independence matters.

The convergence: most platforms now adopt OpenTelemetry-compatible conventions for span attributes. The plumbing is standardizing; the UX, UI, and surrounding workflow are what differentiate them. The framework graveyard of the last three years suggests that products that fight the convergence age poorly.

Choosing an observability platform

Three factors decide it in practice. Team familiarity beats everything else, the framework your team already knows is rarely the bottleneck, and the framework your team doesn’t know often is. Stack alignment, LangChain stacks fit LangSmith; cost-focused setups fit Helicone; eval-driven teams fit Braintrust. Vendor independence: if lock-in concerns dominate, OpenTelemetry’s neutral conventions let you swap platforms behind a stable instrumentation layer.

Deployment patterns and readiness

Five deployment patterns

Going from prototype to production requires more than hosting. Five deployment patterns show up in production agent systems; most mature deployments combine several.

One. Staging environments. Production-like with non-production data. Run regression eval before promoting to production. Catches breakage from prompt and model changes before users see it.

Two. Canary deployment. Roll out changes to a small percentage of traffic first. Monitor cost, latency, and quality. Roll back if metrics regress. The standard pattern for production agent deployments, named, fittingly, after the coal-mine canary that died before the miners did.

Three. A/B testing. Two versions in parallel; statistical significance for quality comparisons. Effective for prompt changes, model swaps, tool design changes. Watch the sample size, LLM outputs are stochastic and small samples mislead.

Four. Shadow deployment. New version runs in parallel without affecting users; outputs are compared offline. Lower risk than A/B; slower to learn from. Useful when the downside of a bad version is severe.

Five. Feature flags. Toggle agent features on and off without a code deploy; enable per-user rollouts; rollback in seconds. The fast-rollback option matters more for agent systems than for traditional services because failure modes are subtler and detection is slower.

# Compare current vs baseline on a fixed dataset; alert on regressions.

BASELINE_SCORES = {
    'task_lookup':       0.92,
    'task_summarize':    0.85,
    'task_code_review':  0.71,
    'task_math':         0.65,
}


def score_current_version(task_name):
    """Mock: in production, runs the current agent against the labeled dataset."""
    import random
    random.seed(hash(task_name) % 1000)
    drift = random.uniform(-0.05, 0.05)
    if task_name == 'task_math':
        return BASELINE_SCORES[task_name] - 0.15  # significant regression
    return BASELINE_SCORES[task_name] + drift


def check_regressions(baseline, threshold=0.05):
    print("=== Regression check ===\n")
    regressions = []
    for task, baseline_score in baseline.items():
        current = score_current_version(task)
        delta = current - baseline_score
        marker = 'FAIL' if delta < -threshold else 'ok  '
        print(f"  [{marker}] {task:<22} baseline={baseline_score:.0%}  current={current:.0%}  d={delta:+.0%}")
        if delta < -threshold:
            regressions.append({'task': task, 'delta': delta})
    print()
    if regressions:
        print(f"!! {len(regressions)} regression(s), BLOCK deployment")
    else:
        print("ok All tasks pass, OK to deploy")
    return regressions


check_regressions(BASELINE_SCORES)

# Observations:
# - Regression monitoring catches quality drops that benchmarks alone miss
# - Threshold (5%) is tunable: too tight = false alarms; too loose = misses
# - 'task_math' regressed 15%: flagged
# - Production CI/CD blocks deployments with significant regressions
# - This is the production-eval workhorse

The production-readiness checklist

The transition from prototype to production is where most agent projects stall. This checklist makes the transition concrete. Each row is a category, not a single check; the items are the ones production agent teams discover (often the hard way) in the first six months of deployment.

The agent arc, reviewed

The four chapters

Part IX covered the agent stack in four chapters, each handling a distinct layer.

The Part IX arc is short to state. Ch 27 to Ch 28: concept to working code. Ch 28 to Ch 29: one agent to many. Ch 29 to Ch 30: building to evaluating. Each chapter builds directly on the prior; each delivers a layer the next composes on top of.

What the reader can now do

After Part IX the reader can design an agent loop appropriate to a task, picking from the patterns of Ch 27. Implement production-grade agents with proper tools, schemas, error handling, and observability, applying the engineering of Ch 28. Decide when multi-agent is warranted and, equally importantly, when it isn’t, using the calibration of Ch 29. Evaluate agent systems with appropriate methodologies, pass$^k$, Pareto, regression monitoring, from this chapter. Deploy agents to production with the right observability and safety patterns. None of these are theoretical; each comes with code the reader has worked through and widgets the reader has played with.

Where agents stand today: agents are real, useful, and improving fast. The $2025$ benchmark numbers this chapter opened with (SWE-bench Verified around $50\%$, OSWorld around $15\%$, $\tau$-bench pass$^4$ around $50\%$) already look dated, by mid-$2026$ frontier systems had pushed all three well past those marks (Section 2), and the numbers will have moved again by whenever you’re reading this. That climb is itself evidence of how fast capability is still being built, but production-readiness hasn’t kept pace with benchmark scores; the gap between “scores well on a benchmark” and “ready for production” (this chapter’s central argument) is the more durable fact than any single percentage. The reader leaving Part IX has the practical foundation to build agent systems that survive production traffic, and the calibration to avoid over-engineering them.

Exercises

Four exercises that lock in the discipline this chapter brings to agent systems. Each is a self-contained problem with a starting template; hints are collapsed by default; try the problem first.

The exercises trace the chapter’s argument: design evaluation for a specific task (Ex 1) → compute pass$^k$ from raw data (Ex 2) → identify the Pareto frontier (Ex 3) → audit a production-readiness gap (Ex 4). After these, the reader has the practical discipline to evaluate any agent system being prepared for production, and to recognize systems that aren’t ready.

Exercise 1 (easy): Evaluation methodology design

For each of four agent tasks, design an evaluation approach: pick the appropriate benchmark (or note “no benchmark fits”); choose methodologies (success rate, pass$^k$, partial credit, human-judged, LLM-as-judge); identify what counts as success. The point is to develop the judgment that no single methodology fits all tasks.

def design_evaluation(task):
    desc = task['description'].lower()

    if 'coding' in desc or 'github issue' in desc:
        if 'javascript' in desc or 'js ' in desc:
            return ('custom', ['success_rate', 'regression_monitoring'],
                    "Issue resolved and tests pass (SWE-bench Verified is Python-only; "
                    "a JS repo needs a custom eval built the same way)")
        return ('SWE-bench Verified', ['success_rate', 'regression_monitoring'],
                "Generated patch makes the target repo's test suite pass")
    if 'customer' in desc or 'refund' in desc:
        return ('custom', ['pass^k', 'human_judged'],
                'Refund resolved per policy across repeated attempts, sampled for human review')
    if 'blog' in desc or 'writes' in desc or 'marketing' in desc:
        return ('custom', ['human_judged', 'llm_as_judge'],
                'Human editors rate voice/quality; LLM-as-judge scores the rest for scale')
    if 'spreadsheet' in desc or 'clicking' in desc or 'excel' in desc:
        return ('OSWorld', ['state_verification'],
                'Final spreadsheet state matches the expected cell values/formulas')
    return ('custom', ['human_judged'], 'No heuristic matched; default to human judgment')

Exercise 2 (medium): pass$^k$ computation from raw trial data

Given raw trial results (lists of pass/fail across multiple independent runs), compute pass$^k$ for $k \in \{1, 2, 4, 8\}$. Demonstrate empirically that pass$^k$ decays fast with $k$, the gap $\tau$-bench surfaced and that production deployments must worry about.

def pass_at_k(trials, k):
    groups = [trials[i:i + k] for i in range(0, len(trials) - len(trials) % k, k)]
    return sum(all(g) for g in groups) / len(groups)

Task                   pass^1     pass^2     pass^4     pass^8
--------------------------------------------------------------
task_easy              92%        84%        70%        44%
task_medium_high       71%        52%        28%        8%
task_medium            57%        31%        10%        0%
task_medium_low        34%        16%        4%         0%
task_hard              18%        4%         0%         0%

Exercise 3 (medium): cost-quality Pareto frontier identification

Given $8$ agent configurations on (cost, quality), identify which are Pareto-optimal (non-dominated) and which are dominated. Visualize the frontier as a sorted list. The exercise reinforces the chapter’s framing: dominated configurations are never the right choice.

def is_dominated_by(c, other):
    return (other['cost'] <= c['cost'] and other['quality'] >= c['quality']
            and (other['cost'] < c['cost'] or other['quality'] > c['quality']))


def pareto_frontier(configs):
    non_dominated = [c for c in configs
                      if not any(is_dominated_by(c, o) for o in configs if o is not c)]
    return sorted(non_dominated, key=lambda c: c['cost'])

Exercise 4 (hard): Production-readiness audit

Given a description of an agent system being prepared for production, audit it against the $8$-category readiness checklist. Identify gaps; propose specific actions to close each gap. The exercise builds the practical skill of evaluating whether a system is genuinely ready to ship.

EVIDENCE = {
    'Manual QA': ['tested manually', 'manual eval'],
    'Per-task budget': ['cost per query is estimated'],
    'Tool sandboxing': ['domain allowlist', "can't write to the", "can't run shell"],
    'All calls traced': ['log every llm call'],
    'Structured logging': ['log every llm call to cloudwatch'],
    # everything else (Benchmark threshold, Edge cases, pass^k threshold,
    # Tool failure handling, Timeouts, Cost alerting, Rate limiting,
    # p50/p95/p99 measured, SLA defined, Latency alerting, Safety eval,
    # PII redaction, Staging env, Canary pattern, Rollback procedure,
    # On-call runbook, Incident response, Post-mortems) has no keywords:
    # the description never addresses them, so they're gaps by default.
}


def audit(description, checklist):
    desc = description.lower()
    result = {}
    for category, checks in checklist.items():
        covered, gaps, actions = [], [], []
        for check in checks:
            if any(kw in desc for kw in EVIDENCE.get(check, [])):
                covered.append(check)
            else:
                gaps.append(check)
                actions.append(f"Add {check.lower()} before launch")
        result[category] = {'covered': covered, 'gaps': gaps, 'actions': actions}
    return result

The curriculum closes

Thirty chapters

Thirty chapters. From numpy primitives in Ch 1 to agent systems in production in Ch 30. Looking back over the arc, the curriculum’s nine parts each carried a distinct piece of the modern LLM stack, and each, in retrospect, set up the part that came after it.

Part I, Foundations (Ch 1–3). Tokens, embeddings, basic neural building blocks. Implemented in numpy from first principles. Matrix multiplication and softmax: the substrate of everything that followed. The chapters that set the discipline of the curriculum: nothing is allowed to remain magic.

Part II, The Transformer (Ch 4–6). Attention, multi-head attention, positional encoding, the full transformer block. The architectural pattern that powered the field. You implemented it from scratch, and from that point on, every model the rest of the curriculum touched was a variation, an extension, or a deployment story for the pattern built here.

Part III, Pre-training (Ch 7–10). Training objectives. Scaling laws (Chinchilla; Hoffmann $2022$). Data curation. Distributed training. How frontier models actually get made: not just in theory but in engineering practice. The infrastructure chapters in particular grounded what “training a frontier model” actually costs and looks like.

Part IV, Alternative Architectures (Ch 11–12). Mixture of experts; state-space models. The alternatives to dense transformers, Mixtral, Mamba, the diversity of the architectural landscape. Most production is still transformer-dominated, but the alternatives matter for what they reveal about the design space.

Part V, Post-training (Ch 13–16). Supervised fine-tuning; RLHF; DPO; constitutional AI; parameter-efficient methods; distillation. From base models to assistants, from “predicts the next token” to “answers helpfully and harmlessly.” The chapters that turned LLMs into the products the field actually ships.

Part VI, Inference (Ch 17–19). KV-caching; quantization; speculative decoding; sampling strategies. The engineering that makes serving fast and affordable: the $100$ms-per-token versus $5$ms-per-token difference between a toy and a product. The chapters most readers will revisit when their costs blow up.

Part VII, Capabilities (Ch 20–23). Reasoning (chain-of-thought, scratchpads, test-time compute). Tool use. Retrieval-augmented generation. Multimodal. What LLMs can do beyond text generation, and the engineering patterns that make each capability tractable.

Part VIII, Discipline (Ch 24–26). Safety, interpretability, evaluation. The three disciplines that turn impressive capability into trustworthy products. This is where the curriculum stopped accumulating features and started insisting on rigor. Ch 26 in particular set up the move this chapter completes.

Part IX, Agents (Ch 27–30). Foundations, engineering, composition, evaluation. The composition of every layer that came before: capable models (Parts I–VII), made trustworthy (Part VIII), assembled into systems that observe, think, act, iterate, and prove themselves in production. The closing arc.

What’s not covered

No curriculum covers everything. This one explicitly does not cover specific model implementations (Llama, Claude, GPT internals, vendor-specific and changing); hardware design (TPUs, chips, beyond the chapters on infrastructure); the latest research papers (the field moves faster than any curriculum can); specific business applications (each industry has its own patterns and constraints). What this curriculum did instead was build the substrate that lets you read those things, papers, vendor docs, industry write-ups, and understand them on first pass.

What comes next

For the field, the directions are clear in shape if not in detail. Better reasoning, test-time compute scaling, reasoning models in the o1 and o3 lineage, novel objectives. Better agents: more reliable, more capable, more autonomous within bounded scopes. Better safety: interpretability scaling, alignment research, the methods that make capability claims trustworthy. Better efficiency: smaller models with frontier capability, the cost curve continuing to drop. Better integration, LLMs as components in larger software systems, not as standalone products.

You have the foundation to follow any of these directions. The chapters that taught the substrate, Ch 1’s numpy, Ch 4’s attention, Ch 26’s evaluation discipline, Ch 28’s agent engineering, are the same chapters that will let you read the papers and write the systems that come next.

One last thing

Thirty chapters. From the first matrix multiplication to the last production agent. You started with numpy and ended with systems that observe, think, act, and iterate at production scale. The field will keep moving, new architectures, new capabilities, new failure modes, new disciplines. What this curriculum gave you is the foundation to follow.

The attention operation is still the same matrix multiplication and softmax it was in Chapter 4. The agent loop is still the same observe-think-act it was in Chapter 27. The principles don’t change as fast as the products. When the next breakthrough lands, and it will, you’ll be reading the paper with the substrate to understand it.

That’s what this curriculum was for.

Now go build.