Reasoning & test-time compute

Part VI ended with a deployable model, efficient on the forward pass, quantized to fit on commodity hardware, and sampling tokens cleanly. But a deployable model is not automatically a useful one. For factual questions (“who painted the Mona Lisa?”) direct generation works fine: the answer is in the weights and one forward pass suffices. For hard reasoning, math word problems, multi-step logic, code with a subtle bug, planning across many moves, direct generation often fails. The model commits to early decisions it cannot revise, and a wrong intermediate step compounds into a wrong final answer. Reasoning techniques address this by giving the model space to think before it answers.

This chapter walks the two-era arc that produced the reasoning capabilities deployed today. The classic era (2022 – 2023) discovered that prompting a model to “think step by step” dramatically improved accuracy without changing the model at all. Wei et al. 2022 introduced chain-of-thought; Wang et al. 2022 added self-consistency (sample $N$ traces, majority vote); Yao et al. 2023 generalized to tree-of-thoughts; Yao et al. 2022 added ReAct as the reasoning-plus-action pattern that became modern agents. The modern era (2024 – 2025) trained reasoning directly. OpenAI’s o1, DeepSeek’s R1, and Google’s Gemini Thinking all use RLVR (Chapter 14) to teach the model to produce long internal reasoning traces autonomously. Test-time compute scaling (Snell 2024) showed empirically that thinking longer monotonically improves accuracy on hard problems, the result that made the o1/R1 paradigm economically viable.

Part VII opens here. Reasoning is the natural starting point because it is the capability that turns a chat model into a problem-solving system. Chapter 21 covers tool use, extending the ReAct pattern into engineered agents. Chapter 22 covers RAG, where retrieval extends the model’s context with external knowledge. Chapter 23 covers multimodal systems that span vision and audio. Together, Part VII turns deployable models into useful ones. After Part VII: safety, interpretability, evaluation (Part VIII), and then agents (Part IX). The deployment work of Parts V and VI was the price of admission; the capability work begins now.

Why reasoning matters

The limit of direct generation

A trained transformer’s forward pass emits tokens autoregressively, one token, then the next, conditioned on everything before. For factual recall this is plenty: the model has the answer in its weights and the right token comes out first. For reasoning it often is not. A single forward pass commits to early decisions. Once the model emits “the answer is $120$,” it cannot easily revise; the sampler will not retroactively decide that the third token should have been different. Without intermediate steps the model has no place to do the work that an arithmetic problem or a multi-step plan demands.

The empirical evidence for this gap is unambiguous and it was the original motivation for the field. GPT-3 on GSM8K (the canonical grade-school math benchmark, free-response, not multiple-choice) scores about $16\%$ with direct generation. With chain-of-thought prompting, GPT-3-class models reach roughly $46\%$, roughly a threefold improvement from nothing more than a change in prompt structure. Larger models show the same pattern with a higher ceiling: PaLM 540B goes from $17\%$ direct to $57\%$ with CoT. The model already had the reasoning capability latent in its weights; what was missing was the inference-time machinery to elicit it.

The chapter’s two-era arc

The story since 2022 splits cleanly into two paradigms with very different implementation surfaces. The classic era (2022 – 2023) achieved reasoning via prompting alone: the model was unchanged; only the prompt structure differed. Few-shot exemplars with intermediate steps, “let’s think step by step,” majority voting over $N$ samples, tree search over partial traces, all inference-time interventions, all applicable to any model. The modern era (2024 – 2025) trains reasoning in directly via RL. The model autonomously produces long internal reasoning traces; the user just sees the answer (o1) or sees both (R1, Gemini Thinking). The capability is no longer a prompting trick but a property of the trained weights.

Both eras are still relevant. CoT prompting works on any model and is essentially free to apply; modern reasoning models are the state of the art for hard math, code, and science. The choice depends on the task and the budget, and the chapter’s final section gives the decision rule.

Chain-of-thought prompting

Few-shot CoT

Wei et al. 2022 introduced chain-of-thought prompting: provide few-shot examples in which each answer includes its intermediate reasoning steps before the final answer. The model imitates the pattern and produces reasoning on the new query. The technique is mechanically trivial, a different prompt template, and empirically dramatic. GPT-3 on GSM8K: direct ~$16\%$; CoT ~$46\%$. PaLM 540B: direct ~$17\%$; CoT ~$57\%$. The paper’s central claim is that the capability was already there; the prompt was what unlocked it. The model had seen enough step-by-step solutions during pre-training to know the form of mathematical reasoning, but in the absence of exemplars it defaulted to direct answer generation.

The choice of exemplars matters more than the count. Few-shot CoT typically uses three to eight examples drawn from the same problem domain, with intermediate steps that match the reasoning style the target query will need. Mismatched exemplars, say, exemplars that use one notation when the target query uses another, hurt performance even when the underlying math is identical. Production systems often retrieve relevant CoT examples per query rather than pinning a single static prompt template.

Zero-shot CoT

Kojima et al. 2022 found that the exemplars are not strictly necessary. Just append “Let’s think step by step.” to the prompt and the model produces a reasoning trace on its own. The technique requires no examples, no domain adaptation, no template engineering, a single-sentence intervention that works across math, logic, and many forms of multi-step reasoning. It does not always match few-shot CoT in absolute accuracy, but it gets close on most tasks and is essentially free.

Zero-shot CoT is one of the most-cited single-sentence interventions in NLP for a reason. It demonstrated that the few-shot exemplars were not transferring task-specific knowledge but unlocking a latent reasoning mode that the model could enter on a verbal cue. That observation is what made trained reasoning seem plausible in the first place: if the capability is latent and a sentence can elicit it, then training the model to enter the mode autonomously is a tractable goal.

Why it works (and where it fails)

The mechanistic story is the one stated in the section 1 callout. Extra reasoning tokens give the model more compute per query, and each one attends back through the KV cache to the entire reasoning so far. The trace functions as scratch paper. Why this works: a $1{,}000$-token reasoning trace gives the model $1{,}000$ forward-pass-worth of compute on the same query, where direct generation gives it one. The latent capability the model has from pre-training was always there; CoT just gives it the room to use it.

The failures are equally instructive. The reasoning-answer gap: the trace and the final answer can disagree, with the model producing apparently correct steps but emitting a wrong final number, or vice versa, with the right answer landing after a muddled trace. Plausible-but-wrong traces: the model can hallucinate reasoning that looks valid step-for-step without any of the steps actually being correct, and the accuracy gains do not always correlate with the trace’s faithfulness. Prompt sensitivity: small phrasing changes (“let us think step by step” vs “let’s think step by step” vs “think carefully”) produce different results. None of these failures kill the technique, but all of them are reminders that CoT is a behavioral pattern, not a guarantee.

The template for the entire family, in compact form:

def call_model(prompt):
    """
    Mock LLM. In production this calls a real API.
    The mock returns the trace a real model typically produces
    for the two prompt styles below.
    """
    if "step by step" in prompt.lower():
        return ("The cafeteria had 23 apples. They used 20 for lunch, so 23 - 20 = 3.\n"
                "Then they bought 6 more, so 3 + 6 = 9.\n"
                "The answer is 9.")
    return "The answer is 29."

question = ("The cafeteria had 23 apples. If they used 20 to make lunch "
            "and bought 6 more, how many apples do they have?")

direct_prompt = f"Q: {question}\nA:"
direct_answer = call_model(direct_prompt)
print("Direct prompt:")
print(f"  {direct_answer}\n")

cot_prompt = f"Q: {question}\nA: Let's think step by step."
cot_answer = call_model(cot_prompt)
print("Zero-shot CoT:")
print(f"  {cot_answer}")

print("\nObservations:")
print("- Direct generation skips intermediate steps and produces wrong answers")
print("- 'Let's think step by step' elicits the model's reasoning capability")
print("- The reasoning trace lets the model 'check its work' at each step")
print("- This is the simplest reasoning technique that works.")

Self-consistency and tree-of-thoughts

Self-consistency

Wang et al. 2022 made the obvious next move: if one CoT trace helps, $N$ traces should help more. Self-consistency samples $N$ independent CoT traces (typically $N = 5$ to $40$), extracts each trace’s final answer, and takes the majority vote. Independence comes from sampling variance, different random seeds, the same prompt, the same model. Wrong traces tend to disagree with each other in different ways; the correct answer tends to appear most often. The technique adds $10$ to $20$ accuracy points over single-sample CoT on GSM8K and similar math benchmarks, and the gain is roughly monotone in $N$ over the useful range.

The mechanism is the simplest form of test-time compute trade-off in the chapter, and the conceptual ancestor of everything in the modern era:

The reason this works at all is the wisdom-of-crowds principle applied to one model sampled $N$ times. Each trace makes its own errors; the errors are diverse; the correct answer is the one most paths converge on. Self-consistency trades $N\times$ inference compute for accuracy, and foreshadows the test-time compute scaling story of section 6. The result was striking in 2022 precisely because the field had assumed that single-sample greedy or low-temperature decoding was close to optimal. It is not: variance, properly aggregated, is worth real accuracy points.

Tree-of-thoughts

Yao et al. 2023 generalized the multi-trace idea to explicit search. Tree-of-thoughts represents reasoning as a tree: each “thought” is a node; child nodes extend the reasoning. A BFS or DFS expands promising branches and prunes weak ones, with the language model itself scoring intermediate states (“does this partial reasoning look like it is heading somewhere?”). The search backtracks when a branch looks unpromising, something linear CoT cannot do. Once CoT commits to a wrong step, the rest of the trace is downstream of it. Some ToT variants swap the plain tree walk for MCTS-style search: instead of a heuristic score deciding which node to expand next, simulate full rollouts to a leaf, back up the result to estimate each node’s value, and bias future expansion toward the highest-value branches, the same explore/exploit machinery behind AlphaGo’s board-position search, repurposed for reasoning traces. MCTS costs more bookkeeping than plain BFS/DFS but handles deceptive intermediate states better, cases where a step looks weak by a shallow heuristic but leads somewhere good several moves later.

The empirical wins are large but narrow. ToT significantly outperforms CoT on planning puzzles, Game of 24, creative writing tasks with multiple drafts, crosswords, where the search structure matches the problem’s branching structure. On standard math benchmarks the win is smaller, and the cost is much higher: ToT is typically $10\times$ to $100\times$ more expensive than CoT for the same query, since each branch is a full sub-rollout. ToT pays off for hard problems that genuinely require backtracking and rarely for problems that do not. In production it is reached for sparingly; for most chat-style reasoning the modern reasoning models of section 7 win on both quality and cost.

The implementation of self-consistency itself is short, extract each trace’s answer, count, return the mode:

import re
from collections import Counter

def extract_answer(trace, pattern=r"answer is (\d+)"):
    """Extract numeric answer from a CoT trace."""
    matches = re.findall(pattern, trace.lower())
    return int(matches[-1]) if matches else None

def self_consistency(traces):
    """Sample N CoT traces; majority-vote the final answers."""
    answers = [extract_answer(t) for t in traces]
    answers = [a for a in answers if a is not None]
    if not answers:
        return None, 0.0
    counter = Counter(answers)
    most_common, count = counter.most_common(1)[0]
    return most_common, count / len(answers)

# Demo: 5 CoT traces for the same problem.
# In production these come from sampling the model N times with different seeds.
mock_traces = [
    "Speed = 60/2 = 30 mph. The answer is 30.",
    "60 miles in 2 hours. Speed is 30. The answer is 30.",
    "Distance/time. The answer is 30.",
    "I think it's 40. The answer is 40.",          # outlier
    "60 / 2 = 30. The answer is 30.",
]

answer, confidence = self_consistency(mock_traces)
print(f"Sampled {len(mock_traces)} traces.")
print(f"Majority answer:  {answer}")
print(f"Confidence:       {confidence:.0%}  (fraction of traces agreeing)")
print()
print("Observations:")
print("- One trace was wrong (outlier said 40)")
print("- 4 of 5 traces agreed on the correct answer")
print("- Majority vote ignores the outlier; result is robust")
print("- This is the simplest test-time compute technique that works.")

ReAct and tool-integrated reasoning

The Thought-Action-Observation pattern

Yao et al. 2022 introduced the pattern that became the foundation of modern LLM agents: interleave reasoning with actions. The model emits a Thought, then an Action (a tool call), receives an Observation (the tool’s result), and continues with the next Thought. The loop runs until the model emits a final answer:

Thought: I need to find the population of France.
Action: search("population of France")
Observation: ~68 million
Thought: Now I need the population of Germany.
Action: search("population of Germany")
Observation: ~83 million
Thought: France 68M, Germany 83M. Germany is larger.
Final answer: Germany

Three things change once actions enter the loop. Grounding, each observation comes from an actual tool, not from the model’s hallucination. Numbers are right because a calculator computed them; facts are right because a search engine returned them; code outputs are right because an interpreter ran them. Compositional reasoning: the model chains multiple lookups across multiple tools, building up state in its reasoning trace that no single forward pass could produce. The foundation of modern LLM agents, every production agent system, whether built on Claude tool use, GPT function calling, or open-source frameworks, runs some form of Thought → Action → Observation underneath. The pattern’s longevity is unusual in this field; the API surface has changed dozens of times since 2022 but the loop structure has not.

Bridge to Chapter 21

This chapter establishes the pattern; Chapter 21 covers the engineering. How are actions parsed? Usually with constrained decoding (Chapter 19) so the action emits as valid JSON or a typed function call. How are observations formatted? Usually as a structured block the model has been trained or prompted to interpret. How does the loop terminate? Either when the model emits a designated final-answer marker, when a tool call returns a terminal result, or when a maximum step count is hit. How are errors handled? The model sees the tool’s error message as the observation and decides whether to retry, fix the call, or give up.

All of that engineering belongs in Chapter 21. For this chapter, ReAct is the conceptual bridge from “model thinking” to “model acting,” and the natural close of Topic 1. The classic era ended with the observation that reasoning and action can interleave; the modern era opens with the observation that the model can be trained to do this autonomously rather than prompted into it.

Process reward models

Scoring each step

Topic 1 covered prompting; Topic 2 covers training. The hinge between them is the reasoning-answer gap problem that section 2 raised: a CoT trace can be wrong even when the answer is right (lucky guess) or right even when the answer is wrong (final-step slip). If the reward signal only sees the final answer, the model has no way to learn that a sloppy trace happened to work or that a careful trace landed in a typo. Process reward models (Lightman et al. 2023) address this by scoring each step of a reasoning trace, not just the final answer.

The reward model is trained on per-step labels: “is this step correct given the steps before it?” Each step gets a score in $[0, 1]$, and the trace’s overall score combines them (minimum, mean, or product, depending on the recipe). The training data is expensive: someone has to label intermediate reasoning steps, not just final answers. The canonical dataset (PRM800K, the dataset behind Lightman 2023) is exactly this: GPT-4 generated the candidate solutions to MATH problems, and human annotators then labeled roughly 800,000 individual steps across those solutions as positive, negative, or neutral. The expense is the human step-level labeling, not the solution generation.

Two uses

PRMs serve two roles in the reasoning stack, and the distinction matters for how aggressive the model needs to be. Inference-time scoring: generate $N$ candidate traces; score each with the PRM; return the highest-scoring trace. This is best-of-$N$ with a learned verifier, and it does not require retraining the policy. Training-time reward: use the PRM as a dense reward signal during RLVR (Chapter 14), giving the policy step-level credit rather than only end-of-rollout credit. Dense reward generally trains faster than sparse reward but is much more expensive to construct.

PRM vs ORM

The simpler alternative is an outcome reward model (ORM) that only scores the final answer. ORMs need only outcome-level labels (much cheaper than per-step labels) and avoid the failure mode where a PRM hallucinates that an intermediate step is “correct” when it is not. PRMs offer denser feedback but introduce reward-hacking risks: the model can learn to perform reasoning steps that look correct to the PRM without those steps actually helping the answer.

The R1 surprise was decisive on this point. DeepSeek showed that pure outcome rewards work for reasoning, no PRM needed; rule-based verifiers (exact-match on math; test-pass rate on code) supplied the reward for a policy trained via GRPO (Chapter 14), producing a model that matches o1 on hard benchmarks. The field has reconsidered whether PRMs are necessary for training. They are still useful for inference-time filtering, and they remain a research topic, but the simplest training recipe that works does not use them.

def mock_prm_score(trace):
    """
    Mock PRM. Scores a trace based on simple surface features.
    A real PRM is a trained transformer that scores each step's correctness.
    """
    has_explicit_math = '=' in trace or '/' in trace or '+' in trace
    length_score = min(len(trace) / 100, 1.0)
    math_bonus = 0.3 if has_explicit_math else 0.0
    return min(0.5 + length_score * 0.3 + math_bonus, 1.0)

def best_of_n_with_prm(traces):
    """Pick the trace with the highest PRM score."""
    return max(((t, mock_prm_score(t)) for t in traces), key=lambda x: x[1])

# Demo: 4 candidate traces of varying quality, same final answer
traces = [
    "The answer is 30.",
    "Speed = 30. The answer is 30.",
    "60 / 2 = 30. The answer is 30.",
    "Distance = 60 miles, time = 2 hours. 60 / 2 = 30 mph. The answer is 30.",
]

scored = sorted(((t, mock_prm_score(t)) for t in traces), key=lambda x: -x[1])
print(f"{'Score':>6} | Trace")
print("-" * 70)
for trace, score in scored:
    print(f"{score:>6.2f} | {trace[:60]}{'...' if len(trace) > 60 else ''}")

print()
print("Observations:")
print("- Detailed traces with explicit math score highest")
print("- A real PRM scores *each step's correctness*, not just length")
print("- Best-of-N with PRM trades N x inference for accuracy")
print("- This is the inference-time use of PRMs; no model retraining needed.")

Test-time compute scaling

The empirical claim

Snell et al. 2024 ran the experiment that retroactively justified the o1/R1 design space. For many tasks, increasing inference-time compute outperforms increasing model size. The setup was a controlled sweep: take a fixed model; run it many ways at inference time, from direct generation ($1\times$ compute) through self-consistency and best-of-$N$ + PRM up to deep tree search ($100\times$+ compute); measure accuracy as a function of compute. The result, across the math benchmarks they studied, was that accuracy scales with inference compute along surprisingly stable curves. Doubling inference compute often beats doubling model size, especially on hard problems short of the very hardest, where a bigger pretrained model still wins.

The framing matters as much as the numbers. Before 2024, the operational story was simple: bigger model, better answers; compute spent at inference time is mostly fixed by the model size. Snell’s result said that this is wrong, inference compute is a separately tunable dimension that genuinely contributes to accuracy. A smaller model with more thinking time can match a larger model with less thinking time on the kinds of problems where extra thinking helps. That observation made the o1/R1 paradigm economically defensible: training the model to think for ten minutes is absurd unless you have evidence that ten minutes of thinking is worth anything, and Snell provided the evidence.

The scaling curve

A schematic form that captures the qualitative shape of the data (not Snell’s specific fitted model, which is an empirical compute-optimal allocation across difficulty bins rather than a single closed-form law): accuracy depends on a product of a model-size effect and a compute effect, each with its own exponent.

The empirical fact that $\beta > 0$ is the load-bearing observation, inference compute contributes, and the fact that $\beta$ is large enough to compete with $\alpha$ for relevant ranges of $(M, C)$ is what makes the trade interesting. The curve flattens on easy problems and on the very-large-$C$ tail (diminishing returns set in), but in the middle range it is essentially linear-in-log.

The implication for deployment is real and the operational evidence is now everywhere. Test-time compute is now a deployment knob. o1’s API exposes reasoning_effort = low | medium | high. R1’s serving stack accepts max_thinking_tokens. Before 2024 there was no such knob; there was just the model. Cost-quality trade-off: $10\times$ to $100\times$ inference compute for hard (but not the very hardest) problems can outperform a $10\times$ larger model, and is often the better economic choice when the harder model would have to be trained, served, and maintained from scratch.

Modern reasoning models: o1, R1, Gemini Thinking

OpenAI o1

OpenAI released o1 in September 2024 as the first widely-deployed reasoning model. The system card and announcement covered the deployable behavior more than the training recipe, but the pattern is clear in the externally visible properties. o1 has an internal chain-of-thought, trained via RL, not a CoT prompt applied at inference, but a reasoning behavior baked into the weights. It thinks for variable time before answering, from roughly ten seconds on easy queries to several minutes on hard ones, with the deployment exposing this through the reasoning_effort knob. The results are substantially ahead of GPT-4 class models on math (AIME, IMO problems), code (Codeforces-level competitive programming), and science (PhD-level physics). o1 does not reveal its internal reasoning trace to the user, a design choice OpenAI made for a mix of competitive and safety reasons. The user sees only the final answer.

o1 itself is history now: OpenAI succeeded it with o3 in April 2025, then folded the reasoning line into the GPT-5 family (through GPT-5.6 as of mid-2026), each generation keeping the same “think before answering, RL-trained” shape while pushing the effort/quality knob further. o1 is still the cleanest first example of the paradigm, which is why it anchors this section; as of $2026$-$07$-$13$ it is no longer the frontier model, and this space moves fast enough that any specific model name here will be superseded again within the year.

DeepSeek R1

DeepSeek released R1 in January 2025 and the field’s center of gravity shifted within weeks. R1 matched o1’s reasoning ability on math and code benchmarks, open-source. The paper’s most surprising result actually belongs to an intermediate model, R1-Zero: apply rule-based outcome rewards (exact match for math; test-pass rate for code) directly to a base model via RLVR through GRPO (Chapter 14). No PRM. No SFT bootstrap at any stage. R1-Zero learned to generate long, effective reasoning traces from outcome reward alone, proof that SFT scaffolding was not strictly necessary to bootstrap reasoning. The released R1 builds on that result but is not itself pure-RL: DeepSeek added a small cold-start SFT stage before RL (and a further rejection-sampled SFT round afterward) specifically to fix R1-Zero’s readability and language-mixing problems, then continued training with rule-based rewards and GRPO. The result, R1, generates long reasoning traces, sometimes $10{,}000$+ thinking tokens, that produce correct answers, with the entire trace exposed in <think>...</think> tags so users can see what the model considered. DeepSeek also released distilled smaller models (1.5B to 70B; Chapter 16) that inherit much of R1’s reasoning capability, the recipe transfers via distillation, and small reasoning models became a practical option overnight.

The implication is the one that mattered most for the field: R1-Zero showed that a pure-outcome RL recipe, with no PRM and no SFT bootstrap, can produce a model that thinks effectively, and R1 showed that a small dose of SFT is enough to fix the rough edges without giving up the RL-driven gains. Before R1(-Zero) the assumed recipe for trained reasoning required PRMs and SFT scaffolding; afterward the field reconsidered whether either was necessary. The open weights and the published recipe meant anyone could reproduce the result, and many groups did within a quarter.

DeepSeek’s own production line has moved on from R1: DeepSeek-V4 (April 2026) is a hybrid thinking/non-thinking model built around a different, more compute-efficient attention architecture, and no separate “R2” has shipped as of $2026$-$07$-$13$. None of that changes why R1 is taught here: it remains the clearest open, reproducible instance of RLVR-plus-GRPO with no PRM and no SFT bootstrap, and every open reasoning model released since has followed the recipe it demonstrated.

RLVR and GRPO: what “trained via GRPO” actually means

Every mention of R1’s training in this chapter has deferred to Chapter 14 for the mechanics. Two ideas are worth unpacking here, because they are the two curriculum-level claims the R1 result rests on. RLVR (RL with Verifiable Rewards) is a reward-design choice: the reward for a rollout comes from a verifier that checks correctness directly, exact-match against a ground-truth number for math, a test suite passing for code, a proof checker accepting a derivation, rather than from a learned reward model trained on human preferences. There is no reward model in the loop at all; the check is deterministic and the same every time it runs.

GRPO (Group Relative Policy Optimization) is the optimizer R1 pairs with that reward. PPO’s advantage estimate needs a learned value function, a second trained model (the critic) that predicts expected return from a state, specifically so the policy update has a low-variance baseline to subtract. GRPO removes the critic entirely. For a given prompt, sample a group of $G$ completions from the current policy, score each one with the reward (the verifier, here), and use the group’s own mean reward as the baseline directly:

Completions that scored above the group average get positive advantage; completions that scored below it get negative advantage; this single scalar $A_i$ is broadcast to every token of completion $y_i$, so the credit-assignment question (“which token in this trace deserves the credit?”) gets resolved by ignoring it, every token in an above-average completion is pushed up equally. No value function is trained, no critic model exists, and the baseline costs nothing beyond computing a mean over $G$ numbers you already had. That is the “shift away from value models” the curriculum points at: pair RLVR’s verifier with GRPO’s group baseline and two of PPO’s models, the reward model and the critic, both disappear, leaving just the policy and a frozen reference to KL against. (20.grpo-advantage) is a simplified read of the objective; Chapter 14 derives the full clipped-surrogate GRPO loss, including the group standard-deviation normalization and the explicit KL term.

The aha moment

DeepSeek’s R1 paper reports something the training recipe never specified: as RLVR training on R1-Zero progressed, the model’s reasoning traces got measurably longer, and their character changed. The model started pausing mid-trace to reconsider a step, trying a second approach after the first one stalled, and explicitly re-checking an intermediate answer before committing to a final one, behavior the paper’s authors describe encountering as an “aha moment” when they read the transcripts. None of this was demonstrated anywhere in training. RLVR’s reward only ever checked the final answer; no SFT data showed the model what backtracking looks like, and no prompt template asked for it. The behavior emerged because outcome reward pushed up the token probabilities of whichever sampled completions happened to land on the correct answer, and some of the completions that happened to land correctly also happened to backtrack along the way.

The paradigm shift

Gemini Thinking (December 2024) is Google’s variant of the same paradigm, internal reasoning trained via RL; thinking traces exposed to the user; similar performance profile on math and code. The exact training details differ but the shape is the same as R1’s and the user-facing behavior is similar. Google’s reasoning line has since become Gemini 3 Deep Think, a heavier reasoning mode layered on Gemini 3 Pro; Claude has converged on the same idea too, with Anthropic’s extended thinking (an effort parameter toggling how much internal reasoning the model spends before answering) shipping through the Opus 4.x line. Three labs converging on “train the model to think longer, expose or hide the trace, expose an effort knob” is the strongest evidence that this is a paradigm and not a one-off trick, and it is why the RLVR and GRPO mechanics covered above are worth learning in general rather than as trivia about any one 2024–2025 model.

The shift from classic to modern is sharp enough to state directly. Classic era: CoT was a prompt applied at inference. Same model, different prompt; the reasoning was something the model could be coaxed into doing. Modern era: reasoning is trained in via RL. The model autonomously decides when and how long to think. The trace lives behind a structural marker (<think>...</think> for R1, hidden for o1) and the model emits as much or as little of it as the problem warrants.

The compact form:

The model emits the reasoning autonomously; the user sees the answer (o1) or both (R1, Gemini Thinking). The bridge back to earlier chapters is direct. Long reasoning traces stress the KV cache aggressively, $10{,}000$+ thinking tokens per query is routine for hard problems, and that memory pressure is what makes PagedAttention (Chapter 17) essential for serving reasoning models efficiently. Some production stacks combine reasoning models with KV cache quantization (Chapter 18) to keep the memory cost tractable. The deployment engineering of Part VI is what made the modern reasoning paradigm shippable.

Exercises

Four exercises that lock in the chapter’s two eras. Each is a self-contained problem with a starting template; hints are collapsed by default; try the problem first.

The exercises span the chapter’s arc: from prompting-era techniques (Ex 1, Ex 2), to verifier-augmented inference (Ex 3), to the test-time compute scaling math that justifies modern reasoning models (Ex 4).

Exercise 1 (easy): Zero-shot CoT and answer extraction

Implement a function that takes a question, applies zero-shot CoT, and extracts a numeric answer from the trace. Test on a few different problem statements; verify that the extractor handles varied trace formats.

import re

def zero_shot_cot_prompt(question):
    return question + " Let's think step by step."

def extract_numeric_answer(trace):
    m = re.search(r"answer is (\d+(?:\.\d+)?)", trace.lower())
    if m:
        return float(m.group(1)) if "." in m.group(1) else int(m.group(1))
    m = re.search(r"answer:\s*(\d+(?:\.\d+)?)", trace.lower())
    if m:
        return float(m.group(1)) if "." in m.group(1) else int(m.group(1))
    nums = re.findall(r"\d+(?:\.\d+)?", trace)
    if not nums:
        return None
    last = nums[-1]
    return float(last) if "." in last else int(last)

Exercise 2 (medium): Self-consistency aggregation

Implement self-consistency: given N traces (with varied final answers), return the majority answer and confidence. Verify the wisdom-of-crowds effect on a noisy trace pool.

from collections import Counter

def self_consistency(traces):
    answers = [extract_numeric_answer(t) for t in traces]
    answers = [a for a in answers if a is not None]
    if not answers:
        return None, 0.0
    counts = Counter(answers)
    majority, count = counts.most_common(1)[0]
    return majority, count / len(answers)

Exercise 3 (medium): Best-of-N with PRM scoring

Implement best-of-N with a verifier (mock PRM). Compare three strategies on the same trace pool: (a) random single trace, (b) self-consistency majority vote, (c) best-of-N + PRM. Verify that (c) outperforms (b) when the PRM is informative.

def mock_prm_score(trace):
    score = 0.0
    score += min(len(trace) / 100, 1.0) * 0.4        # length, capped
    math_ops = sum(trace.count(op) for op in ['=', '/', '+', '-'])
    score += min(math_ops / 4, 1.0) * 0.4              # math density, capped
    lower = trace.lower().strip()
    if not (lower.startswith("i think") or lower.startswith("maybe") or "guess" in lower):
        score += 0.2                                    # confident phrasing
    return score

def best_of_n_prm(traces):
    best_trace = max(traces, key=mock_prm_score)
    return best_trace, extract_numeric_answer(best_trace)

Exercise 4 (hard): Test-time compute scaling math

Derive the majority-vote accuracy curve analytically and reproduce the marquee widget’s pattern. Given a single-trace accuracy $p$, compute the probability that majority vote of $N$ traces is correct (assuming independence and a single dominant wrong answer). Plot accuracy vs $N$; observe the saturation pattern.

import math

def majority_vote_accuracy(p, n):
    return sum(
        math.comb(n, j) * p**j * (1 - p) ** (n - j)
        for j in range(math.ceil(n / 2), n + 1)
    )

The full picture

When to use which technique

The chapter covered eight techniques across two eras. The decision rule across them is straightforward: use the simplest technique that works, escalate only when it does not. The map:

For most chat tasks, modern instruction-tuned models with zero-shot CoT suffice: the model already produces step-by-step reasoning when the prompt invites it, and self-consistency is unnecessary at the accuracy level chat users care about. For hard reasoning, competition math, code with subtle bugs, multi-step science problems, trained reasoning models are now the default. o1 and R1 are the models that defined this category and are still the cleanest teaching examples of the recipe; the frontier has since moved to their successors (OpenAI’s o3/GPT-5 line, DeepSeek’s V4, Gemini 3 Deep Think, Claude’s extended thinking), all running the same “think longer, autonomously, via RL” pattern. They are expensive per query and slow at the high-effort setting, but the alternative is failure on problems that direct generation simply cannot reach.

What’s next in Part VII

Part VII is four chapters and reasoning is the first one. The trajectory:

• Chapter 21 (tool use) extends the ReAct pattern of section 4 into engineered systems. Function calling, structured action parsing (the constrained decoding of Chapter 19 does its operational work here), tool-error handling, and the agent loop.
• Chapter 22 (RAG) covers retrieval-augmented generation. The model looks up relevant context from a vector store and conditions on it; ReAct-style search is a special case.
• Chapter 23 (multimodal) covers vision, audio, and video, how the same transformer backbone extends to non-text inputs.
• After Part VII: safety, interpretability, and evaluation (Part VIII); then agents (Part IX), where reasoning, tools, retrieval, and multimodal all combine into the systems the curriculum has been building toward.

Reasoning is the capability that turns deployable models into useful systems. The classic era, CoT, self-consistency, ToT, ReAct, discovered the techniques that work on any model. The modern era, PRMs, test-time compute scaling, o1, R1, Gemini Thinking, trained reasoning directly via RLVR, with test-time compute as a first-class deployment knob; o1 and R1 defined that era, and every reasoning model shipped since, up through 2026’s o3/GPT-5 line, DeepSeek V4, Gemini 3 Deep Think, and Claude’s extended thinking, still runs the recipe they established. Both eras remain relevant: CoT prompting still works on any model and is essentially free; modern reasoning models are state-of-the-art for hard tasks and are now the default when accuracy on competition-level problems matters more than per-query cost.

Where reasoning gave the model time to think, tools give it the ability to act, call APIs, search the web, execute code, retrieve information from systems the model itself does not know about. The ReAct pattern this chapter introduced is the foundation; Chapter 21 covers the engineering, and Chapter 22 covers RAG while Chapter 23 covers multimodal systems. By the end of Part VII the curriculum has a model that reasons, uses tools, retrieves knowledge, and handles non-text inputs, the full capability stack of a modern AI system, with Parts VIII and IX layering safety, evaluation, and agency on top.