Agent foundations
Agent foundations, the chapter that turns a chat model into an actor. From the operational definition of an agent (LLM controller + tools + loop), through the canonical agentic loop (observe → think → act → observe), ReAct as the foundational pattern (Yao 2022), the 2023 agent boom (AutoGPT, BabyAGI), memory and state, common patterns and anti-patterns, and the 2025 agentic stack (Anthropic MCP, LangGraph, OpenAI's since-deprecated Assistants API). Opens Part IX, the curriculum's final arc, agents from scratch, multi-agent orchestration, and agent evaluation frameworks closing the curriculum.
Part VIII closed with three disciplines complete: safety (what we want), interpretability (what’s there), evaluation (how we measure). Part IX opens here, and Part IX is about composition: capable, disciplined models assembled into systems that observe, think, act, and iterate. This chapter is the foundational one: what an agent is, what the loop looks like, what worked and what didn’t in the 2023 boom, and where the field stands in 2025.
An agent is an LLM acting as a controller in a loop with an environment. That is the operational definition this chapter builds on. The LLM observes; thinks; acts (by invoking a tool or producing output); observes again; repeats until done. ReAct (Yao ) is the canonical pattern, interleave reasoning (chain-of-thought from Ch 20) with actions (tool calls from Ch 21). AutoGPT was the viral moment that proved both the promise and the limits. The reality is bounded autonomy with human oversight: production agents like Claude Code, Cursor, Devin, GitHub Copilot Workspace.
The chapter is engineering with honest limits. Agents work well for narrow, well-defined tasks: GAIA scores at – for frontier agents; SWE-bench Verified at ~; meaningful production deployments. Open-ended autonomous agents remain unsolved, the framing has given way to ‘s bounded-autonomy framing. By the end of this chapter, you’ll have the conceptual toolkit and the historical context. Then Ch 28 builds agents from scratch; Ch 29 composes multiple agents; Ch 30 evaluates them and closes the curriculum.
What is an agent?
The operational definition
The chapter’s central frame is one short equation, and the rest is unpacking it.
An agent is an LLM acting as a controller in a loop with an environment. The LLM observes the current state (initial prompt plus any prior observations); thinks about what to do next; acts by invoking a tool or producing output; observes the result of that action; and repeats until done. Five steps, one feedback loop, one LLM in the controller seat. Every subsequent agentic concept in this chapter, patterns, anti-patterns, frameworks, is some elaboration of those three components.
Distinctions that matter
The word agent gets overloaded in industry conversation. The distinctions worth holding onto are operational, not philosophical.
| System | Defining feature | Example |
|---|---|---|
| Chat model | One prompt → one response | ”Capital of France?” → “Paris.” |
| Tool-using LLM | One prompt → tool calls within the response | ”Weather in Tokyo?” → calls weather tool |
| Agent | Iteration and decision-making across multiple turns | ”Book me a flight to Tokyo for next Tuesday under ”, searches, compares, drafts, confirms |
The defining feature is iteration and decision-making. An agent decides what to do next; a chat model just responds; a tool-using chat model is still single-turn. The distinguishing capability is the loop wrapped around the call, not the call itself.
The empirical scale in early is worth keeping in mind. GAIA: frontier agents – on multi-step real-world tasks (humans about ). SWE-bench Verified: frontier agents around on real GitHub issues. Production deployments include Anthropic Claude Code, GitHub Copilot Workspace, Cursor, Devin, Aider, and OpenAI’s o-series with tool use. In the open-source ecosystem, LangChain sits at roughly GitHub stars and AutoGPT at roughly .
The agentic loop
The canonical loop
The shape of the agentic loop is the same across every framework, every paper, every production system.
┌─────────────────┐
│ Initial prompt │
└────────┬────────┘
│
▼
┌─────────────────┐ ┌──────────────────┐
│ Observation │────▶│ LLM controller │
└─────────────────┘ │ (reason/decide) │
▲ └────────┬─────────┘
│ │
│ ▼
│ ┌──────────────────┐
│ │ Action (tool │
│ │ call OR output) │
│ └────────┬─────────┘
│ │
└───────────────────────┘
(next observation)
Components
Five components carry the loop, and the engineering complexity of agents lives in how each is implemented. Initial state: the user’s request plus any system context. Controller call: the LLM reasons about the current state and decides on an action. Action execution: a tool is called, code is run, or a final answer is produced. Observation: the result of the action becomes the next iteration’s input. Termination: the agent decides it is done, or hits a limit, or fails.
Termination
Termination is where most agent failure modes live. The clean termination conditions are easy to enumerate. Success: the agent produces a final answer. Max iterations: a hard cap prevents runaway loops. Budget exhausted: a token or cost limit is reached. Error: an unrecoverable tool failure that the agent cannot work around. Human interrupt: the user cancels.
Why termination is hard, in three points. Premature termination: the agent gives up before completing the task, declaring victory at the first plausible answer. Late termination: the agent loops forever, refining unnecessarily, polishing what was already done. Hallucinated success: the agent claims to be done when it is not, fabricating progress that does not match the observable state of the environment.
The deceptively simple structure of the loop hides most of the engineering complexity in agents. The loop itself is a dozen lines; making it work reliably is months of edge-case handling. Every production agent team learns this the hard way. The naive implementation runs end-to-end on the first attempt and looks correct; the production-ready version comes after months of triaging edge cases, flaky tools, malformed tool outputs, ambiguous user requests, runaway iteration counts, context windows filling on long sessions, cost spikes when an obvious off-ramp is missed. None of those are interesting individually; together they are most of what “agent engineering” actually means in .
ReAct: the foundational pattern
Interleaving reasoning and acting
ReAct (Yao et al. ) is the canonical agentic pattern: interleave reasoning (chain-of-thought from Ch 20) with actions (tool calls from Ch 21). The agent produces structured output where each step is a Thought, an Action, and the resulting Observation.
Thought: I need to find the current weather in Tokyo.
Action: weather_lookup(city="Tokyo")
Observation: 18°C, partly cloudy
Thought: Good. Now I need today's date for context.
Action: get_current_date()
Observation: 2025-05-22
Thought: I have what I need. The weather in Tokyo today (May 22, 2025) is 18°C and partly cloudy.
Action: final_answer("Today in Tokyo: 18°C, partly cloudy.")
Why it works
Four properties make ReAct the default. Reasoning grounds actions: the agent thinks about why it is calling a tool before calling it, making the call less arbitrary. Actions ground reasoning: observations correct any mistakes in the model’s thinking, so reasoning errors do not compound across turns. Composability: multi-step tasks decompose naturally into thought-action-observation triples, with no extra structure required. Inspectability: the trace is human-readable; failures are diagnosable from the log.
A minimal ReAct agent fits in fifty lines of Python, mock LLM, mock tools, regex parsing, full loop with termination. The point is to see that the loop itself is short; the LLM and parsing handle the complexity.
Variations
ReAct is the default but not the only structure. Plan-and-execute produces a full plan first and then executes the steps, less iterative, more structured, useful when the plan space is bounded. Reflexion (Shinn ) adds a self-critique step: after each failure, the agent writes a verbal reflection that goes into context for the next attempt. Tree-of-Thoughts (Yao ) branches the thought process, exploring multiple reasoning paths before committing to an action.
Why ReAct stays the default in production: it’s simple to implement and works with any LLM that supports chain-of-thought and tool calls. The output is human-readable, which makes failure modes visible. And it needs no exotic infrastructure, most production agents are ReAct with engineering polish on top.
From ReAct to AutoGPT: the 2023 boom
The viral moment
In March-April , AutoGPT was released as an open-source project. It quickly became the most-starred GitHub repo of the year, currently around stars. The pitch was direct: tell an LLM your goal, and it autonomously breaks the goal into sub-tasks and executes them. The architecture was a five-step loop. The user specifies a goal. The LLM generates an initial task list. The LLM picks the next task and executes it with tools. The LLM updates the task list based on the result. The loop repeats until the LLM decides the goal is met.
What worked, what didn’t
Four things worked and stayed. Task decomposition: the planner-executor pattern proved durable. Tool integration: file I/O, web browsing, code execution were stitched together cleanly. The vision: autonomy as a useful framing for many real-world tasks resonated. Open-source momentum: the project sparked the broader agent ecosystem (LangChain, LangGraph, BabyAGI, CrewAI).
Six things did not work and had to be re-engineered. Unbounded loops: agents refined endlessly without finishing. Hallucinated successes: agents claimed “done” without actually completing the task. Context bloat: long task histories overflowed the context window. Cost: running agents until completion was expensive on models. Error recovery: agents struggled to recover from tool failures, often aborting on the first non-trivial error. No grounding: agents generated plans disconnected from real-world constraints.
The lessons that survived
Five engineering lessons came out of the boom and shape every modern agent. Planner-executor decomposition: separating planning from execution makes both more tractable. Memory stores: agents need external memory to handle long horizons. Structured tool calls: JSON schemas instead of free-text actions. Bounded iteration: explicit max-step limits prevent runaway loops. Human-in-the-loop checkpoints: letting humans approve key decisions improves reliability dramatically.
Where the field went from there: from end-to-end autonomous agents back to bounded agents with human oversight. The production-grade agent looks more like Claude Code (developer-supervised, narrow scope, observable trace) than AutoGPT (fully autonomous, broad scope, opaque internals). The framing shift is engineering maturity, not a retreat, every lesson AutoGPT taught about reliability shows up in the architecture of the production agents that succeeded it.
Memory and state
Short-term vs long-term
LLMs have no persistent state across calls. Each agentic loop call is a fresh API request. State must be reconstructed each time, usually by passing it back into the prompt. That single property is what makes agent memory an engineering problem rather than a model property.
Short-term memory is in-context. The conversation history, all prior turns in the current loop, sits inside the prompt. Pros: simple; the LLM sees everything; reasoning has full context. Cons: the context window is finite; long conversations get truncated; cost scales with history length.
Long-term memory is external, and decomposes into four patterns. Vector databases (Ch 22) store past observations as embeddings; retrieval is by semantic similarity. Summary memory periodically summarizes older history into a shorter representation. Structured memory is a key-value store for facts the agent should remember verbatim. Episodic memory retrieves past complete task traces when a similar task recurs.
When long-term memory earns its complexity: long-horizon tasks, anything that will not fit in a single context window; persistent agents, agents that operate across multiple user sessions; self-improvement, Reflexion-style learning from past failures; personalization, agents that adapt to user preferences over time.
The memory tradeoff
More memory means more context, which means more cost and more potential confusion. Less memory means smaller context, which is faster and cheaper but may forget important details. The right amount of memory is task-specific; there is no general answer. The pattern in production is to start short-term-only, add structured memory for facts that must persist, and add retrieval-based long-term memory only when context-window limits force it.
Patterns and anti-patterns
Common patterns
Five patterns recur across modern agent systems, and the choice of pattern depends on the task. Single-agent linear: one agent, one task, one execution path; right for simple tasks where the path is obvious. Single-agent iterative (ReAct): one agent, looping; right for multi-step tasks with unknown depth. Hierarchical (planner + executor): one agent plans, another executes; right for complex multi-step workflows where planning matters. Reflexion (Shinn ): self-critique loops that iterate on failures; right for tasks where iteration improves performance. Multi-agent (preview of Ch 29): multiple agents with specialized roles; right for tasks that genuinely decompose into expertise.
Anti-patterns
Seven anti-patterns recur often enough to enumerate. Unbounded loops: no max-iteration cap; the agent loops forever. Hallucinated tool outputs: the agent fabricates what a tool would return instead of calling it. Context bloat: every observation appended; the context overflows. Over-decomposition: the agent generates a -task plan for a simple problem. No error handling: a tool fails; the agent does not recover; the task aborts. Premature termination: the agent claims done when it is not. Hallucinated progress: the agent claims to have completed steps without actually doing them.
The fixes are not exotic. Max iteration caps with cost budgets. Structured tool calls with JSON schemas instead of free-text actions. Rolling-window or summarized context. Verification steps before the final-answer action. Retry-with-backoff for transient tool failures. Each anti-pattern has a known mitigation; the engineering work is wiring them together.
Defense-in-depth
The framing from Chapter 24’s safety chapter applies here too: no single safeguard catches every failure mode. Production agents stack defenses. Layered defenses: max iterations plus cost cap plus output validation plus human-in-the-loop checkpoints. Observability: every thought, action, and observation logged; alerts on unusual patterns like high iteration counts or repeated tool failures. Bounded autonomy: production agents act within constrained scopes rather than open-ended ones.
The agentic stack today
The 5-layer stack
The modern agentic stack stratifies into five layers, and engineers building production agents touch most of them.
Layer 1, Models. Frontier LLMs with strong tool use: Claude Sonnet/Opus, GPT- family, Gemini Pro. Specialized fine-tunes appear at the edges, Claude Code, GitHub Copilot models, Cursor’s tunes.
Layer 2, Tool protocols. MCP (Model Context Protocol) is Anthropic’s open standard for tool and data connections, the closest thing to a USB-C for agents. Function calling is OpenAI’s structured-output approach; it is widely adopted across providers.
Layer 3, Agent frameworks. LangGraph offers graph-based agent orchestration with explicit state machines. CrewAI is a multi-agent collaboration framework. AutoGen (Microsoft) is a multi-agent conversation framework. Most production agents are bespoke; frameworks handle the loop scaffolding, not the domain.
Layer 4, Hosted services. Anthropic Claude Code is a coding agent with file and terminal access. OpenAI’s Assistants API was a managed agent service with built-in tools, but OpenAI deprecated it in August in favor of the Responses API (paired with a Conversations API for thread and state management); the Assistants endpoints are scheduled for full shutdown on August 26, . As of mid- that shutdown is weeks away, and the Assistants API should be read as superseded, not as a current option. This layer reshuffles fastest, so check the current state before building on it. OpenAI o1 / o3 are reasoning-first models with implicit agentic structure. Cursor, Aider, Continue are coding-focused agent products. Devin is Cognition AI’s autonomous SWE agent.
Layer 5, Production patterns. Bounded autonomy: agents act within constrained scopes with human oversight. Tool-use first, agency second: most production “agents” are tool-using LLMs with light iteration, not autonomous decision-makers. Specialized beats general: domain-specific agents (coding, customer support, data analysis) outperform general-purpose ones in deployment, because the scope constraint is what makes oversight tractable.
Where the field is
Production agents work well in constrained, well-defined domains: coding assistance, customer service, data lookup. They struggle with open-ended, long-horizon tasks: writing a startup business plan from scratch, conducting independent research, planning a multi-week project without supervision. Anthropic, OpenAI, and Google are all investing heavily; frontier capability is moving fast, and the discriminating benchmarks have shifted from MMLU and HumanEval to SWE-bench, GAIA, and OSWorld. The autonomous-AGI framing of has given way to the bounded-agent-with-oversight framing of . That shift is engineering maturity, not a capability ceiling, and it is what makes the production stack work.
Exercises
Four exercises that lock in agent foundations. Each is a self-contained problem with a starting template; hints are collapsed by default; try the problem first.
The exercises trace the chapter’s arc: build a minimal ReAct agent (Ex 1) → add Reflexion-style memory (Ex 2) → add error handling with retries (Ex 3) → design your own agent for a given task (Ex 4). After these, the reader is ready for Ch 28’s engineering deep-dive.
Exercise 1 (easy): Minimal ReAct agent
Implement a minimal ReAct agent that loops over a task, calling tools and parsing thought/action output until it produces a final answer.
Hint
The ReAct loop structure:
- Maintain a
historylist of strings - Each iteration: pass history to the LLM; parse Thought + Action
- If Action is
final_answer(...), return its content - Otherwise: execute the action via the tools dict; append observation to history
- Cap iterations to prevent runaway loops
Real implementations use JSON / function-calling instead of regex parsing, but the loop structure is identical.
Solution
The loop builds the prompt from history, calls the LLM, parses Thought/Action, and branches on whether the action is final_answer; otherwise it executes the tool and appends the observation:
def react_agent(task, tools=TOOLS, max_iterations=8):
history = [f"Task: {task}"]
for _ in range(max_iterations):
prompt = '\n'.join(history)
response = mock_llm(prompt)
thought, action = parse_react(response)
history.append(f"Thought: {thought}")
history.append(f"Action: {action}")
print(f"Thought: {thought}")
print(f"Action: {action}")
if action.startswith('final_answer'):
match = re.search(r'final_answer\("(.+)"\)', action)
return match.group(1) if match else action
observation = execute_action(action, tools)
history.append(f"Observation: {observation}")
print(f"Observation: {observation}")
return "Max iterations reached."Running react_agent("What's the weather in Tokyo, and what date is it today?") traces three iterations: weather("Tokyo") -> "18°C, partly cloudy in Tokyo", date() -> "2025-05-22", then final_answer(...), returning "Today (May 22, 2025) in Tokyo: 18°C, partly cloudy." Every fact in the answer came from an observed tool call, not a fabrication.
Exercise 2 (medium): Agent with Reflexion-style memory
Add a Reflexion-style memory that captures verbal critiques after failures and feeds them into future attempts.
Hint
The Reflexion pattern:
- Agent attempts a task
- If it fails (e.g., test doesn’t pass), agent writes a verbal reflection (“what I did wrong, what to try next”)
- On retry, the reflection is added to the system prompt
- Each subsequent attempt sees all prior reflections
For this exercise, mock the “verifier” with a hardcoded pass/fail. Real Reflexion uses programmatic verification (test results, compiler errors, etc.).
Solution
Canned reflections keyed on how many the agent has already written: attempt 1 fails with the first canned line, attempt 2 fails with the second, and attempt()’s mock succeeds once len(self.reflections) >= 2:
def reflect_on_failure(self, task, failed_attempt):
if len(self.reflections) == 0:
reflection = "I tried X but it didn't work. Next time I should try Y."
else:
reflection = "Y was closer but still wrong. The actual issue is Z."
self.reflections.append(reflection)reflexion_run(...) then succeeds on attempt 3: attempt 1 fails and adds reflection 1, attempt 2 fails (sees 1 reflection) and adds reflection 2, attempt 3 sees 2 reflections and attempt() returns ("Solution to '...'", True).
Exercise 3 (medium): Error handling with retry
Add robust error handling to an agent: catch tool failures, retry with exponential backoff, and gracefully fall back when retries are exhausted.
Hint
The error handling pattern:
- Wrap tool calls in try/except
- On exception, retry with exponential backoff (1s, 2s, 4s…)
- After max retries, return a structured error
- The agent should be able to continue (try a different tool, or report failure)
Real production agents also use: circuit breakers (stop hitting a failing tool entirely after N failures), fallback tools, and structured error categorization (retryable vs not).
Solution
Retry inside a loop, sleeping with exponential backoff between attempts, returning (None, error) once the budget is exhausted:
def execute_with_retry(tool, args, max_retries=3, base_wait=0.001):
for attempt in range(max_retries):
try:
return (tool(args), None)
except Exception as e:
wait = base_wait * (2 ** attempt)
time.sleep(wait)
return (None, f"Failed after {max_retries} attempts")With random.seed(42) and this plan, robust_agent reports: search('Bhutan population') succeeds, search('Bhutan area km') exhausts its 3 retries and fails, calc('787000 / 38394') succeeds (20.4979944783039), and search('population density meaning') succeeds, a 3/4 summary. The exact pass/fail pattern depends on random’s internal state consumption (each retry attempt burns one random.random() call), so it is deterministic given the seed but not “obviously” predictable by inspection; that’s the point of exponential backoff plus retry budgets: some flaky calls recover, some don’t, and the agent keeps going either way.
Exercise 4 (hard): Design-your-own-agent
For each of three different tasks, pick the most appropriate agent pattern and justify your choice. Implement a sketch of one of them.
Hint
The mapping from task → pattern:
- One-shot lookups → Linear
- Multi-step tasks of unknown depth → ReAct iterative
- Complex workflows that benefit from planning → Hierarchical
- Tasks with verifiable feedback → Reflexion
- Tasks that genuinely decompose into specialized roles → Multi-agent
Each task below has a “right” pattern (not unique, multiple can work). Read the task carefully; what does it require?
Solution
Apply the heuristics in the order given, checking the most specific signal (reflexion keywords) first and the chained-steps signal (“then” / multiple commas) before the generic one-tool-lookup signal, so a multi-step task never gets misclassified as linear:
def choose_pattern(task_description):
d = task_description.lower()
if any(kw in d for kw in ['fix', 'retry', 'test', 'iterate']):
return 'reflexion'
if 'then' in d or d.count(',') >= 1:
return 'react'
if any(kw in d for kw in ['plan', 'build', 'design a full system']):
return 'hierarchical'
if any(kw in d for kw in ['a researcher', 'a critic', 'a writer']):
return 'multi-agent'
if 'look up' in d or 'find' in d or 'tell me' in d:
return 'linear'
return 'react'
def sketch_react_for_task_2():
population = 787000
area = 38394
print("Turn 1: think -> search_population_bhutan")
print(f" Observation: {population}")
print("Turn 2: think -> search_area_bhutan")
print(f" Observation: {area}")
density = population / area
print("Turn 3: think -> calculator(787000 / 38394)")
print(f" Observation: {density:.2f}")
print("Turn 4: think -> final_answer")
return f"Bhutan's population density is approximately {density:.2f} people per square kilometer."All three tasks classify correctly: task_1 -> linear (no chain, a lookup verb), task_2 -> react (contains “then”), task_3 -> reflexion (contains “fix” and “test”). sketch_react_for_task_2() returns “Bhutan’s population density is approximately 20.50 people per square kilometer.” (787000 / 38394 = 20.4979944783039, rounded to 20.50.)
Part IX: chapter map
The composition arc
Part IX is the curriculum’s final arc, and its name is composition, assembling capable, disciplined models into systems that observe, think, act, and iterate. The four chapters trace a deliberate path from concept to system.
| Chapter | Topic | What it covers |
|---|---|---|
| Ch 27 (this chapter) | Agent foundations | The agentic loop, ReAct, AutoGPT, memory, patterns |
| Ch 28 | Agents from scratch | Building real agents: tool implementation, error recovery, scaffolding |
| Ch 29 | Multi-agent | Orchestration, agent-to-agent communication, role specialization |
| Ch 30 | Agent eval and frameworks | How to evaluate agents (connecting back to Ch 26); production frameworks; the curriculum’s close |
The three parts of this curriculum’s later arc fit together cleanly. Part VII (Capabilities): what individual models can do. Part VIII (Disciplines): making capable development trustworthy. Part IX (Composition): assembling models into systems.
Why agents close the curriculum
Agents are the highest level of composition the curriculum addresses, combining capability (Part VII), discipline (Part VIII), and orchestration into systems that act in the world rather than just answer questions about it. After Ch 30, the reader has the full stack from numpy primitives (Ch 1) to production agent systems. The trajectory of this chapter into Ch 28 is also deliberate: Ch 27 is conceptual, what an agent is and how the loop works; Ch 28 is engineering, actually building agents end-to-end with real tools.
An agent is an LLM acting as a controller in a loop with an environment. That is the operational definition. The loop observes, thinks, acts, and observes again, until termination. ReAct interleaves reasoning with action and produces inspectable traces. AutoGPT proved the pattern’s promise and revealed its limits; the agentic stack, Anthropic Claude Code, OpenAI’s since-deprecated Assistants API, LangGraph, Cursor, Devin, runs on the lessons that survived. Memory and state turn single-shot loops into persistent agents. Patterns and anti-patterns turn experiments into production. Bounded autonomy with human oversight turns agents from research demos into reliable products. The capability is the loop, not the model.
Ch 28 builds agents end-to-end from real tools. Ch 29 composes multiple agents into orchestrated systems. Ch 30 evaluates agents, closing both the eval discipline from Ch 26 and the curriculum itself.