An AI agent that works in a demo and an AI agent that works in production are different systems. This guide covers what actually breaks — compounding step failure, tool design, silent partial success, unbounded loops, context growth — and the evaluation and guardrail work that separates the two.
Building an AI agent that works once is a weekend project. Building one that works on the four-hundredth request, on an input nobody anticipated, against a third-party API that has started returning 503s, without quietly corrupting a customer record — that is a different discipline entirely. The gap between those two systems is where most agent projects die, and it is almost never a modelling problem.
The demo version is genuinely impressive. You define a few tools, write a system prompt, and watch the model reason its way through a multi-step task. It looks like the hard part is done. It isn't. What you have built is the happy path, and the happy path is perhaps five per cent of the work. The remaining ninety-five is error handling, evaluation, guardrails, cost control, and observability — the unglamorous engineering that determines whether the agent is an asset or a liability.
This guide is about that ninety-five per cent. It covers what an agent actually is at the mechanical level, why reliability degrades so sharply with the number of steps, how to design a tool surface the model can use correctly, the specific failure modes that show up in production and rarely in testing, and how to evaluate and constrain an agent so that its mistakes are survivable.
Strip away the marketing and an agent is a loop with four steps. You send the model a conversation history plus a set of tool definitions. The model responds either with a final answer or with one or more structured tool call requests. Your code executes those calls against real systems and appends the results to the conversation. You send the whole thing back. Repeat until the model returns an answer instead of a tool call.
That is the entire mechanism. There is no autonomy in the model itself — the model emits a structured request, and your harness decides whether to honour it. This distinction matters more than almost anything else in agent engineering, because it means every safety property, every retry policy, and every audit trail lives in code you control, not in prompt text you hope the model respects.
Three consequences follow directly from the loop's shape, and they explain most of what goes wrong later:
Not everything that touches an LLM should be an agent, and treating a deterministic problem as an agentic one is a common and expensive mistake. The useful distinction is whether the sequence of steps is knowable in advance.
| Shape | When it fits | Control flow |
|---|---|---|
| Single call | Classification, extraction, summarisation, rewriting — one input, one output | None; your code calls once and uses the result |
| Workflow | Multi-step process where you know the steps: extract → validate → enrich → write | Your code. The LLM handles individual steps; the sequence is hard-coded |
| Agent | The steps depend on what earlier steps discover, and the task is hard to fully specify up front | The model. Your code executes and constrains |
The bias should be toward the simplest tier that solves the problem. A workflow is cheaper, faster, easier to test, and dramatically easier to debug than an agent, because its control flow is written in a language with a stack trace. Reach for an agent when the branching genuinely cannot be enumerated — investigating a support ticket where the next lookup depends on what the last one returned, or reconciling records across systems where the discrepancy type is unknown at the start.
Four questions worth answering honestly before committing: Is the task genuinely open-ended? Does the outcome justify the latency and cost? Is the model actually competent at this class of task? And can errors be detected and recovered from? A "no" on any of them is a signal to move down a tier.
The single most useful piece of arithmetic in agent engineering is the compounding of per-step reliability. If each step in a task succeeds independently with probability p, the probability that an n-step task completes correctly is pn. The numbers are unforgiving:
| Per-step accuracy | 5 steps | 10 steps | 20 steps | 50 steps |
|---|---|---|---|---|
| 90% | 59% | 35% | 12% | 0.5% |
| 95% | 77% | 60% | 36% | 8% |
| 99% | 95% | 90% | 82% | 61% |
| 99.9% | 99.5% | 99% | 98% | 95% |
This is why "the model isn't good enough" is usually the wrong diagnosis. A 95%-per-step agent failing 64% of twenty-step tasks does not need a smarter model — it needs the per-step number moved from 95% to 99%, and almost all of that improvement comes from engineering rather than from model selection. Tighter tool schemas, better error messages fed back into the loop, validation before execution, and retry logic that actually converges all move that number.
It also explains why shortening tasks is such a powerful lever. Collapsing three tool calls into one well-designed tool does not just save latency and tokens; it removes two independent opportunities for failure. The most reliable agents in production tend to have fewer, more capable tools rather than many granular ones — the opposite of what a clean API design instinct suggests.
The independence assumption in that table is also optimistic in one direction and pessimistic in another. Failures correlate: an agent that misreads a schema early will keep misreading it. But agents also self-correct — a tool error returned clearly to the model is frequently recovered from on the next turn. That recovery capability is not free; it exists only if your harness returns errors as information rather than swallowing them.
Most teams spend their optimisation effort on the system prompt. The higher-leverage surface is almost always the tool definitions. Tool names, descriptions, parameter schemas, and returned error strings are the interface through which the model perceives and acts on your systems — and they are read by the model on every single turn.
The most common defect in a tool definition is a description that documents the function without saying when to reach for it. "Searches the customer database" tells the model what the tool does and nothing about when it applies. "Look up a customer by email or account ID. Call this before any operation that modifies customer data, to confirm the account exists and is active. Do not use this to search by name — use search_customers_by_name instead." tells the model how to behave.
Being prescriptive about trigger conditions measurably improves how often a tool is called at the right moment. It also does the reverse job: stating when not to use a tool is the cleanest fix for an agent that over-reaches for the wrong one.
Every constraint you can express in the input schema is a constraint the model cannot violate. Enums beat free-text strings. Required fields beat optional ones with implied defaults. A parameter typed as {"type": "string", "enum": ["pending", "shipped", "cancelled"]} cannot receive "in transit"; the same parameter typed as a bare string will eventually receive exactly that.
Several providers also support strict schema enforcement, which guarantees that tool inputs validate against the declared schema rather than merely tending to. Where it is available, use it — it eliminates an entire class of parse-and-retry code.
This is the single change that most improves agent recovery behaviour. When a tool fails, the failure should be returned into the conversation as a tool result flagged as an error, containing a message written for the model to act on. Not a stack trace. Not "Error: 400". Something like: "No customer found with email 'j.smith@exmaple.com'. Check the spelling of the domain, or use search_customers_by_name if you only have a name."
An agent given that message will usually fix the typo and continue. An agent given a raw 400 will typically retry the identical call two or three times and then give up or invent an answer. The error string is part of your prompt engineering surface, and it is the part almost nobody edits.
Beyond roughly twenty to thirty tools, selection accuracy starts to degrade — the model has more opportunities to pick a plausible-but-wrong option, and every schema consumes context on every turn. Where a large tool library is genuinely required, tool-search mechanisms that load only relevant schemas on demand are now available from major providers and are strongly preferable to loading everything.
The other structural fix is consolidation. Two tools whose descriptions are hard to tell apart are a bug: either merge them, or rewrite both descriptions to state the boundary explicitly. If a human engineer would have to read the implementation to choose between them, the model has no chance.
These are the specific behaviours that rarely appear in a demo and reliably appear at volume.
| Failure mode | What it looks like | Mitigation |
|---|---|---|
| Silent partial success | A multi-record operation updates three of five rows; the tool returns success; the agent reports the task complete | Tools return per-item outcomes, not a boolean. The model can only report what the result tells it |
| Plausible hallucinated parameters | The model invents an order ID with the right shape, and the lookup returns someone else's order | Validate that IDs came from earlier tool output, not from the model's own text. Never let the model originate an identifier |
| Non-idempotent retries | A timeout on a payment or email call is retried; the original request had already succeeded | Idempotency keys on every mutating tool. Retry the key, not the action |
| Unbounded loops | The agent alternates between two tools indefinitely, or retries a failing call forever | Hard step ceiling, repeated-call detection, and a wall-clock timeout on the whole run |
| Context degradation | Deep into a long run, the agent forgets earlier constraints or repeats completed work | Context editing to clear stale tool results, or compaction to summarise history before the window fills |
| Stale reads | The agent reads a record at step 3 and writes based on it at step 15; something changed in between | Write tools verify current state, or use conditional writes that fail on mismatch |
| Cascading tool errors | One upstream failure produces a stream of downstream errors the model tries to reason through | Circuit-break: after N consecutive errors from the same tool, terminate the run and escalate |
| Prompt injection via tool output | Content the agent retrieves — an email body, a web page, a support ticket — contains instructions the model follows | Treat all tool output as untrusted data. Never let retrieved content authorise a privileged action; keep permissions in the harness |
That last one deserves emphasis, because it is the failure mode with the worst consequences and the least coverage in typical testing. Any agent that reads content produced by a third party — inbox triage, ticket routing, document processing, web research — is processing text that may have been written specifically to manipulate it. The defence is architectural, not prompt-based: an agent whose credentials cannot perform a destructive action cannot be talked into performing it, no matter what the retrieved text says.
Teams that ship reliable agents have an evaluation suite. Teams that don't, iterate by vibes — changing a prompt, running three examples, deciding it feels better, and discovering two weeks later that they broke a case they had fixed a month earlier. Evaluation is the difference between engineering and guessing, and for agents it has to work at two levels.
Outcome evaluation asks whether the final state is correct. Did the refund get issued for the right amount? Does the generated file contain the right data? This is what the business cares about, and it is the right primary metric — but on its own it is nearly useless for debugging, because a failed twenty-step run tells you only that something went wrong somewhere.
Trajectory evaluation asks whether the steps were right. Did the agent call the tools you would expect, in a defensible order, with valid arguments? Did it verify before writing? Did it skip a required check and get the right answer by luck? Trajectory evaluation is what makes a regression diagnosable, and it catches the dangerous case of a correct outcome reached through an unsafe path.
A workable evaluation setup does not need to be elaborate:
The most common objection is that building this takes time that could be spent on features. That is true for about three weeks, after which the team without an eval suite is spending most of its time re-fixing regressions and cannot safely change anything. The suite is not overhead; it is what makes the second month of development possible.
The most useful axis for deciding what an agent may do autonomously is not confidence, importance, or risk score — it is reversibility. Confidence scores from a language model are not well calibrated and should not gate destructive actions. Reversibility is a property of the action itself, knowable at design time, and it maps cleanly onto policy.
| Action class | Examples | Policy |
|---|---|---|
| Read-only | Lookups, searches, report generation | Fully autonomous; parallelise freely |
| Reversible writes | Draft creation, internal status changes, tagging | Autonomous, with an audit trail and an undo path |
| Externally visible | Sending email, posting to a customer channel, publishing | Human approval, or a hold-and-review queue |
| Irreversible or financial | Payments, refunds, deletions, contract actions | Explicit approval per action, with amount ceilings enforced in code |
Two implementation notes matter here. First, enforce limits in the harness, not the prompt. "Never refund more than £500" in a system prompt is a suggestion; a check in the refund tool that rejects amounts above £500 is a control. Anything you would be uncomfortable explaining to a regulator as "we asked the model not to" belongs in code.
Second, scope the credentials. An agent operating with database write access it never uses is carrying risk for no benefit. Give the agent's service account exactly the permissions its tools require and nothing more — this converts an entire category of prompt-injection and reasoning failures from incidents into permission errors, which the agent will then report cleanly.
Promoting an action from a general-purpose capability to a dedicated tool is also what makes gating possible at all. A single shell tool gives your harness an opaque command string; a dedicated send_email tool gives it typed arguments it can inspect, render for approval, log, and rate-limit. Breadth is convenient during development and expensive in production.
Agent economics surprise teams because the intuition from single-call LLM usage does not transfer. In a single call you pay for one prompt and one response. In an agent loop you re-send the entire accumulated conversation on every turn, so token consumption grows with roughly the square of the task length.
Concretely: a twenty-step task where each tool result adds about 2,000 tokens means turn twenty carries around 40,000 tokens of history. Summed across all twenty turns, that is roughly 400,000 input tokens for a single task — before counting the system prompt and tool schemas, which are re-sent every turn as well. A task you estimated at "a few thousand tokens" is two orders of magnitude off.
Four levers address this, in rough order of effort-to-payoff:
On latency, the arithmetic is similarly structural: an agent's response time is the sum of every model call and every tool execution in the chain. A ten-step task with three-second model calls and one-second tool calls takes forty seconds regardless of how fast any individual component is. If the interaction is user-facing, this shapes the product — stream progress, execute independent tool calls in parallel, and design for asynchronous completion rather than a spinner. Our guidance on the real cost of running AI in production covers the wider cost stack in detail.
When an agent produces a wrong outcome, the question is always "at which step did this go wrong, and what did it see?" An agent without tracing cannot answer that, and debugging degenerates into re-running the task and hoping it misbehaves the same way.
The minimum useful trace records, for every run: a run identifier, the full sequence of model requests and responses, every tool call with its exact arguments and its exact returned content, per-step latency, per-step token counts split by input, output, and cached, and the terminating condition. Store this even when the run succeeds — success traces are what you compare failures against, and they are the raw material for your golden evaluation set.
Two metrics beyond raw traces earn their place on a dashboard. Steps-to-completion, tracked as a distribution rather than an average, surfaces degradation early: a task type that used to finish in four steps and now averages seven is telling you something broke before the failure rate moves. And tool error rate by tool localises problems immediately — a spike in one tool is an integration issue, while a spike across all of them is usually a prompt or model change.
An honest guide has to include the cases where the answer is no. Agents are the wrong shape when the task is fully specifiable in advance — that is a workflow, and it will be cheaper, faster, and testable. They are the wrong shape when you need deterministic guarantees, because a system that must produce identical output for identical input should be code. They are the wrong shape when the cost of an undetected error exceeds the value of the automation, and no practical review step closes that gap.
They are also the wrong shape when the underlying process is undocumented. An agent automating a workflow that only exists in one person's head will encode that person's undocumented exceptions incorrectly, and nobody will be able to say whether its behaviour is right. The prerequisite work there is process documentation, not model integration.
Where agents do earn their keep is the class of problem with genuine branching, tolerable error costs, and a verification path: investigating and resolving support cases that require looking things up across systems, reconciling data where the discrepancies are not enumerable in advance, and multi-system operations that a person currently performs by reading one screen and typing into another. For a broader view of where these patterns are heading, see our overview of emerging patterns in AI software development.
For a well-scoped agent against systems that already have decent APIs, expect six to ten weeks to something genuinely production-ready — but the shape of that time is not what most people assume. Roughly two weeks gets you a working loop with real tools that handles the happy path. The remaining time goes to error handling, the evaluation suite, guardrails, observability, and the iteration cycle those enable. Teams that ship in three weeks have usually shipped the two-week version and are about to discover the rest of the schedule in production. Where the underlying APIs are poor or the process is undocumented, add time for that work before the agent work starts.
The loop itself is about fifty lines of code and worth writing once so you understand it. Beyond that, the useful question is what you actually need. Most provider SDKs now include a tool-runner helper that drives the loop while still exposing per-turn hooks for approval gates, logging, and result inspection — that covers the majority of custom-tool agents without a heavyweight dependency. Larger frameworks are worth it when you need what they bundle: built-in file and shell tools, sub-agent orchestration, session persistence. The failure mode to avoid is adopting a framework to skip understanding the loop, because every production problem in this guide lives inside it.
Three defences, in order of effectiveness. Constrain the schema so invalid values cannot be expressed — enums, required fields, format constraints, and strict schema enforcement where your provider supports it. Second, never let the model originate an identifier: validate that any ID passed to a tool appeared in earlier tool output, and reject it otherwise. Third, return specific, actionable errors when validation fails, so the model corrects rather than retries. Prompt instructions asking the model to be careful are the weakest of the four options and the one teams try first.
For a well-built agent on a defined task class, 60–80% full automation with the remainder escalated to a person is a reasonable target, and it is worth designing for that split explicitly rather than treating escalation as failure. The escalation path is a feature: it is what makes the automated portion safe to run unattended. Be sceptical of claimed rates above 90% unless the task is narrow — and be sceptical of any rate quoted without a corresponding error rate, since pushing automation higher almost always means accepting more wrong answers rather than fewer escalations.
Long-running agents need three things that short ones do not. Durable state, so a crash at step 40 does not restart at step 1 — persist the conversation and enough context to resume. Context management, because a multi-hour run will exceed any context window without clearing or compacting history. And asynchronous delivery, meaning the user submits work and is notified on completion rather than holding a connection open. For workloads at this scale it is also worth evaluating managed agent platforms, which handle session persistence, sandboxed execution, and scheduling as infrastructure rather than as code you maintain.
Yes, with the same controls you would apply to any system touching that data, plus two specific to agents. Every tool call must be logged with its arguments and results, because "which records did this system access and why" is a question you will be asked. And the agent's credentials must be scoped to the minimum its tools require — the audit conversation is very different when the agent architecturally could not have accessed the records in question. Our guides on HIPAA-compliant AI development and privacy law across borders cover the surrounding requirements.
If you are weighing an agent build and want a straight answer on whether it is the right shape for your problem, our AI development team works through exactly this assessment with clients — including the cases where the answer is that a workflow would serve you better.
Last updated: August 2026
Get expert guidance on implementing AI solutions that actually work. Our team will help you design, build, and deploy custom automation tailored to your business needs.
Discover how AI automation is transforming law offices—from document review and legal research to client intake, billing, and compliance—so legal teams can focus on what matters most.
Discover how AI automation development is transforming businesses—improving efficiency, reducing costs, and driving growth across manufacturing, healthcare, finance, and retail.
AI is reshaping occupational health by making return-to-work assessments faster, more consistent, and better for injured workers. Explore how automation is transforming disability management, functional assessments, and workplace rehabilitation programs.