Why AI agents work in demos and fail in production

AI Agents · By Caleb Sakala · April 3, 2026

The demo always works.

You show the AI agent handling a support ticket. It reads the email, classifies the issue, drafts a response, checks the CRM, and closes the loop. Your stakeholders are impressed. You ship to production.

Three weeks later, you have a monitoring dashboard full of failures, a PagerDuty alert at 11pm, and a support ticket from a confused customer who got the wrong response for the fourth time that week.

This is the pattern. Not the exception.

A December 2025 UC Berkeley study surveyed over 300 teams with operational AI agents and found that reliability remains the top development challenge. More telling: 68% of production agents execute fewer than 10 steps before requiring human intervention. Not because the teams are being cautious. Because anything longer breaks.

The standard explanation is that agents fail due to hallucinations, compounding error rates, or poorly defined tool schemas. Those are real problems. But fixing them won't save most production deployments, because the failure runs deeper.

The math makes it obvious

This is arithmetic, not opinion.

If your AI agent is 85% accurate on each individual action (a reasonable expectation for a capable LLM with good prompting), a 10-step workflow succeeds end-to-end about 20% of the time. Temporal's engineering team worked this through explicitly: (0.85)^10 ≈ 0.197. Four out of five runs fail on some step.

To get a 10-step workflow to 95% reliability with that per-step accuracy, you'd need each step to be roughly 99.5% reliable. No current model achieves that in an open-ended agentic loop on real-world tasks.

So production engineers do what the data shows: they make their agents smaller. Fewer steps. More human checkpoints. Narrower scope. The same UC Berkeley study found that 92.5% of agents in actual production deliver output to a human, not to downstream software. The agent proposes; a person decides.

That's not agentic automation. That's a fancy draft generator.

What the data is telling you

The reason production teams constrain their agents so heavily isn't a failure of execution. It's signal.

These systems work when they don't need to act autonomously. They work as classifiers, as drafters, as suggestion engines. They break when given the authority to take consequential, multi-step actions without human review.

Here's the contrarian read: that's not a bug in current models. That's the correct behavior for a decision-making system operating under genuine uncertainty. An agent that's unsure what to do next is right to pause. The problem is that most production automation use cases don't require decision-making under uncertainty. They require reliable execution of known logic.

Routing invoices above $10,000 to senior approval doesn't require deliberation. It requires a rule. Sending a Slack alert when a GitHub action fails doesn't require reasoning. It requires a trigger and a message. Syncing a new CRM contact to an outreach sequence doesn't require judgment. It requires a lookup and an insert.

The workflows that hold up production are mostly deterministic. Engineers are building agents to execute them anyway, then discovering that agents are bad at behaving deterministically.

What holds up in production

When teams at McKinsey-tracked companies were asked how many had internal AI tools beyond the pilot phase, less than 10% said yes. The ones that did succeed weren't running general-purpose agents. They were running constrained, purpose-built workflows with AI embedded at specific decision points.

The architectural pattern: the workflow is deterministic. Steps execute in order. Each step has a defined input schema and a defined output contract. AI gets invoked only where it matters: classifying an ambiguous input, generating a draft, or scoring a confidence level. Everything else is explicit logic.

Consider an invoice processing workflow: an LLM step extracts line items from a vendor PDF. A validation step checks amounts against stored contract terms. A routing step applies a threshold rule: anything over $10,000 goes to senior approval, everything else auto-processes. A payment step fires only after confirmation. Each step knows its exact role. AI handles the ambiguous part (parsing unstructured documents); deterministic logic handles everything downstream.

This is the pattern Chase Agents was built to run. Each automation step declares its inputs, its outputs, and which tools it can invoke. The AI helps construct and refine the workflow, but nothing in the running automation asks a model to "figure out what to do next." The execution is predictable. The AI did the hard work earlier, in the construction phase.

The one case where agents are the right call

This isn't a blanket argument against AI agents. There are genuine use cases where agentic behavior is what you need.

Open-ended research tasks don't have a predetermined execution path. "Find everything relevant to this acquisition target" requires the agent to decide what to look up next based on what it found. Security threat hunting works the same way. Exploratory data analysis where the analysis path depends on what emerges. Code debugging when the failure mode is unknown.

What these have in common: the output is information or a plan, not a business-critical action. The agent's output goes to a human who decides what to do next. Which is exactly what the UC Berkeley data shows is happening in 92.5% of production deployments already.

The mistake is using the same architecture for tasks that need reliable execution. A payment disbursement workflow should not run as an agent deciding which accounts to credit. An automated recruitment screener should not use an agent to decide which candidates to advance. Those require deterministic logic with clear authorization boundaries: logic that can be audited, tested, and predicted.

When those workflows run in Chase Agents, a step that processes purchase recommendations and a step that initiates payments are structurally separate, with tool access explicitly granted or withheld at each step. The platform routes by action type rather than leaving tool selection to a model at runtime. This makes the authorization boundary an architectural constraint, not a convention that someone remembers to follow.

What to do today

Pull up your most fragile production agent. Count the steps in a typical run. For each step, ask: is the correct action here genuinely ambiguous, or would a deterministic rule produce the right answer 98% of the time?

Most teams find that 70-80% of their agent's steps are executing logic that didn't need to be decided at runtime. Those steps shouldn't be agent actions. They should be workflow nodes with defined inputs and outputs, with AI invoked only where ambiguity is real.

That's not a workaround for bad agents. It's a more accurate model of what the problem actually requires.