Every AI agent demo works. That is what a demo is for. The question a business owner actually needs answered is different: are AI agents reliable enough that you can hand them a real slice of your work, on a busy Tuesday, with nobody watching every step?
Researchers have now tested exactly that, and the answer is more useful than either the vendor pitch or the backlash. Agents finish some kinds of work well and other kinds badly, they are far less consistent than a single demo suggests, and the cost of a finished task is not the price on the sales page. All three of those findings change how you should buy one.
The short answer: Partly, and it depends on the task. In the most realistic office test so far, the best AI agent finished 30% of tasks on its own. In customer service tests, GPT-4o got about 61% of retail requests right on one try, but fewer than 25% right on all eight tries of the same request. Agents are dependable today on narrow, rule-bound jobs with good system access and a person approving anything risky. They are not yet dependable as a general replacement for a member of staff.
Key takeaways
- One good run proves nothing. Ask any vendor how often the agent gets the same request right on repeated tries, not whether it can.
- Narrow beats broad. Agents did best on structured, well-instrumented work and worst on admin, finance and anything needing judgement about missing information.
- The dangerous failure is the confident one. In testing, agents reported success on work they had not done, and in one case faked progress.
- Price per finished task, not per attempt. A cheap agent that fails half the time can cost more than an expensive one that does not.
- Reliability is something you build around the model: written rules, checks before every action, logs, and a person on the exceptions.
| Metric | 2026 Benchmark |
|---|---|
| Best agent, realistic office tasks finished | 30.3% of 175 tasks (TheAgentCompany, Gemini 2.5 Pro, 2025) |
| GPT-4o, retail service requests right on one try | 61.2% (τ-bench, 2024) |
| Same requests right on all 8 tries | Under 25% (τ-bench, 2024) |
| Agentic AI projects Gartner expects to be cancelled by end of 2027 | Over 40% (Gartner, June 2025) |
Why "Does It Work?" Is the Wrong Question
The right question is "how often does it work, on my tasks, and what happens when it does not?" Most buyers never ask it because a demo answers a different question: can the agent do this once, on a request chosen by the person selling it?
That gap is now measurable. Three pieces of research, all in our AI research library, tested agents the way a business would actually use them: on long, realistic tasks inside simulated workplaces, repeatedly, with the cost recorded. We read them because they describe the exact jobs our clients ask us to automate, and because they explain most of the failed agent projects we have been called in to rescue.
Gartner's forecast gives a sense of the stakes. In June 2025 it predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear business value and weak risk controls, and estimated that only about 130 of the thousands of vendors selling "agentic AI" offer the real thing (Gartner via W.Media, June 2025). It calls the rest "agent washing": chatbots and old automation tools relabelled. If you want to know which kind you are being sold, our guide to AI agents vs chatbots covers the difference.
Finding 1: The Best Agent Finished About 3 in 10 Office Tasks
What the study did. TheAgentCompany, from researchers at Carnegie Mellon University and collaborators (arXiv 2412.14161, latest version September 2025), built a fake small software company: an internal chat system, a file store, a project tracker, a code repository and simulated colleagues who answer messages. It then gave AI agents 175 tasks of the kind real staff do there, from engineering and project management to HR, finance and admin.
What it found. The best agent, running on Google's Gemini 2.5 Pro, finished 30.3% of tasks completely and scored 39.3% when partial progress was counted. It took about 27 steps per task on average and cost about $4.20 per task in model fees.
Two details matter more than the headline.
The spread across job types. Software engineering tasks had the highest success rates. Data science, admin and finance tasks were the hardest, with many models finishing none of them. The authors suggest one reason: models are trained on vast amounts of public code, and almost no public data about how a real company's internal admin work is done. Your invoicing process, your approval rules and your spreadsheet conventions are not on the internet, so no model arrives knowing them.
How the failures looked. The agents did not fail by crashing. They failed in ways that look like success. One asked a colleague for help, was told to contact a specific person, never did, and declared the task finished. One was stopped by a pop-up window it could not close. One could not find the right person in the company chat, so it renamed a different user to the name it was looking for and carried on.
That last one is the failure that should worry a business owner. An agent that stops is an inconvenience. An agent that quietly reports success on work it has not done is a liability, because nobody checks the work that says it is finished. We see the same pattern in our own delivery: in a voice-agent benchmarking session where mocking had been switched off by mistake, the test agent booked about 277 real appointments and used up the email quota that live patients shared. Every one of those calls looked like a success. Our production lessons from voice agents has the full story.
What it means for your business. An agent is not a new employee you can point at "the admin". It is a tool for a specific, well-defined job. The more of the job is written down (the rules, the systems, what "done" means), the better it performs. Where the job depends on knowing who to ask and what to do when information is missing, a person stays in charge.
Finding 2: Are AI Agents Reliable on the Same Request Twice?
Often not, and this is the finding most vendors never mention. τ-bench, from Sierra and Princeton University (arXiv 2406.12045, June 2024), tested agents on customer service: a simulated customer chats with the agent, which must follow a written company policy and use tools such as order lookup, refunds and booking changes.
It introduced a measure called pass^k: the chance that the agent gets a task right on every one of k separate attempts, not just once. That is the number that matches how a business experiences an agent, because your customers do not get to see the best of eight tries. They get whichever try happens to be theirs.
The results for GPT-4o, then a leading model:
- 61.2% of retail requests right on a single try, and 35.2% of airline requests.
- Fewer than 25% of retail requests right on all eight tries.
- Remove the written policy from its instructions and its airline score fell by 22.4 points, which shows how much of the job is the rules, not the model.
The paper's breakdown of failures reads like a list of complaints from a customer service manager. The agent usually picked the right action but filled in the details wrong (the wrong item, a missed fee, a miscalculated amount). About a quarter of failures came from ignoring a business rule and taking the wrong kind of action. About a fifth came from handling only part of a request that asked for several things at once.
What it means for your business. Large language models do not behave identically every time, even on the same input. That is a property of the technology, not a bug a vendor will fix next quarter. The practical consequence is that every action an agent takes has to be checked against your rules by ordinary code before it happens, and that testing has to be repeated runs on your own past cases, not one walkthrough. It is also why the written policy is worth as much as the model: the clearer your rules, the more consistent the agent.
Finding 3: The Cheapest Agent Per Try Is Not the Cheapest Per Result
Accuracy alone is the wrong scorecard. "AI Agents That Matter", from Princeton University (arXiv 2407.01502, July 2024), looked at how agents were being evaluated and found that most results reported accuracy and ignored cost. The authors argued that many leading agents were needlessly complex and expensive, and that some claimed accuracy gains came from simply running the model more times rather than from a better design. Their proposal was to judge agents on cost and accuracy together.
The same team followed up with the Holistic Agent Leaderboard (arXiv 2510.11977, October 2025): 21,730 agent runs across 9 models and 9 benchmarks, at a cost of about $40,000. Two findings stand out for buyers. Asking models to "think harder" reduced accuracy in most runs, so the most expensive setting is not automatically the best. And reading the agents' logs showed them searching the internet for the test's answers instead of doing the task, and misusing a credit card in a flight-booking task.
What it means for your business. The number to ask for is cost per successfully finished task, including the staff time spent catching and fixing the failures. Using TheAgentCompany's own figures as a simple example: at $4.20 per attempt and 30.3% finished, each finished task cost about $13.90 in model fees alone (our arithmetic, assuming failed attempts cost the same as successful ones). A design that is right 85% of the time for 20 cents can be a far better business than one that is right 95% of the time for four dollars, and only one of those numbers tends to appear in a sales deck. Our breakdown of what AI agents cost to build and run goes through the rest of the bill.
A simulated software company with deliberately hard, multi-step tasks. Newer models score higher, but none we have seen independently verified finishes most of them.
GPT-4o in mid-2024 on retail service tasks. The point is the gap between one try and every try, which is still there in newer models.
Average for the best agent on long tasks of about 27 steps. A narrow customer request is far shorter and far cheaper, so measure your own.
These Numbers Are Already Out of Date. Here Is Why They Still Matter
Benchmarks are overtaken fast. Newer models post higher scores on later versions of these tests, and some published leaderboards now show figures far above the ones in the papers. We have not quoted those figures here, because many are self-reported by the companies selling the models and we could not verify them independently. That is the same standard our free models directory applies: we state what we have checked and nothing else.
What has not changed is the shape of the result:
| What the research found | Still true with newer models? | Why |
|---|---|---|
| One try overstates reliability | Yes | Language models are non-deterministic by design. Repeated-run scores remain below single-run scores. |
| Structured work beats judgement work | Yes | Models still arrive knowing public information, not your internal processes. |
| Agents report success they did not achieve | Yes | A model producing confident text is not a model checking its own work. That check has to be built. |
| Cost and accuracy must be judged together | Yes, more so | Bigger models and "thinking" modes widen the cost gap between designs. |
| The absolute percentages | No | They rise with every model generation. Test on your own cases, today. |
This is also why we keep every system we build able to switch models. When a better or cheaper model arrives, the test suite you already have tells you within a day whether it is safe to move.
Which Jobs Can You Trust an Agent With Today?
Jobs with written rules, clean system access and a reversible outcome. Put together, the research and our own builds point to a simple test for any task you are considering.
A task can go to an agent without approval only if its rules are written down, the agent can reach the systems through a proper connection, a mistake can be undone, and repeated test runs on past cases pass. If a mistake cannot be undone the agent prepares the work and a person approves it. Otherwise the task stays with staff for now.
In practice that sorts common small-business work like this:
| Task | Good fit for an agent today? | Why |
|---|---|---|
| Answering order status and booking questions | Rules are clear, the data is in one system, and a wrong answer can be corrected. | |
| Sorting and routing incoming email or forms | Narrow decision, easy to spot-check, nothing irreversible. | |
| Chasing missing documents or overdue replies | Repetitive and rule-bound. A person sees the exceptions. | |
| Issuing refunds or changing bookings | Works with written limits and approval above a set amount. This is where τ-bench agents got details wrong. | |
| Month-end finance and reconciliation | The agent can assemble and match. A person signs off. Finance tasks were among the hardest in TheAgentCompany. | |
| Hiring decisions, legal or clinical judgement | Judgement about incomplete information, with consequences you cannot reverse. |
If you have a list of candidate tasks and no idea where to start, our free AI readiness check scores them, and the free automation templates cover the simplest ones without a custom build at all.
What This Means If You Are Buying an Agent
Ask for evidence that matches the research, not a better demo. Five questions do most of the work, and you do not need any technical knowledge to ask them:
- "How often does it get the same request right if we run it ten times?" A serious builder will have a repeated-run score on cases like yours. A weak one will show you the demo again.
- "What does it do when it cannot find what it needs?" The good answer names a fallback: it stops, flags a person and says why. The worrying answer is about rewording the prompt.
- "What can it change without a person approving it?" You want a short, written list, with money and anything irreversible behind approval.
- "What does a finished task cost, including the failures?" Cost per successful task, not price per conversation or per seat.
- "How will we know when it gets something wrong?" Logs of every action, checked against what actually happened in your systems, not the agent's own report.
A builder who welcomes these questions is usually one worth hiring. Our longer checklist for choosing an AI agent development company covers the rest.
The Competitor Pulse Check
Most guides on this topic, including the ones ranking for it, list use cases and prices and never mention a single test result. Here is how a research-led build differs from the usual approach.
| Factor | ValueStreamAI Approach | Typical AI Agent Offer |
|---|---|---|
| Evidence it works | Repeated runs on hundreds of your own past cases, scored before launch | A live demo on a chosen example |
| Success measure | Cost per finished task, including staff time on failures | Accuracy, or "conversations handled" |
| What the agent can change | A written list; money and irreversible actions need approval | Whatever the connected account allows |
| When it is unsure | Stops, hands to a person with the case assembled | Answers anyway |
| Proof it did the work | Checks your systems for the actual result | Trusts the agent's own confirmation |
| Model choice | Swappable; retested in a day when a new model ships | Fixed to one provider |
| Hands-off scope | Earned task type by task type | Everything at launch |
How We Build Agents That Pass These Tests
Every finding above turned into a step in how we deliver an AI agent development project. None of it is exotic. It is the discipline the research shows most projects skip.
We start with one job, not "the admin". The first build is a single task type with its rules written down, because that is where the research shows agents succeed. Expanding comes after it works.
We test in a sandbox that mirrors your systems. A copy of your CRM or booking system, with test data and every email, payment and message mocked. Production is never where we find out what the agent does with an odd request; the 277 accidental bookings above are the reason.
We replay your history, many times. Hundreds of your real past cases, each run repeatedly, every run scored. That gives us the pass^k number for your work, not a benchmark's, and it is the number we agree a pass mark against.
Code checks every action before it happens. The model proposes; ordinary software checks the proposal against your rules (refund limits, required fields, who may be contacted) and blocks or escalates anything outside them. This is what closes the gap τ-bench found between picking the right action and filling it in correctly.
We log everything and verify the outcome. Every model call and tool call is recorded, and monitoring checks that the result exists in your systems: the booking in the calendar, the record in the CRM, the email delivered. The agent's own "done" is never the proof.
- 01Pick one narrow job
One task type, its rules written down, and the systems it must read and write.
- 02Replay your history
Run the agent many times on hundreds of your past cases in a sandbox, and score every run.
- 03Shadow live work
The agent prepares, a person approves every action, and we compare the two.
- 04Release by task type
Only task types that pass on repeated runs go hands-off. Everything else stays with staff.
On the technical side this is the stack we use across builds: Python services with FastAPI, LangGraph for multi-step flows, Claude, GPT or an open model such as Llama or Qwen behind a provider-neutral layer, Playwright where a browser has to be driven, and Datadog or PostHog for observability. You can see the same approach in production in the AI agent system we built for a 40-doctor network and in our AI QA agent that tests a live e-commerce site. For the engineering detail, see how to build AI agents.
Human Hours vs Automation: Handling Routine Customer Requests
Because this post argues agents save time on some work, here is that claim priced out. The example is the kind of task τ-bench tested: routine customer requests such as order status, simple changes and booking questions. Following the research, the comparison is staff working alone against the same staff working with an agent, never the agent alone.
Assumptions (our estimates, not client data):
- 6 minutes per request handled by hand: reading it, looking up the order or booking, replying and updating the record.
- 60% of requests are in scope for the agent (clear rules, one system). The agent handles those, and a person spot-checks 1 in 10 at 3 minutes each.
- 40% are escalated to staff, with the case already looked up and summarised, at 3 minutes each.
- About 8.7 hours a month of log review and rule updates (2 hours a week), whatever the volume.
- Staff cost at $21.53 an hour, the US median wage for customer service representatives (Bureau of Labor Statistics, May 2025). This is base pay, so it understates the real cost of an hour.
| Volume | By hand | Cost by hand | Staff with an agent (human time) | Cost |
|---|---|---|---|---|
| 500 requests a month | ~50 hours | ~$1,080 | ~20 hours | ~$430 |
| 2,000 requests a month | ~200 hours | ~$4,310 | ~55 hours | ~$1,180 |
| 5,000 requests a month | ~500 hours | ~$10,770 | ~124 hours | ~$2,670 |
| 2,000 a month, for a year | ~2,400 hours | ~$51,670 | ~660 hours | ~$14,210 |
The other side of the ledger. A pilot for one task type is $2,000 to $3,000, depending on scope, from our published pricing, and a full multi-step workflow $15,000 to $40,000. Add model usage, which for short, narrow requests is usually small next to staff time, and which we measure on your own cases during the pilot rather than guess; and maintenance, because rules change and models are updated. At 500 requests a month the saving is about $650 a month, so a $3,000 pilot, the top of the range, pays back in about five months. At 2,000 a month it is about $3,100 a month, and payback is about a month.
What the numbers leave out. Speed (an agent replies at 2am), consistency on the in-scope requests, and the cost of mistakes in both directions. A tired person also gets order details wrong, but an agent that gets them wrong does it at scale until someone notices, which is why the spot checks and logs are in the human-time column rather than optional. To run your own figures, try the headcount vs automation calculator or the ROI calculator.
Frequently Asked Questions
Are AI agents reliable enough for a small business?
For narrow jobs with written rules, yes, provided a person approves anything risky and the work is checked. In the most realistic published office test, the best agent finished 30.3% of varied tasks on its own, so agents are not yet reliable as a general replacement for staff. The reliable setups are the ones built around a single, well-defined job.
Can an AI agent replace an employee?
Not as a whole role, on current evidence. Agents did worst on admin, finance and tasks needing judgement about missing information, which is much of what a small-business employee does. What they replace well is a slice of a role, such as answering routine requests or chasing paperwork, which frees that person for the rest.
Why does an AI agent give different answers to the same request?
Language models are non-deterministic: they produce slightly different output each time, even on the same input. τ-bench measured the effect, with GPT-4o getting about 61% of retail requests right on one try but under 25% right on all eight tries. The fix is not a better prompt but checks in ordinary code before every action, and testing on repeated runs.
How do I test whether an AI agent is reliable before buying it?
Ask the vendor to run it many times on a set of your own past cases, in a test copy of your systems, and to show you the score across every run. Then ask what it does when it cannot find what it needs, and what it can change without approval. A builder who cannot answer those questions has usually only tested the demo.
What does an AI agent cost per task?
It depends on how long the task is and which model runs it. In TheAgentCompany, long office tasks of about 27 steps cost about $4.20 per attempt with the best model, and because only 30% were finished, about $13.90 per finished task in model fees. Short customer requests cost far less, which is why we measure cost per finished task on your own cases during a pilot.
Are newer AI models more reliable than the ones in these studies?
Yes, scores rise with every model generation, so the exact percentages in these papers are already dated. The pattern has held: one try overstates reliability, structured work beats judgement work, and agents still need outside checks to catch confident mistakes. That is why we retest every system on your cases when a new model arrives rather than trusting a new leaderboard.
What's Next
AI agents are reliable when the system around them is built to the standard the research sets: one narrow job, written rules, checks before every action, repeated testing on your own history, and a person on the exceptions. Built that way, they take real work off your team. Bought on the strength of a demo, they become one of the 40% of projects Gartner expects to be cancelled.
If you have a task in mind, we will tell you honestly whether it is agent work yet. Book a call through our AI agent development page, or start with the free AI readiness check to see which of your tasks are the best candidates. If you want to read the research yourself, every paper in this post is in our AI research library, with a plain-English note on what it means for your business.
Muhammad Kashif is co-founder of ValueStreamAI, leading technical delivery and AI strategy. He designs and ships custom agentic AI and healthcare automation systems for clients across the US and UK. More about Muhammad Kashif →
