homeservicesworkaboutblogfree templatescontactFree Tools →Free AI ModelsResearch LibraryROI CalculatorSavings CalculatorAI Readiness ScoreHire vs. AutomateAutomation Quote
book a 30-min call →
home / blog / Are AI Agents Reliable Enough to Run Part of Your Business? What the Research Says

Are AI Agents Reliable Enough to Run Part of Your Business? What the Research Says

Are AI agents reliable enough to hand real work to? Three research studies tested agents on office tasks and customer service. Here is what they found, what it means for a small business, and how we build agents that pass the same tests.

Are AI Agents Reliable Enough to Run Part of Your Business? What the Research Says

Every AI agent demo works. That is what a demo is for. The question a business owner actually needs answered is different: are AI agents reliable enough that you can hand them a real slice of your work, on a busy Tuesday, with nobody watching every step?

Researchers have now tested exactly that, and the answer is more useful than either the vendor pitch or the backlash. Agents finish some kinds of work well and other kinds badly, they are far less consistent than a single demo suggests, and the cost of a finished task is not the price on the sales page. All three of those findings change how you should buy one.

The short answer: Partly, and it depends on the task. In the most realistic office test so far, the best AI agent finished 30% of tasks on its own. In customer service tests, GPT-4o got about 61% of retail requests right on one try, but fewer than 25% right on all eight tries of the same request. Agents are dependable today on narrow, rule-bound jobs with good system access and a person approving anything risky. They are not yet dependable as a general replacement for a member of staff.

Key takeaways

  • One good run proves nothing. Ask any vendor how often the agent gets the same request right on repeated tries, not whether it can.
  • Narrow beats broad. Agents did best on structured, well-instrumented work and worst on admin, finance and anything needing judgement about missing information.
  • The dangerous failure is the confident one. In testing, agents reported success on work they had not done, and in one case faked progress.
  • Price per finished task, not per attempt. A cheap agent that fails half the time can cost more than an expensive one that does not.
  • Reliability is something you build around the model: written rules, checks before every action, logs, and a person on the exceptions.
Metric 2026 Benchmark
Best agent, realistic office tasks finished 30.3% of 175 tasks (TheAgentCompany, Gemini 2.5 Pro, 2025)
GPT-4o, retail service requests right on one try 61.2% (τ-bench, 2024)
Same requests right on all 8 tries Under 25% (τ-bench, 2024)
Agentic AI projects Gartner expects to be cancelled by end of 2027 Over 40% (Gartner, June 2025)

Why "Does It Work?" Is the Wrong Question

The right question is "how often does it work, on my tasks, and what happens when it does not?" Most buyers never ask it because a demo answers a different question: can the agent do this once, on a request chosen by the person selling it?

That gap is now measurable. Three pieces of research, all in our AI research library, tested agents the way a business would actually use them: on long, realistic tasks inside simulated workplaces, repeatedly, with the cost recorded. We read them because they describe the exact jobs our clients ask us to automate, and because they explain most of the failed agent projects we have been called in to rescue.

Gartner's forecast gives a sense of the stakes. In June 2025 it predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear business value and weak risk controls, and estimated that only about 130 of the thousands of vendors selling "agentic AI" offer the real thing (Gartner via W.Media, June 2025). It calls the rest "agent washing": chatbots and old automation tools relabelled. If you want to know which kind you are being sold, our guide to AI agents vs chatbots covers the difference.

what the studies measuredThree findings every buyer should know
0.0%of office tasks finished by the best agent (Gemini 2.5 Pro)
0tasks across engineering, HR, finance, admin and project work
<0%of retail service tasks GPT-4o got right on all 8 tries
TheAgentCompany (arXiv 2412.14161, v3, September 2025) and τ-bench (arXiv 2406.12045, June 2024). Each figure is the paper's own, for the models it tested.

Finding 1: The Best Agent Finished About 3 in 10 Office Tasks

What the study did. TheAgentCompany, from researchers at Carnegie Mellon University and collaborators (arXiv 2412.14161, latest version September 2025), built a fake small software company: an internal chat system, a file store, a project tracker, a code repository and simulated colleagues who answer messages. It then gave AI agents 175 tasks of the kind real staff do there, from engineering and project management to HR, finance and admin.

What it found. The best agent, running on Google's Gemini 2.5 Pro, finished 30.3% of tasks completely and scored 39.3% when partial progress was counted. It took about 27 steps per task on average and cost about $4.20 per task in model fees.

out of every ten office tasksHow many tasks the best agent finished on its own
Finished in full3/10
Not finished7/10
TheAgentCompany, best result: Gemini 2.5 Pro finished 30.3% of 175 tasks in full. Rounded to the nearest whole task out of ten.

Two details matter more than the headline.

The spread across job types. Software engineering tasks had the highest success rates. Data science, admin and finance tasks were the hardest, with many models finishing none of them. The authors suggest one reason: models are trained on vast amounts of public code, and almost no public data about how a real company's internal admin work is done. Your invoicing process, your approval rules and your spreadsheet conventions are not on the internet, so no model arrives knowing them.

How the failures looked. The agents did not fail by crashing. They failed in ways that look like success. One asked a colleague for help, was told to contact a specific person, never did, and declared the task finished. One was stopped by a pop-up window it could not close. One could not find the right person in the company chat, so it renamed a different user to the name it was looking for and carried on.

That last one is the failure that should worry a business owner. An agent that stops is an inconvenience. An agent that quietly reports success on work it has not done is a liability, because nobody checks the work that says it is finished. We see the same pattern in our own delivery: in a voice-agent benchmarking session where mocking had been switched off by mistake, the test agent booked about 277 real appointments and used up the email quota that live patients shared. Every one of those calls looked like a success. Our production lessons from voice agents has the full story.

What it means for your business. An agent is not a new employee you can point at "the admin". It is a tool for a specific, well-defined job. The more of the job is written down (the rules, the systems, what "done" means), the better it performs. Where the job depends on knowing who to ask and what to do when information is missing, a person stays in charge.


Finding 2: Are AI Agents Reliable on the Same Request Twice?

Often not, and this is the finding most vendors never mention. τ-bench, from Sierra and Princeton University (arXiv 2406.12045, June 2024), tested agents on customer service: a simulated customer chats with the agent, which must follow a written company policy and use tools such as order lookup, refunds and booking changes.

It introduced a measure called pass^k: the chance that the agent gets a task right on every one of k separate attempts, not just once. That is the number that matches how a business experiences an agent, because your customers do not get to see the best of eight tries. They get whichever try happens to be theirs.

single try vs every tryGPT-4o on customer-service tasks: one attempt vs eight
Retail, right on one try
0%
Airline, right on one try
0%
Retail, right on all 8 tries
0%
τ-bench (Sierra and Princeton, June 2024), Table 2 and abstract. The 8-try retail figure is reported as under 25%, so it is drawn at 25 as a ceiling.

The results for GPT-4o, then a leading model:

  • 61.2% of retail requests right on a single try, and 35.2% of airline requests.
  • Fewer than 25% of retail requests right on all eight tries.
  • Remove the written policy from its instructions and its airline score fell by 22.4 points, which shows how much of the job is the rules, not the model.

The paper's breakdown of failures reads like a list of complaints from a customer service manager. The agent usually picked the right action but filled in the details wrong (the wrong item, a missed fee, a miscalculated amount). About a quarter of failures came from ignoring a business rule and taking the wrong kind of action. About a fifth came from handling only part of a request that asked for several things at once.

What it means for your business. Large language models do not behave identically every time, even on the same input. That is a property of the technology, not a bug a vendor will fix next quarter. The practical consequence is that every action an agent takes has to be checked against your rules by ordinary code before it happens, and that testing has to be repeated runs on your own past cases, not one walkthrough. It is also why the written policy is worth as much as the model: the clearer your rules, the more consistent the agent.


Finding 3: The Cheapest Agent Per Try Is Not the Cheapest Per Result

Accuracy alone is the wrong scorecard. "AI Agents That Matter", from Princeton University (arXiv 2407.01502, July 2024), looked at how agents were being evaluated and found that most results reported accuracy and ignored cost. The authors argued that many leading agents were needlessly complex and expensive, and that some claimed accuracy gains came from simply running the model more times rather than from a better design. Their proposal was to judge agents on cost and accuracy together.

The same team followed up with the Holistic Agent Leaderboard (arXiv 2510.11977, October 2025): 21,730 agent runs across 9 models and 9 benchmarks, at a cost of about $40,000. Two findings stand out for buyers. Asking models to "think harder" reduced accuracy in most runs, so the most expensive setting is not automatically the best. And reading the agents' logs showed them searching the internet for the test's answers instead of doing the task, and misusing a credit card in a flight-booking task.

What it means for your business. The number to ask for is cost per successfully finished task, including the staff time spent catching and fixing the failures. Using TheAgentCompany's own figures as a simple example: at $4.20 per attempt and 30.3% finished, each finished task cost about $13.90 in model fees alone (our arithmetic, assuming failed attempts cost the same as successful ones). A design that is right 85% of the time for 20 cents can be a far better business than one that is right 95% of the time for four dollars, and only one of those numbers tends to appear in a sales deck. Our breakdown of what AI agents cost to build and run goes through the rest of the bill.

read the number, then the caveatWhat each headline figure does and does not mean
30.3%office tasks finished

A simulated software company with deliberately hard, multi-step tasks. Newer models score higher, but none we have seen independently verified finishes most of them.

<25%right on all 8 tries

GPT-4o in mid-2024 on retail service tasks. The point is the gap between one try and every try, which is still there in newer models.

$4.20cost per office task

Average for the best agent on long tasks of about 27 steps. A narrow customer request is far shorter and far cheaper, so measure your own.

hover or tap a card
The papers cited in this section. Caveats are the authors' own limitations plus our reading of them.

These Numbers Are Already Out of Date. Here Is Why They Still Matter

Benchmarks are overtaken fast. Newer models post higher scores on later versions of these tests, and some published leaderboards now show figures far above the ones in the papers. We have not quoted those figures here, because many are self-reported by the companies selling the models and we could not verify them independently. That is the same standard our free models directory applies: we state what we have checked and nothing else.

What has not changed is the shape of the result:

What the research found Still true with newer models? Why
One try overstates reliability Yes Language models are non-deterministic by design. Repeated-run scores remain below single-run scores.
Structured work beats judgement work Yes Models still arrive knowing public information, not your internal processes.
Agents report success they did not achieve Yes A model producing confident text is not a model checking its own work. That check has to be built.
Cost and accuracy must be judged together Yes, more so Bigger models and "thinking" modes widen the cost gap between designs.
The absolute percentages No They rise with every model generation. Test on your own cases, today.

This is also why we keep every system we build able to switch models. When a better or cheaper model arrives, the test suite you already have tells you within a day whether it is safe to move.


Which Jobs Can You Trust an Agent With Today?

Jobs with written rules, clean system access and a reversible outcome. Put together, the research and our own builds point to a simple test for any task you are considering.

the hand-off testShould an AI agent do this task on its own?

A task can go to an agent without approval only if its rules are written down, the agent can reach the systems through a proper connection, a mistake can be undone, and repeated test runs on past cases pass. If a mistake cannot be undone the agent prepares the work and a person approves it. Otherwise the task stays with staff for now.

The criteria set out in this section.

In practice that sorts common small-business work like this:

Task Good fit for an agent today? Why
Answering order status and booking questions Rules are clear, the data is in one system, and a wrong answer can be corrected.
Sorting and routing incoming email or forms Narrow decision, easy to spot-check, nothing irreversible.
Chasing missing documents or overdue replies Repetitive and rule-bound. A person sees the exceptions.
Issuing refunds or changing bookings Works with written limits and approval above a set amount. This is where τ-bench agents got details wrong.
Month-end finance and reconciliation The agent can assemble and match. A person signs off. Finance tasks were among the hardest in TheAgentCompany.
Hiring decisions, legal or clinical judgement Judgement about incomplete information, with consequences you cannot reverse.

If you have a list of candidate tasks and no idea where to start, our free AI readiness check scores them, and the free automation templates cover the simplest ones without a custom build at all.


What This Means If You Are Buying an Agent

Ask for evidence that matches the research, not a better demo. Five questions do most of the work, and you do not need any technical knowledge to ask them:

  1. "How often does it get the same request right if we run it ten times?" A serious builder will have a repeated-run score on cases like yours. A weak one will show you the demo again.
  2. "What does it do when it cannot find what it needs?" The good answer names a fallback: it stops, flags a person and says why. The worrying answer is about rewording the prompt.
  3. "What can it change without a person approving it?" You want a short, written list, with money and anything irreversible behind approval.
  4. "What does a finished task cost, including the failures?" Cost per successful task, not price per conversation or per seat.
  5. "How will we know when it gets something wrong?" Logs of every action, checked against what actually happened in your systems, not the agent's own report.

A builder who welcomes these questions is usually one worth hiring. Our longer checklist for choosing an AI agent development company covers the rest.


The Competitor Pulse Check

Most guides on this topic, including the ones ranking for it, list use cases and prices and never mention a single test result. Here is how a research-led build differs from the usual approach.

Factor ValueStreamAI Approach Typical AI Agent Offer
Evidence it works Repeated runs on hundreds of your own past cases, scored before launch A live demo on a chosen example
Success measure Cost per finished task, including staff time on failures Accuracy, or "conversations handled"
What the agent can change A written list; money and irreversible actions need approval Whatever the connected account allows
When it is unsure Stops, hands to a person with the case assembled Answers anyway
Proof it did the work Checks your systems for the actual result Trusts the agent's own confirmation
Model choice Swappable; retested in a day when a new model ships Fixed to one provider
Hands-off scope Earned task type by task type Everything at launch

How We Build Agents That Pass These Tests

Every finding above turned into a step in how we deliver an AI agent development project. None of it is exotic. It is the discipline the research shows most projects skip.

We start with one job, not "the admin". The first build is a single task type with its rules written down, because that is where the research shows agents succeed. Expanding comes after it works.

We test in a sandbox that mirrors your systems. A copy of your CRM or booking system, with test data and every email, payment and message mocked. Production is never where we find out what the agent does with an odd request; the 277 accidental bookings above are the reason.

We replay your history, many times. Hundreds of your real past cases, each run repeatedly, every run scored. That gives us the pass^k number for your work, not a benchmark's, and it is the number we agree a pass mark against.

Code checks every action before it happens. The model proposes; ordinary software checks the proposal against your rules (refund limits, required fields, who may be contacted) and blocks or escalates anything outside them. This is what closes the gap τ-bench found between picking the right action and filling it in correctly.

We log everything and verify the outcome. Every model call and tool call is recorded, and monitoring checks that the result exists in your systems: the booking in the calendar, the record in the CRM, the email delivered. The agent's own "done" is never the proof.

how we get an agent to productionFour stages, each with a pass mark
  1. 01
    Pick one narrow jobweek 1

    One task type, its rules written down, and the systems it must read and write.

  2. 02
    Replay your historyweeks 2 to 4

    Run the agent many times on hundreds of your past cases in a sandbox, and score every run.

  3. 03
    Shadow live workweeks 5 to 8

    The agent prepares, a person approves every action, and we compare the two.

  4. 04
    Release by task typeafter the pass mark

    Only task types that pass on repeated runs go hands-off. Everything else stays with staff.

Our delivery process for agent builds, as described in this section.

On the technical side this is the stack we use across builds: Python services with FastAPI, LangGraph for multi-step flows, Claude, GPT or an open model such as Llama or Qwen behind a provider-neutral layer, Playwright where a browser has to be driven, and Datadog or PostHog for observability. You can see the same approach in production in the AI agent system we built for a 40-doctor network and in our AI QA agent that tests a live e-commerce site. For the engineering detail, see how to build AI agents.


Human Hours vs Automation: Handling Routine Customer Requests

Because this post argues agents save time on some work, here is that claim priced out. The example is the kind of task τ-bench tested: routine customer requests such as order status, simple changes and booking questions. Following the research, the comparison is staff working alone against the same staff working with an agent, never the agent alone.

Assumptions (our estimates, not client data):

  • 6 minutes per request handled by hand: reading it, looking up the order or booking, replying and updating the record.
  • 60% of requests are in scope for the agent (clear rules, one system). The agent handles those, and a person spot-checks 1 in 10 at 3 minutes each.
  • 40% are escalated to staff, with the case already looked up and summarised, at 3 minutes each.
  • About 8.7 hours a month of log review and rule updates (2 hours a week), whatever the volume.
  • Staff cost at $21.53 an hour, the US median wage for customer service representatives (Bureau of Labor Statistics, May 2025). This is base pay, so it understates the real cost of an hour.
Volume By hand Cost by hand Staff with an agent (human time) Cost
500 requests a month ~50 hours ~$1,080 ~20 hours ~$430
2,000 requests a month ~200 hours ~$4,310 ~55 hours ~$1,180
5,000 requests a month ~500 hours ~$10,770 ~124 hours ~$2,670
2,000 a month, for a year ~2,400 hours ~$51,670 ~660 hours ~$14,210
staff hours a monthCustomer requests: staff alone vs staff working with an agent
500 a month
50 hrs
20 hrs
2,000 a month
200 hrs
55 hrs
5,000 a month
500 hrs
124 hrs
Our estimate, not client data. Assumptions in the table above: 6 minutes per request by hand, 60% in scope for the agent, BLS May 2025 median wage for customer service representatives.

The other side of the ledger. A pilot for one task type is $2,000 to $3,000, depending on scope, from our published pricing, and a full multi-step workflow $15,000 to $40,000. Add model usage, which for short, narrow requests is usually small next to staff time, and which we measure on your own cases during the pilot rather than guess; and maintenance, because rules change and models are updated. At 500 requests a month the saving is about $650 a month, so a $3,000 pilot, the top of the range, pays back in about five months. At 2,000 a month it is about $3,100 a month, and payback is about a month.

What the numbers leave out. Speed (an agent replies at 2am), consistency on the in-scope requests, and the cost of mistakes in both directions. A tired person also gets order details wrong, but an agent that gets them wrong does it at scale until someone notices, which is why the spot checks and logs are in the human-time column rather than optional. To run your own figures, try the headcount vs automation calculator or the ROI calculator.


Frequently Asked Questions

Are AI agents reliable enough for a small business?

For narrow jobs with written rules, yes, provided a person approves anything risky and the work is checked. In the most realistic published office test, the best agent finished 30.3% of varied tasks on its own, so agents are not yet reliable as a general replacement for staff. The reliable setups are the ones built around a single, well-defined job.

Can an AI agent replace an employee?

Not as a whole role, on current evidence. Agents did worst on admin, finance and tasks needing judgement about missing information, which is much of what a small-business employee does. What they replace well is a slice of a role, such as answering routine requests or chasing paperwork, which frees that person for the rest.

Why does an AI agent give different answers to the same request?

Language models are non-deterministic: they produce slightly different output each time, even on the same input. τ-bench measured the effect, with GPT-4o getting about 61% of retail requests right on one try but under 25% right on all eight tries. The fix is not a better prompt but checks in ordinary code before every action, and testing on repeated runs.

How do I test whether an AI agent is reliable before buying it?

Ask the vendor to run it many times on a set of your own past cases, in a test copy of your systems, and to show you the score across every run. Then ask what it does when it cannot find what it needs, and what it can change without approval. A builder who cannot answer those questions has usually only tested the demo.

What does an AI agent cost per task?

It depends on how long the task is and which model runs it. In TheAgentCompany, long office tasks of about 27 steps cost about $4.20 per attempt with the best model, and because only 30% were finished, about $13.90 per finished task in model fees. Short customer requests cost far less, which is why we measure cost per finished task on your own cases during a pilot.

Are newer AI models more reliable than the ones in these studies?

Yes, scores rise with every model generation, so the exact percentages in these papers are already dated. The pattern has held: one try overstates reliability, structured work beats judgement work, and agents still need outside checks to catch confident mistakes. That is why we retest every system on your cases when a new model arrives rather than trusting a new leaderboard.


What's Next

AI agents are reliable when the system around them is built to the standard the research sets: one narrow job, written rules, checks before every action, repeated testing on your own history, and a person on the exceptions. Built that way, they take real work off your team. Bought on the strength of a demo, they become one of the 40% of projects Gartner expects to be cancelled.

If you have a task in mind, we will tell you honestly whether it is agent work yet. Book a call through our AI agent development page, or start with the free AI readiness check to see which of your tasks are the best candidates. If you want to read the research yourself, every paper in this post is in our AI research library, with a plain-English note on what it means for your business.

Disclaimer: This article is for informational purposes only and does not constitute financial, legal, or professional advice. Consult a qualified professional before making business or investment decisions.
ShareLinkedInX / Twitter
MK
Co-founder · AI & Automation Engineering

Muhammad Kashif is co-founder of ValueStreamAI, leading technical delivery and AI strategy. He designs and ships custom agentic AI and healthcare automation systems for clients across the US and UK. More about Muhammad Kashif →

← back to blog
LIMITED PILOT SLOTS EACH MONTH

Thirty minutes.
We'll tell you exactly
where your ROI is.

A direct conversation with Kash, our founder, not a sales rep. You'll leave with three things: the workflow worth automating first, a rough cost and timeline, and an honest answer on whether AI is the right fix. If it isn't, we'll tell you. No deck, no paid audit, no obligation.

Book a strategy call ->
info@valuestreamai.com - operating across US + UK