A working demo used to be reasonable evidence. Building something that looked finished required roughly the same skills as actually finishing it, so if a vendor showed you a polished conversational flow, you could reasonably infer they could build the real thing.
That link is broken. AI-assisted development means a convincing end-to-end demo can now be produced by someone who could not debug it, extend it, or explain why it works. This is not an argument against AI-assisted development, we use it heavily. It is an argument that the demo has stopped being a signal, and buyers need a different one.
This guide gives you seven questions that work as that replacement signal. You do not need to understand the answers technically. You only need to notice whether each answer arrives immediate and specific, or general and reassuring.
| What you are testing | What a real answer sounds like |
|---|---|
| Failure handling | A named path: retry, fallback, escalate to a human |
| Integration depth | Named systems, named APIs, named auth method |
| State and recovery | Where state lives, what happens if the process dies |
| Who does the work | An individual's name, not a department |
RAG vs AI Agent: The Stack Behind Most Fake AI Agencies
Before the questions, one pattern worth recognising, because once you see it you cannot unsee it and it explains a large share of what is currently sold as agentic AI.
The pattern is: a database, an OpenAI API key, a set of documents embedded into a vector store, and a thin interface on top. That stack is then named after whichever industry the vendor is selling into. Embed CVs and it becomes "an AI resume-screening system for HR." Embed policy documents and it becomes "a compliance agent." Embed patient information leaflets and it becomes "a medical AI assistant." The architecture never changes. Only the label does.
The important thing is that this stack is genuinely useful for one job: semantic search across a document set. If that is what you need, it is a reasonable solution and you should not overpay for it.
It is inadequate for what it is usually sold as, because retrieval is not reasoning. A system that finds the passage most similar to your question is not making a decision. It does not take actions, it holds no state between turns, it has no concept of having been wrong, and it cannot do anything about it if it were. Calling that an agent is not mild exaggeration, it describes a different category of system entirely.
How to check without technical knowledge: notice whether the proposal describes what the system knows or what it does. Thin wrappers are described in terms of knowledge: trained on your documents, understands your policies, knows your product catalogue. Real agents get described in terms of actions: writes to this system, updates that record, escalates in this circumstance. If the entire proposal is about knowledge and none of it is about actions, you are looking at search with a chat interface.
7 Questions to Ask an AI Agency Before You Sign
1. What happens when the model returns malformed JSON mid-task?
Why it works: anyone who has run one of these in production has been burned by this, because it is not an edge case, it is a weekly occurrence. Language models are non-deterministic. Even with structured output modes enabled, a multi-step task will eventually produce something unparseable.
A real answer names a validation layer, a retry with a corrective prompt, and a defined fallback when retries are exhausted. A weak answer is that it does not really happen, or that the newer models handle it, or a general statement about the prompt being well written.
2. Where does state live between steps, and what happens if the process dies halfway?
Why it works: this separates people who have built a workflow from people who have built a demo. Demos run start to finish on a happy path. Production systems get interrupted.
A real answer names where state persists and describes what resuming looks like, including how the system avoids repeating an action it already completed. A weak answer treats the question as hypothetical.
3. What is your retry policy on a failed tool call, and how do you avoid executing a write twice?
Why it works: it is the specific version of question two, and it exposes whether they have thought about idempotency. This is the failure that creates duplicate invoices, duplicate bookings, and duplicate customer emails.
A real answer mentions idempotency keys, lookup-before-write, or a deduplication check, and can tell you which operations are safe to retry and which are not. A weak answer says the system retries on failure, without distinguishing reads from writes.
4. Which of our systems will this integrate with, and does each one have a documented API?
Why it works: integration is where most projects actually fail, and it is checkable before any work starts. A vendor who has scoped honestly will have looked.
A real answer names each system and its integration path, and flags any that lack one as a risk with a proposed approach. A weak answer talks about seamless integration with your existing tools without naming them.
This question is worth asking even if you never hire anyone, because you can run the check yourself: list every tool the workflow touches and ask whether each publishes an API or documented integration path. Where the answer is yes across the board, the project is predictable. Where a critical system has no doorway, the work becomes browser automation or a prerequisite re-architecture, and both the cost and the risk profile change substantially.
5. What does the system do when it cannot find a relevant answer?
Why it works: this is the direct test for the stack signature above. A retrieval tool and a real agent give structurally different answers.
A real answer describes a fallback path: escalate to a human, ask a clarifying question, return an explicit "I do not know" rather than a plausible guess. A weak answer is about improving the prompt or adding more documents, which tells you the system has no concept of not knowing.
6. Who specifically will build this, and will they be on calls throughout?
Why it works: the single clearest predictor of a disappointing engagement is the leadership handoff. You have discovery with the founder or technical lead, sign, and then never speak to a technical person again. Your contact becomes a project manager relaying questions to a delivery team that may be subcontracted.
A real answer names an individual with a verifiable background and says yes. A weak answer is "our team will be on the calls." Note that the honest answer to "is any of this outsourced?" matters less than whether they hesitate before giving it.
7. Show me something you built that could not have been produced by prompting.
Why it works: this is the hardest question to fake, and the best single filter on the list. Ask for work where the difficulty was in the constraints rather than the code generation: extracting data from a system with no export function, an integration against an undocumented API, a migration that had to preserve referential integrity across mismatched schemas.
A real answer comes with specifics about what made it hard and what they tried first that did not work. A weak answer is a portfolio of dashboards and chat interfaces, which are exactly the artefacts AI assistance produces well.
For what a genuine answer to this looks like, our patient data migration case study covers extracting a full clinical registry from a legacy EHR that had no export button. That is not a prompting problem, and no amount of model capability substitutes for the work.
AI Agency Red Flags: The Failure Sequence to Watch For
If a vendor without the depth gets the contract anyway, the arc is consistent enough to predict, and knowing it lets you intervene at week four rather than week twelve.
Weeks one to three go well. This is genuine, not a con. The parts AI assistance handles well, scaffolding, UI, the happy path, are exactly the parts you see first. Demos are strong. Confidence is high on both sides.
Then it stalls at the integration boundary. The work stops being generation and starts being judgment about somebody else's undocumented system. Progress updates get vaguer. Timelines move by a week at a time rather than being re-baselined honestly.
Then the quiet rescue. Freelancers or contract engineers get brought in to fix specific pieces, usually without telling you. You find out at handover, when the codebase contains work nobody on the original team can explain.
The early warning sign is the shift from specific updates to general ones. "We finished the Pipedrive write-back and are testing dedupe" becoming "we are making good progress on integrations" is the tell, and it usually appears two to three weeks before the timeline officially slips.
What an AI Agency Quote Actually Tells You About Them
Price is a weaker signal than most buyers assume, and it misleads in both directions.
An unusually low quote is the one people are trained to be suspicious of, and often for the wrong reason. It rarely means shoddy work in the sense of bad code. It usually means the scope has been read differently: the quote covers the happy path and excludes error handling, testing against your real data, and the integration work that has not been checked yet. The quote is not dishonest, it is answering a smaller question than the one you asked.
An unusually high quote is not automatically reassurance either. In a market where demand outstrips genuine capability, confident pricing is easy to imitate. We have seen the same thin-wrapper build quoted at wildly different numbers by different vendors, which tells you price is tracking sales confidence rather than engineering depth.
The check that works better than either: ask what specifically would make the number go up. A team that has scoped the work will name the risks, usually an integration they have not verified or a data quality unknown. A team that has not will tell you the price is all-inclusive, which is the least plausible claim in software.
The Competitor Pulse Check
| Factor | A Team That Can Build It | A Thin Wrapper Operation |
|---|---|---|
| Proposal describes | What the system does, and to which systems | What the system knows |
| Demo shows | A completed action, and a failure path | Question answering only |
| On failure handling | Named retry, fallback, escalation | "The newer models handle that" |
| Portfolio | Constraint-driven work: migrations, legacy integrations | Dashboards and chat interfaces |
| Delivery team | Named individuals, on calls throughout | "Our team", contact is a PM |
| Architecture across verticals | Differs by problem | Identical diagram, different label |
What This Does Not Mean
Three fair caveats, because a checklist like this is easy to misuse.
No-code and RAG are not fraudulent. They are appropriate for a real class of problems. A vendor proposing a no-code build for a simple linear workflow may be giving you the honest, cheapest answer. The problem is not the tool, it is the mismatch between the tool and the label on the invoice.
A small team is not a red flag. Some of the best builders in this space work alone or in pairs. The questions above test depth, not headcount.
Failing one question is not disqualifying. Nobody has every answer instantly. What matters is the pattern across all seven, and specifically whether uncertainty gets acknowledged ("we would need to check how your EHR exposes that") or papered over.
Frequently Asked Questions
How can I vet an AI agency if I am not technical?
You do not need to grade the answers, only their shape. Ask the seven questions above and watch whether each response is immediate and specific or general and reassuring. Someone who has built these systems answers question one in a sentence. Someone who has not will talk around it.
What is the difference between a RAG system and an AI agent?
RAG retrieves relevant text and uses it to answer a question. An agent takes actions: it writes to systems, makes decisions, holds state across steps, and handles failure. A large share of what is sold as an agent is retrieval with a chat interface, which is useful for search and inadequate for anything requiring an action.
Is it a red flag if an agency uses no-code tools?
Not by itself. It is a red flag if they use no-code tools and price the work as custom engineering, or if they cannot tell you where that stack will hit its ceiling. A good vendor will tell you when no-code is genuinely the right answer, even though it earns them less.
How do I know if my current vendor has quietly outsourced the work?
Ask who specifically is writing the code and whether you can speak to them. Then watch for the pattern: progress updates becoming less specific over time, timelines slipping a week at a time, and handover documentation that does not match how the project was described.
What is the single best question if I only ask one?
"Show me something you built that could not have been produced by prompting." Portfolios full of dashboards and chatbots tell you what AI assistance produces well. Work constrained by legacy systems, missing APIs, or data integrity requirements tells you what the team can do when the model cannot do it for them.
What's Next
If you are earlier in the process, our buyer's guide to choosing an AI automation company covers the commercial evaluation and includes a scorecard, and how to choose an AI agent development company goes deeper on the technical side. If you are weighing an agency against hiring, see agency vs in-house hire and how to hire AI agent developers. For why projects with capable vendors still stall, why AI pilots fail covers the structural causes, and why no-code fails at enterprise scale covers where that ceiling actually sits.
Want to run these questions on us? Talk to our team. We will tell you on the first call if your problem does not need custom engineering, which happens more often than you would expect from a page like this.
Muhammad Kashif is co-founder of ValueStreamAI, leading technical delivery and AI strategy. He designs and ships custom agentic AI and healthcare automation systems for clients across the US and UK. Connect on LinkedIn →