An AI proof of concept is a short, timeboxed experiment that tests whether AI can do one specific job well enough, on your real data, to justify building it properly. A good one ends in a clear go or no-go decision, judged against a measured baseline and success criteria agreed before any building starts.
The problem is that many PoCs do not end that way. They end in a demo that impressed people, a vague sense that "it mostly works" and no decision. This guide covers how to set one up so the answer is clear, what to deliver at the end, and why so many stall.
What is an AI proof of concept, and what is it not?
An AI PoC answers one question: can this approach reach the accuracy, speed or cost we need on this task, with this data? It is deliberately narrow. It does not need a polished interface, single sign-on or production hosting.
It is also not a pilot or an MVP. People use the terms loosely, which causes confusion about what is being promised and paid for.
What is the difference between an AI PoC, a pilot and an MVP?
| Proof of concept | Pilot | MVP | |
|---|---|---|---|
| Question it answers | Can AI do this task well enough on our data? | Does it work in the real process, with real users? | Will people use and value the product? |
| Scope | One narrow use case | One team, site or process, live | Smallest usable product for a wider audience |
| Data | Real historical data, offline | Live data, real transactions | Live data at production volume |
| Users | Project team and subject experts | A small group of real users | Real customers or staff |
| Typical length | A few weeks | Several weeks to a few months | Months |
| Main output | Measured results and a go/no-go decision | Operational evidence and rollout plan | A working product and adoption data |
| Production engineering | Minimal | Partial (security, monitoring, support) | Full |
The order matters. A PoC proves the core capability. A pilot proves it survives the real process. An MVP puts a usable product in front of a wider group. Skipping the PoC means discovering accuracy problems after you have paid for integrations and user interfaces.
How do you choose the right use case for an AI PoC?
Pick one task, not a theme. "Use AI in finance" is a theme. "Match supplier invoices to purchase orders and goods receipts, and flag mismatches" is a task.
A good PoC use case has four properties:
- It happens often enough to matter. Hundreds or thousands of times a month, not ten.
- You can measure today's performance. Time per item, error rate, cost per item or turnaround.
- You have historical examples with known outcomes. These become your test data.
- Someone owns the process and will make the go/no-go call.
Avoid tasks where a wrong answer is catastrophic and there is no human check, or where the rules are fully fixed and ordinary automation would do the job more cheaply.
What should you decide before building anything?
This is the step most often skipped, and it is the one that makes the final decision easy.
Measure the baseline
Record how the task performs now. How long does a person take per item? What is the error rate? How many items wait more than a day? Without a baseline, every result looks either impressive or disappointing depending on who is in the room.
Define success criteria and a kill threshold
Write down, before building, what result means go, and what result means stop. For example:
- Go: at least 95% of invoices matched correctly on the test set, with every mismatch flagged rather than silently accepted.
- Kill: below 85% correct after two rounds of improvement, or any case where a wrong match is passed without a flag.
The gap between the two is a grey zone where you may extend or narrow scope. The kill threshold is what stops a PoC drifting for months. Agree it with the process owner and the budget holder, and put it in writing.
Check data access and data protection
Confirm you can actually get the data out of the systems that hold it, in a usable format, within the first week. If the PoC uses personal data, check whether you need a data protection impact assessment. The ICO's DPIA guidance explains when processing counts as high risk, including where innovative technology is involved.
How do you run an AI proof of concept step by step?
Get real data in the first week
Demo data hides the hard cases. Real invoices have handwritten notes, credit lines and supplier typos. Real emails are forwarded chains with attachments. If the first week ends without real data in hand, treat it as a warning sign for the whole project.
Build an evaluation set
Take a few hundred real historical cases where the correct answer is known, including awkward ones. Split off a portion you never look at while tuning, so the final result is honest. Every change to prompts, models or logic gets run against the set, and the score is recorded.
Historical data is often the best evaluation set you have. In our Steel Quoting Engine work for a steel fabricator, the engine was backtested on past quotes before it was trusted, and quote turnaround went from 2 days to 4 minutes. Backtesting against decisions your team has already made is a direct way to compare the AI with the baseline.
Design for exceptions from the start
Most useful business AI does not handle every case. It handles the routine ones and routes the rest to a person. In the Invoice Reconciliation AI project, a finance team with around 900 suppliers reached a 99.98% three-way match rate, with only exceptions reaching a person. Measure how many cases the system is confident about, how accurate it is on those, and how cleanly it hands off the rest.
Timebox the work
A PoC should run for a few weeks, with a fixed end date. Weekly check-ins should show the evaluation score moving. If it has flattened well below the success line by the midpoint, discuss stopping early. A short PoC that ends in a "no" is a good outcome: it saved you the cost of a full build.
What should you get at the end of an AI PoC?
Insist on a decision pack rather than just a demo. It should include:
- Results against the baseline and criteria, on the held-out test data.
- An error analysis: which cases fail and why.
- A recommendation: go, extend with a narrower scope, or stop.
- Running-cost estimate at production volume, based on measured token use and calls per task. Our guide to AI agent development cost in the UK shows how to work this out.
- A path to production: integrations needed, security and data requirements, human review design and an outline plan for a pilot.
- The code, prompts and evaluation set, so the work is not lost if you change supplier.
Why do AI proofs of concept stall?
Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. In the PoCs we see, the causes are usually more mundane:
- No baseline. Nobody can say whether 88% accuracy is good, because nobody measured the current process.
- Demo data. Clean sample data produces great results that collapse on real inputs.
- No owner. The PoC is run by IT or an innovation team, and no process owner is willing to make the call.
- No path to production. The PoC works, but nobody has budgeted for integrations, security review or support, so it sits on a laptop.
- Moving goalposts. Success criteria were never written down, so they change after the results arrive.
Every one of these is fixed before building, not after.
Frequently asked questions
How long should an AI proof of concept take?
Most AI PoCs should run for a few weeks with a fixed end date. That is long enough to get real data, build an evaluation set and iterate a few times. If the work has no end date, it tends to drift into an unplanned pilot without a decision.
What is the difference between an AI PoC and an MVP?
An AI PoC tests whether the AI can do a specific task well enough on real data, usually offline and with minimal engineering. An MVP is a usable product put in front of real users to test whether they adopt and value it. A PoC comes first and de-risks the MVP.
What is an AI pilot?
An AI pilot runs the system live in a real process, with a small group of real users, after a PoC has shown the core capability works. It tests operational fit: handovers, exceptions, user trust and support. It produces the evidence needed for a wider rollout.
What success criteria should an AI PoC have?
Success criteria should be measurable on held-out real data and compared with a baseline from the current process. Typical measures are accuracy, the share of cases handled without a person, time per item and cost per item. Set a kill threshold at the same time, so a poor result leads to a clear stop.
What happens if the PoC fails?
A PoC that misses its criteria has done its job, because it stopped you funding a full build that would not work. The error analysis often shows a narrower task that does pass, or a data problem to fix first. Keep the evaluation set, as it makes any second attempt faster and cheaper.
Where to start
If you have a process in mind and want to test it properly before committing to a full build, we run timeboxed PoCs on your own data with the success criteria and kill threshold agreed up front. See how our AI proof of concept service works.
