The difference between an AI demo and a system that runs the business

Every enterprise has now seen the demo. A model reads a contract, drafts a reply, answers a question about last quarter's numbers. It is impressive, it takes an afternoon to build, and it tells you almost nothing about whether the same capability will survive contact with your business.

The gap between that demo and a system that runs the business is where most enterprise AI budgets quietly disappear. After building production systems for financial institutions, law firms and public bodies, we think the gap comes down to seven things.

1. Your data is the specification

Demos run on clean, cooperative inputs. Production runs on scanned PDFs with coffee stains, spreadsheets with merged cells, emails in three languages and the template a regional office invented in 2014.

The only reliable way to know whether a system works is to build it against your real inputs from the first week. That is why our engagements start by assembling a representative sample of the documents, tickets or records the system will actually face, edge cases included.

2. Accuracy is a number you agree on

"It works pretty well" is not an acceptance criterion. Before we optimise anything, we build a golden dataset with the people who own the process and agree how success is measured: field-level extraction accuracy, resolution rate, false-positive rate, time to answer.

That dataset becomes an evaluation harness that runs on every change. When a model is upgraded, a prompt is edited or a new document type appears, you know within minutes whether quality moved, and in which direction.

3. The model is the smallest part of the system

In a typical production deployment, the call to the model is a small fraction of the code. The rest is ingestion, classification, validation rules, retries, review queues, integrations with systems of record, access control, monitoring and cost controls.

Those unglamorous parts decide whether a system can be trusted at 3 a.m. on the last day of the quarter. They are also what most pilots skip.

4. Compliance is a feature, not a phase

Under GDPR and the EU AI Act, logging, transparency, data minimisation and human oversight are design requirements, not paperwork to complete after launch. Retrofitting them is expensive; designing them in is cheap.

We treat audit trails, data residency, redaction and explainability as first-class features, so that when legal, risk or a regulator asks how a decision was made, the answer is a link, not a project.

5. Humans stay in the loop where it matters

Full automation is rarely the goal. The goal is putting human attention exactly where it creates value. Confidence thresholds route uncertain cases to reviewers; approvals gate irreversible actions; everything else flows straight through.

Done well, this is how accuracy climbs over time: every human correction becomes new evaluation data.

6. You should own what you pay for

Production AI should not be a black-box licence. Our clients own the code, the prompts, the evaluation suites and the infrastructure-as-code. Architectures stay model-agnostic, so you can move between frontier and open-weight models as prices, performance and regulation change.

7. Unit economics are part of the design

A system that is accurate but costs more per document than the analyst it replaces is not a success. We engineer cost per task from the start, with right-sized models, caching, batching and routing, and we report it next to accuracy.

A short checklist before you approve a pilot

• Has it been tested on a representative sample of your data, including the ugly cases?
• Is there a golden dataset and an agreed accuracy metric?
• What happens when the model is uncertain, and who sees it?
• Where does the data live, and what is logged?
• Who owns the code, prompts and evaluations when the engagement ends?
• What does one processed document, ticket or task cost?

If those questions have clear answers, you are not looking at a demo any more. You are looking at a system that can run part of your business.