From Pilot to Production

The pattern is familiar enough to be predictable. A team builds an AI prototype in a few weeks. The demo lands well. There is genuine enthusiasm, a steering committee, sometimes a press mention. Then eight months later the same capability is still described as a pilot, the original engineer has moved to another project, and someone in finance is asking what the platform contract is actually for.

The instinct is to blame the technology, and it is almost always wrong. In our engagements, pilots that stall rarely stall on model quality. They stall because nobody agreed in advance what would justify promoting the pilot, because the data access that made the demo possible was a one-off export, because no team owns the thing in production, or because the last twenty percent of the work is the integration nobody scoped.

This guide is written for the person who has to decide whether to fund the next phase. It covers how to design a pilot that can be promoted, the six blockers that actually stop them, and a 90-day sequence for getting one across the line.

What this guide covers

1. Why the demo-to-production gap is structural, not technical
2. Designing a pilot with a promotion bar defined up front
3. The six blockers, in the order they appear
4. Ownership: the question that decides more outcomes than any other
5. Run cost, and why the pilot's economics mislead
6. A 90-day sequence from scoped pilot to production
7. The promotion decision, and how to kill a pilot well
8. Questions to ask before funding anything

The gap is structural

A prototype and a production system are optimised for different things, and the difference is not polish. A prototype exists to answer "could this work?" as cheaply as possible. It is allowed to use a hand-cleaned data extract, run on one person's laptop, be evaluated on examples chosen by its author, and have no error handling, monitoring, or access control. Those are not shortcuts; they are correct choices for the question being asked.

A production system answers a different question: "will this keep working, for everyone, without supervision?" That requires a live data path, a defined owner, permissions, monitoring, a rollback plan, a support route for when it is wrong, and a cost that survives contact with real volume. Almost none of that work is visible in a demo, which is precisely why the demo is a poor predictor of effort remaining.

The demo is not eighty percent of the system. It is the part of the system that was easy to build, which is a different statement entirely.

This is the honest framing to bring to a steering committee. It reframes "why is this taking so long" from a delivery failure into a scope conversation that can be planned and funded.

Design the pilot so it can be promoted

The single highest-leverage intervention happens before any code is written: write down what result would justify promoting the pilot, and get the sponsor to agree to it in writing. Pilots without a promotion bar do not fail, which sounds good and is the actual problem. They enter an indefinite state where every review produces another round of refinement because there is no threshold to clear.

A promotion bar has four parts:

• A quality threshold, measured on an agreed set. Not "the answers look good" but a number on a held-out set of real cases, labelled by the business expert who will use the system. Include the baseline: what does the current manual process score on the same set? That number is frequently lower than anyone expects, and it reframes the entire discussion.
• A business metric with a direction and a size. Hours saved per week, percentage of tickets deflected, days off a cycle time. It does not need to be precise; it needs to exist and to be something the sponsor cares about.
• An operating cost ceiling. A per-unit cost above which the economics do not work, calculated at realistic volume rather than pilot volume.
• A failure policy. What happens on the cases the system gets wrong. Human review queue, confidence routing, or explicit scope limits. A pilot that has not answered this is not ready to touch real users regardless of its accuracy.

Writing these four down takes an afternoon and changes the trajectory of the project more than any modelling decision that follows.

The six blockers

These appear in a fairly consistent order, and each has a characteristic symptom.

1. Data access was never real

The pilot ran on an extract someone pulled manually. Production needs a live, permitted, monitored path to the same data, which touches source system owners, security review, and sometimes a vendor contract. Symptom: the phrase "we just need to productionise the data pipeline" appearing in three consecutive status updates. Mitigation: in the pilot, ingest through the same mechanism production will use, even if it runs nightly and covers one table. Prove the path, not just the model.

2. Nobody owns it

The pilot was built by an innovation team, a data science group, or an external partner. None of them run production services. Symptom: the promotion discussion keeps deferring on "which team this sits in". Mitigation: name the receiving team before the build starts and involve them in the pilot, even lightly. A system with no owner does not get promoted no matter how well it performs.

3. Integration was out of scope

A capability that lives in its own interface gets used by the people who were in the demo. Value shows up when it appears inside the tool people already work in: the CRM, the ticketing system, the ERP screen. That integration is usually the largest single piece of remaining work and it is routinely omitted from the original estimate. Mitigation: scope the integration surface at the same time as the model work, and treat "where does this appear in someone's day" as a design requirement rather than a rollout detail.

4. Security and compliance arrive late

The security review that a pilot skipped is mandatory for production, and it can add months if it starts at the end. Data residency, PII handling in prompts, subprocessor agreements with model providers, retention, and audit logging are all answerable, but only on a schedule. Mitigation: book the security conversation in the first two weeks. Arriving with a written data-flow answer changes the tone of that meeting completely.

5. The economics were never modelled at volume

Pilot inference costs are trivially small because pilot volume is trivially small. Multiply by real traffic, add retries, add the long tail of large inputs, and the picture can change materially. Mitigation: model per-unit cost at projected volume during the pilot, and treat cost per successful outcome as a first-class metric alongside accuracy.

6. There is no way to tell whether a change helped

Without an evaluation harness, every improvement is anecdotal, and teams enter a cycle of changing prompts in response to the most recent complaint while quietly regressing cases that used to work. Symptom: disagreement about whether the system is better than it was a month ago, with no way to settle it. Mitigation: build the eval set during the pilot. It is the asset that makes everything after the pilot improvable on a schedule.

Ownership decides more than anything else

If we could ask one question to predict whether an AI pilot reaches production, it would not be about the model, the data, or the budget. It would be: which named team will be paged when this is wrong at 9am on a Tuesday, and have they agreed?

Ownership is what converts a capability into a service. It brings an on-call rotation, a support path, a change process, and a place in a roadmap. Pilots built entirely outside the owning team tend to arrive as an unwanted inheritance: undocumented, unfamiliar, and arriving with an expectation that it already works.

The practical move is to involve the receiving team from the first week, even at ten percent of their time, and to make the handoff document a deliverable rather than an afterthought. If a stranger cannot operate, debug, and extend the system from that document, the handoff has not happened regardless of what the project plan says.

Run cost, honestly

Three costs are routinely underestimated when a pilot is promoted:

• Inference at real volume, including the retries and the long-input tail. Bound context length deliberately; it is the main lever.
• The human in the loop. If the failure policy routes a percentage of cases to a reviewer, that reviewer's time is part of the system's cost. A system at ninety percent accuracy with a ten percent review queue is a real and often correct design, but only if the queue is staffed and counted.
• Maintenance. Model deprecations, prompt drift as the corpus changes, eval set upkeep, and the ordinary care of a running service. Budget continuing engineering time rather than treating launch as the end of spend.

The metric worth reporting to a sponsor is cost per successful outcome, not cost per API call. It is the only number that compares sensibly against the manual process being replaced, and it makes the human-review component visible instead of hidden.

A 90-day sequence

This is the shape we use when a customer has an idea and wants it in production rather than in a demo. It assumes one engineer with part-time access to a business expert.

Days 1 to 15: scope and access

• Discovery interviews with the people who do the work today; quantify the current process.
• Write the one-pager: problem, proposed slice, promotion bar, out-of-scope list. Get written confirmation.
• File every data access and environment request. This is the critical path; start it on day one.
• Book the security conversation and bring a draft data-flow answer.
• Build the eval set: 30 to 100 real cases labelled by the business expert, plus the manual baseline.

Days 16 to 45: thin slice, real path

• Build the narrowest end-to-end version that produces a real output for a real case.
• Ingest through the mechanism production will use, even if it is nightly and partial.
• Score against the eval set weekly; keep the harness in CI from the first week.
• Demo every week to the sponsor on their data, including the failures.

Days 46 to 70: integration and hardening

• Put the capability where the work already happens, rather than in its own interface.
• Implement the failure policy: confidence routing, review queue, or scope limits.
• Add monitoring, per-request tracing, permissions, and audit logging.
• Model cost at projected volume; tune context and caching against that number.

Days 71 to 90: limited production and handoff

• Release to a bounded real audience, with a rollback path and a named owner on call.
• Measure the business metric, not just the quality metric.
• Write the runbook and the handoff document; have the owning team operate it while you watch.
• Hold the promotion decision against the bar written on day 10.

Ninety days is not a guarantee, and a genuinely hard integration or a slow access process will stretch it. What the sequence buys is that the expensive unknowns - access, security, ownership, and cost - are surfaced in the first two weeks instead of the last two.

The promotion decision, and killing a pilot well

At the end, the pilot meets the bar or it does not. Both outcomes are acceptable; the unacceptable outcome is the third one, where the pilot continues indefinitely because no one wants to make the call.

A pilot that misses the bar has still produced things worth keeping: an eval set, a working data path, a documented understanding of why the problem is harder than it looked, and a cost model. Write a short honest note covering what was tried, what the numbers were, and what would need to be true for this to work later. That note is what stops the same idea being re-piloted from zero in eighteen months by someone who never heard about this attempt.

An organisation that cannot kill a pilot cleanly will run out of capacity to start new ones. The willingness to stop is what makes the portfolio work.

Questions to ask before funding anything

If you take one thing from this guide, take this list into the next steering meeting:

1. What result would justify promoting this, and who agreed to that number?
2. What is the current manual process scoring on the same cases?
3. Which named team will own this in production, and have they agreed?
4. Is the pilot reading data through the path production will use?
5. Has security seen the data flow, and what did they say?
6. What does this cost per successful outcome at real volume?
7. What happens to the cases it gets wrong?
8. Where does this appear in someone's working day?
9. How will we know next quarter whether a change made it better or worse?
10. What is the date on which we decide to promote or stop?

A team that can answer these is not necessarily going to succeed, but a team that cannot answer questions three, five, and ten is going to spend a year finding out the hard way.

Honest boundaries

None of this rescues a project whose underlying idea does not work, and some ideas do not. A promotion bar makes that visible in ten weeks rather than ten months, which is the point. Nor does the sequence above suit every context: a genuinely exploratory research effort should be funded and judged as research, not squeezed into a production timeline.

What it does address is the specific and very common failure of a capability that works and still never ships, because the organisational questions were treated as details to sort out later. They are not details. They are the project.