Shipping ML that survives contact with production

The hard part of machine learning was never the notebook score. It is keeping that score in production, where the data shifts, users adapt to the model, and traffic spikes at the worst moment. The industry's own numbers frame the problem: the oft-cited figures have 87 to 90 percent of models never reaching production, and among those that attempt the crossing, half take three months or more. The ones that make it and then quietly rot are their own category - because ML systems don't fail like software; they degrade silently while the uptime dashboard stays green.

A model that survives contact with production is not a smarter model. It is an ordinary model wrapped in a deliberately boring delivery system: shadow validation, gated canaries, tested rollback, and the testing rubric Google formalized years ago that most teams still haven't adopted. This post lays out that system in the order you should build it - and the order is the point, because every piece exists to catch a specific, named failure before users do.

Table of contents
1. How production actually kills models
2. The staged rollout: never ship directly
3. Shadow mode: the cheapest insurance in ML
4. Canary: blast radius with a measurement plan
5. Rollback is a capability, not a runbook page
6. The ML Test Score: grade yourself before production does
7. Watch the system, not the model
8. The honest costs
9. Takeaways

1. How production actually kills models

The postmortem catalog compresses into four mechanisms, none of which is "the model was bad":

• The environment mismatch. As an engineer who worked on Instagram Reels ranking puts it: when models fail after deployment it's rarely the model - production environments differ fundamentally from training environments, and they're dynamic. Dependency skew, serialization bugs, preprocessing that behaves differently under the serving stack - artifacts that ran perfectly in the notebook crash or, worse, run wrong.
• Training-serving skew - the number-one silent killer, covered in depth in our companion post on broken features. The model is fine; its inputs stopped meaning what they meant.
• Launch-window concept drift. Users interact with the live model in ways the training distribution never represented - metric divergence in the first 24 to 72 hours is the signature. Related: the model's own outputs shape tomorrow's training data, the feedback loop that makes ML systems self-influencing in a way normal software never is.
• Averages hiding segment failures. Aggregate accuracy holds while one segment - a geography, a device type, a customer tier - fails badly. Weeks later the business metric slips and the incident review starts from zero.

Every component below maps to one or more of these. That mapping discipline matters: infrastructure justified by a named failure mode gets maintained; infrastructure installed as ceremony gets deleted in the next reorg.

2. The staged rollout: never ship directly

The mature pipeline, per the consolidated deployment guidance, is a gated progression:

offline eval gate          # beats champion on held-out + slices
  -> shadow deploy         # full traffic, invisible outputs
  -> canary 1% -> 5% -> 25%  # automated metric gates between steps
  -> A/B experiment        # statistical, if the decision needs it
  -> champion status       # with drift monitors + auto-rollback armed

# Each arrow is a GATE with:  promote / hold / rollback criteria
# written BEFORE the deploy, not negotiated during it.

The gate criteria are the substance. A canary defined as "send 5 percent of traffic to the new model" is a deployment strategy without a measurement strategy - incomplete in exactly the way that produces 2 a.m. arguments about whether to proceed. Every gate needs three numbers agreed in advance: the metric that must improve (or hold), the guardrails that must not degrade, and the observation window.

3. Shadow mode: the cheapest insurance in ML

Shadow deployment runs the challenger against full production traffic, logging its predictions while users see only the champion's. It costs one more inference per request for a week or two, and it is the only stage that tests the true production environment with zero user risk. What it catches, concretely:

• Environment breakage - the dependency and preprocessing mismatches from section 1 surface here as crashes and latency spikes nobody's customer experiences.
• Skew on real traffic - compare the challenger's live feature values and prediction distribution against its offline expectations; a gap is a bug found before launch instead of after.
• Segment surprises - slice the shadow logs by every dimension that matters commercially. This is where "great on average, terrible in Germany" gets caught.

The discipline shadow mode enforces is deeper than the mechanism: it forces you to define, in logs and queries, what "behaving correctly" means before any user is exposed. Teams that can't write the shadow-analysis queries aren't ready to canary - that inability is the finding.

4. Canary: blast radius with a measurement plan

The canary phase trades shadow mode's safety for the one thing shadow can't give you: real user responses (clicks, conversions, complaints) to the challenger's outputs. Managing the trade:

• Success metrics AND guardrails, separately tracked. The canonical trap: the model improves CTR while silently degrading page latency or revenue per session - offline evaluation cannot see this because it measures the model in isolation, not the system end to end. Guardrails are the system metrics.
• Statistics at small percentages are real. At 1 percent traffic, rare-event metrics need days to say anything; decide the observation window from power math, not impatience. (For low-traffic products, skip fine-grained canaries - 10 percent steps with longer windows beat theatrical 1 percent steps with meaningless reads.)
• Run canary analysis against production traffic only. Staging metrics diverge from production because traffic distributions differ - a staging-validated canary gate is a gate made of paper.
• Sticky routing for session-shaped products. A user bouncing between champion and challenger mid-session contaminates both measurements; route by user, not by request, wherever the experience is stateful.

5. Rollback is a capability, not a runbook page

The failure nobody plans for, straight from the field guides: rollbacks fail because the previous artifact cannot be rebuilt - teams version code but skip data versions, configs, and environments. "Revert to the old model" assumes the old model still exists as a runnable, byte-identical artifact with its exact feature transforms and config. Making that true requires:

• Immutable, complete lineage per deployment: model binary, training-data hash, code SHA, feature-pipeline version, serving config - every deployment traceable to a Git commit and training run. If any element is "latest," your rollback is a rebuild, and rebuilds fail at 2 a.m.
• Automated triggers with pre-agreed thresholds: when a canary gate's guardrail breaches, traffic reverts without a meeting. Minutes-to-safe, not hours-to-consensus.
• The previous champion kept warm - not archived - for at least the post-promotion observation window.
• Drills. The guidance worth underlining: quarterly rollback drills under time pressure, validating that measured MTTR meets the SLA. A rollback procedure that has never been executed is a hypothesis with a pager attached.

6. The ML Test Score: grade yourself before production does

For the "are we actually ready" question, the best instrument remains Google's ML Test Score (Breck et al.): 28 concrete tests across four sections - data dependencies, model development, serving infrastructure, and monitoring. Two design choices make it bite:

• The score is the MINIMUM across sections, not the sum. Brilliant monitoring cannot compensate for untested data pipelines; the rubric enforces the weakest-link reality of production ML. Zero points reads as "a research project, not a production system"; 3-4 is a first pass at productionization.
• It finds specific rot. The paper's own anecdote: one team's structured walkthrough surfaced a thousand-line, completely untested file that generated their model's input. Every team that runs the rubric finds its own version of that file.

Run it as a half-day workshop per production system, before launch and annually after. One honest caveat: the rubric predates the generative era - for LLM applications, extend it with task-success, safety-pass-rate, and P95-latency release gates in the same promote/hold/rollback shape; the newer quality-gate frameworks are that extension.

7. Watch the system, not the model

Post-promotion, the monitoring stack in priority order - inputs first, because that is where problems announce themselves earliest (the full argument is in the broken-feature post): per-feature distributions against training baselines; prediction distribution and calibration; guardrail system metrics (latency, cost per request - see why cost is a product number); segment-level performance, never just aggregates; and delayed ground-truth accuracy when labels arrive. Plus the one artifact that pays for everything: log the full feature vector with every prediction, so any future incident starts from a query instead of a reconstruction.

And route each alert to the team that can act on it: feature drift to the data-pipeline owners, guardrail breaches to the serving owners, quality decay to the model owners. Misrouted ML alerts age into ignored ML alerts on a schedule of about two weeks.

8. The honest costs

• Time-to-first-launch roughly doubles. Shadow weeks, canary windows, lineage plumbing - the first model through the pipeline pays for the pipeline. The second model ships faster than it would have under ship-and-pray, and every model after that compounds the return. Amortize consciously or the org will cut the investment at the first deadline.
• Double-inference bills during shadow. Real but bounded; for large models, shadow on a traffic sample instead of the full stream and accept the weaker segment coverage.
• This stack is sized for models that matter. An internal tool scoring leads for one team does not need quarterly rollback drills. Tier your rigor: full pipeline for revenue-touching and user-facing models, shadow-plus-rollback minimum for the middle tier, notebook-and-honesty for true experiments - with the tier written down, so "experiment" doesn't become the permanent label that dodges the pipeline (the prototype-purgatory post is about exactly that dodge).

9. Takeaways

• Production kills models via environment mismatch, skew, launch-window drift, and hidden segment failures - not notebook error. Build for those four, by name.
• Never ship directly: offline gate, shadow, gated canary, then champion - with promote/hold/rollback criteria written before each deploy.
• Shadow mode is the cheapest insurance in ML: full production reality, zero user risk. If you can't write its analysis queries, you're not ready to canary.
• Canaries need success metrics and guardrails tracked separately, real statistical windows, and production traffic only.
• Rollback is an artifact-lineage capability, proven by quarterly drills - not a paragraph in a runbook.
• Score yourself with the ML Test Score (minimum across sections, not the sum), extended with LLM-era gates where relevant - and monitor the system, inputs first, forever.

References

1. The ML Test Score: a rubric for ML production readiness (Google Research / IEEE)
2. Safe ML model rollout: canary, shadow mode, and rollback (CalibreOS)
3. AI deployment automation guide 2026: CI/CD, GitOps and MLOps (ValueStreamAI)
4. Advanced deployment patterns: canary and shadow testing (APXML)
5. Why ML models break after deployment (DEV)
6. How ML models fail in production (Interview Node)
7. Evaluating ML system readiness (TrueFoundry)
8. Why 87 percent of models fail to reach production (Innovatics)

Hero image: Dominik Lückmann on Unsplash.

We wrap models in the delivery system that keeps them alive. Explore our other insights or get in touch if you would like to talk it through.