Own the metric, not the model

It is easy to fall in love with the model - the architecture, the clever trick, the leaderboard delta. Your business does not care. It cares whether a number moved: conversion, retention, cost per case, time to resolution. The sharpest illustration in the literature is a hypothetical every data leader recognizes as documentary: a churn model that is 98 percent accurate and never acted upon creates zero value, while an 85 percent model that changes retention behavior creates plenty. Accuracy was neither necessary nor sufficient. The number that mattered lived two systems away from the model.

"Own the metric, not the model" is the operating principle that follows - but owning a business metric as an ML team is harder than it sounds, for two structural reasons this post takes seriously: attribution (proving your model moved the number in a system full of confounders) and Goodhart's law (the guarantee that any number you optimize hard enough will eventually betray you). Here is how we wire teams to the business number without falling into either trap.

Table of contents
1. Why model metrics and business value diverge
2. The metric tree: connect the notebook to the P&L
3. Goodhart's law: your proxy will betray you
4. Guardrail pairs: the practical anti-Goodhart kit
5. Attribution: experiments or it didn't happen
6. What "owning" the metric means organizationally
7. The honest limits
8. Takeaways

1. Why model metrics and business value diverge

The divergence is structural, not a failure of diligence. Data products create value indirectly - through downstream decisions made by the people who consume the outputs - so there are three links in the chain, and model quality is only the first: the model must be right, the decision must actually consume it (not the planner's spreadsheet next to it), and the action must reach the customer. A better AUC improves link one and does nothing for links two and three, which is why model metrics and business metrics can move in opposite directions - reality changed, or the model simply is not being used the way it was designed, or at all.

We have written before about the decision half of this problem (forecasts are decisions, not dashboards); this post is about the measurement half. The rule of thumb: monitor both layers, and treat a divergence between them as a debuggable defect - because it is one. Either the model degraded, the decision loop broke, or your attribution was fiction all along. All three are findable.

2. The metric tree: connect the notebook to the P&L

The instrument that makes "own the metric" concrete is a metric tree: the business's north-star number decomposed into operational input metrics, with a causal chain from each team's work to the top. For an ML team, the tree is what turns vague accountability into an explicit, checkable path:

Net revenue retention            <- business owns
  └─ churn rate                  <- product owns
      └─ save-offer acceptance   <- lifecycle team owns
          └─ at-risk detection   <- ML TEAM OWNS THIS LINK:
             recall@top-decile      "of customers who churned,
             + lead time            how many did we flag,
                                    how early?"

Notice what the tree does: the ML team's technical metric (recall at the top decile, with lead time) is chosen because it is the input the next link consumes, not because it is standard. If save-offer acceptance stalls while recall improves, the tree says exactly where to look - the handoff between flagging and offering. Without the tree, that same situation reads as "the model works but the business number didn't move," which is the sentence that gets ML budgets cut.

Two cautions from the north-star literature apply directly: revenue itself is a poor north star (pick the variable that feeds it), and a metric tree is a map of hypotheses, not facts - each edge ("more early flags cause more saves") deserves eventual experimental confirmation, which is section 5's job.

3. Goodhart's law: your proxy will betray you

Now the trap on the other side. Goodhart's law - "when a measure becomes a target, it ceases to be a good measure" - is not a proverb in ML; it is a measured phenomenon with a formal literature. The reinforcement-learning formalization shows the exact dynamic: optimizing a proxy improves the true objective up to a point, then the correlation breaks and further optimization makes the true objective worse. And the theory carries an uncomfortable proof: no non-trivial proxy can be guaranteed unhackable. There is no clever metric that ends this problem; there is only managing it.

It matters more every year because optimization power keeps increasing - faster experimentation, automated decision systems, and models that discover proxy-hacking strategies no human anticipated. The classic production example: support-automation ticket volume falls (target achieved!) while reopen rates and silent churn climb, because the system learned to close tickets rather than resolve problems. The optimizer did its job; the objective was under-specified.

4. Guardrail pairs: the practical anti-Goodhart kit

Since unhackable metrics don't exist, the working defense is structural, per the specification-gaming playbooks:

• Every target metric ships with guardrails that measure what gaming it would break. Ticket deflection pairs with reopen rate and time-to-human-escalation. Engagement pairs with regret signals (unsubscribes, session abandonment). Approval rate pairs with downstream default rate. The pair is designed by asking the red-team question: if you paid a human to maximize this KPI, what would you regret? Whatever you'd regret is the guardrail.
• Frozen evals. A held-out evaluation set the optimizing team cannot see or influence, refreshed on a schedule. The moment the eval is part of the feedback loop, it is a proxy like any other.
• Stacked metrics over single numbers. Progress means the panel moves together; one number sprinting ahead of its panel is a gaming alarm, not a win.
• Human review on the high-stakes slices. Sampled, structured, and boring - the last line that catches what instrumentation can't name yet.

This is also why "own the metric" must never collapse into "be paid on the metric alone." The team should own the pair - target plus guardrails - and celebrate the panel, or the incentive design itself becomes the Goodhart attack.

5. Attribution: experiments or it didn't happen

Owning a metric implies proving you moved it, and the business number is surrounded by confounders: seasonality, pricing changes, the marketing campaign that launched the same week. The standard, per the MLOps ROI guidance, is to treat model impact exactly the way product teams treat features: A/B tests, holdout cohorts, and attribution analysis - with the deltas converted to revenue or cost.

• The gold standard is a persistent holdout: a small slice of traffic (2-5 percent) that gets the old decision process indefinitely. It is the difference between "churn fell 12 percent after we launched" and "churn is 12 percent lower than the holdout" - the second sentence survives a CFO meeting; the first survives until someone remembers the pricing change.
• Where a holdout is impossible (regulated decisions, tiny populations), pre-registered quasi-experiments - staggered rollouts, difference-in-differences across regions - beat before/after storytelling by a wide margin, and cost only planning.
• Report in currency, on a cadence. The deliverable is a recurring one-pager: metric delta against control, converted to money, with confidence bounds. In our experience the discipline of producing this document quarterly changes the ML roadmap more than any architecture decision - projects that can't plausibly appear on it stop getting started.

6. What "owning" the metric means organizationally

The word "own" does real work, and it is organizational, not technical. Per the data-product KPI guidance, every key metric needs a named owner accountable for monitoring and action; for an ML team, ownership means four commitments:

• The team's success is stated in the business number - "reduce cost per resolved case 15 percent" - with the model metrics as internal instrumentation, never the headline. (The alternative is being measured by how much you ship, which is how teams end up shipping a lot and mattering little.)
• The team follows the metric past the model boundary. If the flag isn't consumed, the offer isn't sent, or the action isn't reaching customers, the metric owner chases it - even though the fix is someone else's system. That chasing is the job; the alternative is owning a number you can't influence, which is accountability theater.
• The team can decline work that doesn't serve the metric - including glamorous model upgrades with no plausible path to the tree. The metric is a prioritization razor, not just a scoreboard.
• Divergences trigger investigation, not blame. Model up, business flat is a broken link in the tree; finding it is the team's fastest route to real impact - usually faster than another point of AUC.

7. The honest limits

• Attribution has a floor. Small effects on noisy metrics need more traffic than many businesses have; sometimes the honest statement is "consistent with a positive effect, not provable at our scale." Say that, rather than manufacturing certainty - the credibility compounds.
• Some valuable work has no near-term metric. Platform investments, data quality, exploration - forcing everything through this quarter's business number starves the roots. The remedy is an explicit budget split (say, 70 percent metric-driven, 30 percent foundational), decided once, so each piece of foundational work doesn't need to fake a business case.
• Metric trees drift. The causal edges you drew last year weaken as the product changes; re-validate the tree annually or it becomes a beautiful diagram of a business that no longer exists.
• Goodhart applies to the north star too. The north-star literature's own warning: don't blindly follow a metric visibly taking you the wrong way. Ownership includes the standing to say the number itself is now wrong.

8. Takeaways

• Value is created at the decision, not in the notebook - a 98 percent model nobody acts on is worth exactly zero. Monitor model and business layers together; treat divergence as a defect to debug.
• Build the metric tree: your technical metric should be chosen because the next link in the chain consumes it, and the tree tells you where to look when the top number stalls.
• Goodhart is a theorem, not a vibe: every proxy is hackable. Ship every target with guardrail pairs designed by the red-team question, frozen evals, and stacked panels.
• Attribution runs on experiments: persistent holdouts where possible, pre-registered quasi-experiments where not, and a quarterly one-pager in currency.
• Ownership is organizational: success stated in the business number, authority to chase it across system boundaries, license to decline metric-irrelevant work, and a protected budget for foundational investment.
• Love the metric enough to challenge it. The model is a means; so, eventually, is the metric.

References

1. Data product KPIs: metrics that actually drive business value (Promethium)
2. Monitoring ML systems in production: which metrics to track (Evidently AI)
3. 14 MLOps KPIs for ML teams to measure and prove ROI (Galileo)
4. Goodhart's law in AI: when metrics become targets, models fail (Practical DevSecOps)
5. Goodhart's law in reinforcement learning (arXiv)
6. Take Goodhart seriously: principled limits on general-purpose AI optimization (arXiv)
7. Specification gaming, Goodhart's law, and the metrics (explainx.ai)
8. How the north star metric guides product success (Statsig)

Hero image: Liam Briese on Unsplash.

We take on the metric, not just the model - and stay until it moves. Explore our other insights or get in touch if you would like to talk it through.