Instrumenting cost and quality in AI production

"Corporate AI ROI" argued that most AI initiatives fail to prove return not for lack of value, but for lack of instrumentation. This article details how to build that instrumentation — per-use-case telemetry, cost attribution, budgets and alerts, production quality metrics, and sampling for human review.

Why "cost per token" isn't enough

Cost per token is easy to measure and nearly useless for business decisions. The question that matters is: how much does one unit of work cost? A unit of work is defined by the use case — a resolved support ticket, a reviewed contract, a generated report, a question answered accurately enough to avoid human rework.

To calculate cost per unit of work, you need three linked data points:

1. 1.Input and output tokens per call (including intermediate agent and retrieval calls).
2. 2.Number of model calls required to complete one unit of work — don't assume 1:1.
3. 3.Success rate of the unit of work (how often the result was accepted without needing a retry or correction).

\\\text cost_per_unit_of_work = sum(token_cost_across_all_calls) / success_rate \\\

If a task costs $0.08 in tokens but is only accepted 70% of the time, the real cost per delivered unit of work is $0.11, not $0.08. Ignoring the success rate is the most common way to underestimate real cost.

Attribution by team and use case

Without attribution, AI cost becomes a single unowned line on the infrastructure bill. That stalls two sides: finance can't assess ROI per area, and teams don't feel the cost of design decisions (excessive context, unnecessary retries, choosing a bigger model than needed).

Minimum implementation:

• Every call to the AI gateway carries mandatory metadata: team, use case, environment (production/test), and ideally end user or originating process.
• The gateway joins this metadata with the actual cost of the call (not an estimate) and writes it to a queryable store — not just a text log.
• A dashboard allows querying by team and use case, with month-over-month comparison.

This enables decisions like: "use case X consumes 40% of the AI budget and delivers low measurable value — worth revisiting the design or discontinuing" — a decision impossible without attribution.

Budget and alerts

A budget without automatic alerts is a spreadsheet nobody checks until month-end. Structure it in tiers:

"Anomalous burn rate" alerts — consumption growing much faster than usage volume — catch problems before the budget cycle ends, like a poorly tuned prompt that doubled context size.

Production quality metrics

Quality can't be measured only through offline evaluation before launch. In production, continuously monitor:

• Acceptance rate: response used without edit or correction (when there's a user signal, like accepting a suggestion).
• Escalation rate: how often the interaction needs human intervention.
• Model refusal/error rate: when the model refuses to answer or returns a structured error.
• Quality drift: periodic comparison against the reference evaluation set, to detect degradation after a model, prompt, or retrieval data change.

These metrics should live on the same dashboard as cost — quality and cost taken in isolation lead to the wrong optimizations (cutting cost without measuring the effect on quality, or keeping an expensive model without proof the extra quality is being used).

Sampling for human review

Reviewing 100% of interactions doesn't scale; reviewing none means operating blind. The recommended practice is stratified sampling:

• Fixed random sample (e.g., 2-5% of all interactions) to catch unforeseen problems.
• Risk-signal-driven sample: interactions with low model confidence, sensitive topics, or user complaints — review 100% of this subset.
• "Near-miss" sample: cases where the model hesitated or asked for more information, to calibrate confidence thresholds.

The results of this review feed back into the evaluation set, closing the loop between production and testing.

Backing the ROI calculation

With these pieces in place, the ROI calculation stops being an estimate and becomes a dashboard reading: real cost per unit of work, attributed by team, cross-referenced with acceptance and escalation rates. This also improves the internal credibility of the AI initiative — auditable numbers carry more weight than projections in a slide deck.

What to do on Monday

1. 1.Pick one use case and explicitly define its "unit of work" and the cost-per-delivered-unit formula.
2. 2.Add mandatory metadata (team, use case, environment) to every call through your existing AI gateway.
3. 3.Set up three-tier budget alerts for at least your two highest-spend use cases.
4. 4.Implement stratified human review sampling, starting with 2-5% random plus 100% of risk-flagged interactions.
5. 5.Put cost and quality on the same dashboard before your next AI budget review.

Further reading

Executive track:

• Enterprise AI ROI: measuring what is not on the invoice — the business view of this same topic.