Token costs are easy to measure. A dashboard tells you how much you spent this week, which models were used, and what each task cost. The cost of a bad change is harder to see.
It might surface three weeks later as a rollback, an incident, extra review work, lost engineering time, or a production outage. And that cost is rarely traced back to the AI coding decision that produced the change in the first place.
This isn't an argument against optimizing AI coding costs. It's an argument for measuring the other side of the equation too.
The cost everyone can already see
- Tokens per task
- Model selection
- Cost per request
- Routing simple tasks to cheaper models
- Cost per generated change
These are legitimate engineering metrics, and they're also easy to instrument, which is most of why they get instrumented. Anything that arrives on an invoice gets a dashboard. Anything that arrives as a postmortem three weeks later usually doesn't.
The cost that arrives later
Suppose routing a class of tasks to a cheaper model saves money. That's a real result. But what happened to the resulting code?
- Did it require more review?
- Did it get reverted?
- Did it introduce an incident?
- Did it require additional debugging?
- Did it create verification debt?
Why cost optimization can't answer the whole question
A cost optimization system is doing its job if it reduces API spend. It doesn't need to know whether the resulting code was safe to merge, and it would be a strange product if it tried.
That's a different question at a different point in the pipeline. Cost optimization asks how cheaply we generated the change. Production risk asks what could happen if we ship it. Both are legitimate; only one of them currently gets a dashboard in most organizations.
What to measure alongside AI coding cost
Revert rate for agent-authored PRs
Track agent-authored and human-authored changes separately. Not because one is automatically worse, but because if you don't split them you can't see whether change source correlates with different outcomes at all.
Incidents traced to agent runs
Don't stop at “an AI wrote part of this.” Trace incidents to the specific change and, where possible, the agent run that produced it. The agentic postmortem template is built around those fields.
Review time per merged agent PR
Compare the tokens saved against the human time spent reviewing the result. A cheaper generation that costs forty extra minutes of senior review hasn't saved anything.
Caught-after that could have been caught-before
When an incident reveals something a pre-merge check could have found, record it. Over time that list becomes the specification for what your pre-merge checks should actually be looking for.
What cost dashboards measure vs what they miss
| Cost optimization tracks | It doesn't answer |
|---|---|
| Tokens per request | Was the output correct? |
| Cost per model | Did model choice change production risk? |
| Cost per task | How much review did the change require? |
| Cost per bug | Should the change have merged? |
| Spend over time | Which agent changes caused incidents? |
The left column isn't wrong. It's just not the whole ledger.
Does a cheaper model produce riskier code?
Not necessarily, and we're not going to claim a direct relationship between model price and production risk that we can't support. Treat it as an open question.
The more concrete problem is measurement. If teams don't connect model choice, code changes, review effort, and production incidents, they can't know whether their cost optimization is reducing total engineering cost or just moving it somewhere that isn't itemized. That's a tractable gap, and closing it doesn't require settling the harder question first.
Cheaper code generation doesn't automatically mean cheaper software delivery.
The missing number
AI coding makes code generation cheaper. That's useful and worth keeping.
But a useful engineering system should eventually be able to answer both halves: how much did it cost to produce this change, and how risky was it to ship? The first belongs on the cost dashboard, where it already lives. The second is where production risk analysis comes in — and where, in most pipelines, there's currently no number at all.
Put a second number next to your cost-per-change metric: how much of what shipped went out without a pre-merge risk check. You will not like the first reading, which is the point of taking it.
If review capacity rather than spend is the constraint you're feeling, verification debt is the same problem viewed from the other end, and the governance buyer's checklist covers what to ask of anything claiming to close it.
Frequently asked questions
Is optimizing AI coding costs a bad idea?
No. Routing simple tasks to cheaper models and cutting token waste are reasonable engineering decisions. The problem isn't the optimization — it's treating cost as the only number that matters when correctness has its own cost that shows up later.
Does a cheaper model actually produce riskier code?
Not inherently, and we're not aware of solid public data proving a direct correlation either way, so treat it as an open question rather than a claim. What's not in question is that most teams aren't tracking the two together closely enough to know.
How do you measure the cost of a bad AI-generated change once it ships?
Start with revert rate and incidents traced back to a specific agent run. If you don't have a way to trace an incident back to the change and the context that produced it, that trace is a separate problem worth solving first.
Is this the same problem as verification debt?
Closely related. Verification debt is what accumulates when review capacity doesn't scale with how much code an agent produces. Cost optimization that ignores correctness is one of the ways that debt gets taken on faster without anyone deciding to. See The Verification Debt Nobody Put on the Balance Sheet.
What should a post-incident review capture if an agent caused it?
The specific run, the prompt and context that produced the change, and whether a pre-merge check would have caught it. The Agentic Incident Postmortem Template is built around exactly those fields.
Keep optimizing cost. Just put a number next to it for what ships without a check. That second number is what Tomosu is built to catch, before it becomes the incident nobody budgeted for. Book a call →
