How to Measure AI Productivity Without Falling for Vanity Metrics

Your team adopted AI coding tools months ago. Seats are filled. Prompt volume is up and to the right. Leadership is happy.

Now answer a harder question. Is any of that activity actually making the team faster, or are you just paying more to do the same work?

Most teams cannot answer that. They measure how much AI gets used. They do not measure whether it works. That gap is the single biggest blind spot in AI adoption today, and it is costing companies real money.

Activity is not productivity

When leaders try to measure AI, they reach for the numbers that are easy to pull. Number of seats. Number of prompts fired. Tokens consumed. Tools adopted.

These are vanity metrics. They go up whether AI is helping or not. A developer who fires 200 prompts to fix one bug looks more "productive" than one who fixed it in two. Volume tells you AI is being used. It says nothing about output.

Real AI productivity is about a different set of questions. Is the work getting done faster? Is the quality holding? Is the spend justified by what comes out the other side? Activity metrics cannot see any of that.

Why DORA metrics break on AI work

A common instinct is to fall back on DORA metrics or other legacy engineering measures. The problem is that those frameworks were built for human-paced, deterministic work. They were not designed for autonomous and semi-autonomous AI systems.

Applying old delivery metrics to AI-assisted work masks what is actually happening underneath. A pipeline can look healthy on the surface while large amounts of compute get burned on inefficient reasoning, retries, and oversized model calls. The dashboard stays green. The waste stays hidden.

The shift that matters is from measuring volume to measuring efficiency. Not how much the system generated, but how efficiently it reached a useful result.

The hidden cost: inference waste

Here is where the money leaks. Most AI spend goes toward inference that did not need to happen, or that ran on a model far larger than the task required.

Three patterns drive it:

• A heavy model gets used for a task a smaller, cheaper one would handle just as well.
• Prompts get retried and reworked because the first attempt missed.
• Company AI tools quietly get used for personal work that has nothing to do with shipping product.

None of this shows up in seat counts. All of it shows up in the bill. When leaders cannot see efficiency per task, they cannot tell the difference between AI that earns its cost and AI that quietly drains it.

What real AI productivity measurement looks like

Measuring AI properly comes down to three questions a leader should be able to answer at any time.

First, is AI actually improving output? That means looking at real signals like work completed and time saved, tied to prompts and tokens, not raw usage counts.

Second, is company AI being used for company work? Personal use of paid AI tools is common and almost never tracked. It inflates cost and distorts every other metric.

Third, is each task running on the right model? The cheapest model that does the job well is the right one. Defaulting everything to the biggest model is how budgets balloon.

Answer those three and you move from "AI is being used" to "AI is working, and here is the proof."

The manual way most teams try this today

Plenty of teams know they should measure this. So they try to build it themselves.

That usually means wiring up APIs across every AI tool in use, pulling usage and cost data into one place, cleaning it, building pipelines to keep it fresh, and standing up dashboards. Then the tools change, a vendor updates an endpoint, the dashboard breaks, and someone spends another week fixing it instead of doing their actual job.

It is a grind that never really ends, and it pulls engineering time away from the work that matters. Most teams either give up partway or keep limping along with numbers they do not trust.

Where Tommbo fits

Tommbo is an AI Usage Intelligence platform that gives leaders a real view of how AI gets used across the organization, without the manual plumbing.

It surfaces real AI productivity from prompts, output, tokens, and time. It flags when company AI tools are being used for personal work. And it points to the right model per task so spend stays tied to value. The output is real-time dashboards, per-employee scorecards, and alerts, across engineering, support, ops, and internal copilots.

The point is simple. Stop guessing whether AI is worth it. See it.

Frequently asked questions

What is the difference between AI activity and AI productivity? Activity measures how much AI is used, such as seats, prompts, and tokens. Productivity measures whether that usage produces real output. Activity can rise while productivity stays flat, which is why activity alone misleads.

Why do DORA metrics fall short for AI? DORA metrics were built for human-paced delivery. They do not account for inference efficiency, model selection, or compute waste in autonomous AI systems, so they can show healthy delivery while masking cost and inefficiency.

What is inference waste? Inference waste is compute spent on AI work that was unnecessary or oversized, such as using a large model for a simple task or repeated retries. It rarely appears in usage counts but shows up directly in cost.

How do you measure AI ROI on an engineering team? Tie AI usage to real output and time saved, confirm the AI is being used for company work, and check that each task runs on a cost-appropriate model. Those three together give a defensible read on return.

Want to see what your team's AI usage actually looks like? Take a look at Tommbo.