Someone who implements AI for companies put it cleanly on my podcast: you don't want the person on your team who uses no AI, and you don't want someone who is AI-maxing just to run the tokens up. You want people who use those tools to maximize effectiveness and drive results. And we don't know how to measure that yet.
That last line is the part most board decks skip.
Activity is easy. Results are the job.
It's been my experience that the first AI reports look impressive. Logins up. Prompts up. Tokens up. Then the CFO asks what moved in the P&L, the warehouse or the project schedule — and the room gets quiet.
Using the tool is not the same as getting something useful out of it. One is attendance. The other is work.
The tension: adoption vs. effectiveness
On the one hand, leaders need people to try the tools or nothing changes. On the other, celebrating raw usage creates the wrong heroes: the person who pastes everything into a chat window and calls it transformation, and the person who refuses to touch it and calls it principle.
I kind of land in the middle. Require a fair try on named workflows. Reward the outcomes those workflows produce. Don't score people on how loud their usage graph is.
What "driving results" looks like
- A holdout. Never opens the tool. Fine for a week. A problem after you've paved a clear path.
- A token-maxer. Runs volume for the badge. Output quality doesn't improve. Reviews get noisier.
- A results driver. Uses AI to shrink cycle time, cut rework or raise first-pass quality on a job that matters — and can show the before and after.
An example
Say a manufacturing firm rolls out AI for change-order summaries and vendor email triage. Team A logs heavy use and still ships change packages late. Team B uses the tool less often, but average turnaround drops from two days to four hours and rework tickets fall. If you only watch usage, you celebrate Team A. If you watch results, you study Team B and copy what they did.
Signs you're measuring the wrong thing
- Dashboards lead with seats, prompts or tokens.
- Nobody can name the business metric each AI workflow is supposed to move.
- The loudest AI user is also creating the most cleanup for peers.
- Performance reviews mention "uses AI" with no tie to quality, speed or cost.
What to do this week
Pick one workflow where AI is already in play — claims drafting, bid research, exception triage, meeting notes into action lists. Write the outcome you care about in one line (hours to close, first-pass acceptance, customer response time). For the next ten working days, track that outcome next to usage. Ignore the vanity graph until you can see whether the outcome moved.
Common mistakes
- Equating adoption with value. Opening the app is not finishing the job.
- Punishing holdouts without a paved path. Mandate without a clear workflow creates quiet resistance.
- Rewarding volume. Token maxing is not effectiveness.
- Waiting for a perfect company-wide scorecard. One workflow, one metric, ten days. Start there.
Who should own this
The function leader owns the outcome metric for each AI-assisted workflow in their area. HR and finance should not invent a universal "AI score" that ignores the work. The CEO asks one question in reviews: what result moved, and how do we know?
Measure the work, not the wattage.
Start by measuring one outcome on one workflow for ten days. When you can tell a results driver from a token-maxer on that path, you're ready to talk about the rest of the org.