Stop Measuring AI Adoption by Who Has Copilot Open

I spent most of last quarter staring at an AI adoption dashboard that told me nothing useful. 72% weekly active across the org. Looked great in a slide. Got a few nods in the leadership review. Then someone asked “so are we actually getting faster?” and nobody had an answer.

That 72% meant that 72% of our engineers had opened Copilot or Cursor at least once that week. Some of them were using Composer to scaffold entire test suites from requirements docs. Others just let tab-complete fill in a closing bracket now and then. Both counted the same. Both were “active users.” We’d been measuring the wrong thing for months and congratulating ourselves for it.

Weekly active is a vanity metric

I know that’s a strong statement. Weekly active made sense early on, when we were figuring out if people would even install the tools. But most orgs are past that phase now. The question isn’t “are people using AI” anymore. It’s “are people using AI well, and is it making a difference.” Weekly active can’t answer that. It’s binary. Engineer opened the tool, or didn’t. No depth to it at all.

We have seven streams and about 600 engineers. The weekly active numbers ranged from 60% to 84%. Motors and Platform were above 80, Real Estate sat at 60. Obvious conclusion: help Real Estate catch up.

Except when I talked to engineers in the high-adoption streams, a lot of them were doing the bare minimum. Copilot was on, sure. They’d accept the occasional autocomplete. But multi-file context? Chat-driven refactoring? Using AI to generate tests or docs? Almost nobody. The tools were running but nobody was really using them. So I had streams at 84% “adoption” where actual usage depth was embarrassingly shallow, and no metric that could show me that.

The metric we switched to

We started measuring the percentage of lines in commits that are AI-generated or AI-assisted.

This metric has problems. I want to be upfront about that. There are edge cases around what counts as “AI-assisted” when you heavily edit the output. The telemetry isn’t perfect. We’re still working through attribution logic with Copilot and Cursor’s data. But even a rough version of this metric is miles ahead of weekly active, because it captures something real: how much of your actual engineering output is shaped by AI tools. Not “did you open the tool” but “did the tool show up in what you shipped.”

Once you have that number, you can ask the questions that matter. Which teams are high? Which are low? Is there a difference between mobile and back-end? Between senior and junior engineers? Between streams that got early enablement and streams that didn’t? You can suddenly see things. That’s what was missing before.

We stopped using proficiency levels

Most frameworks for this use proficiency levels. L1 through L4, with behavioural descriptions at each level. “At L2, the engineer demonstrates context-aware prompting and leverages multi-file editing.” That sort of thing. We tried it. It was a pain.

Managers didn’t know how to assess it consistently. Two managers would look at the same engineer and place them at different levels. The descriptions were too open to interpretation, so calibrations turned into debates about what “demonstrates” means instead of conversations about actual performance. So we flipped it. Instead of levels defined by behaviours, we built segments defined by a metric, with behaviours as supporting evidence.

Basic Adoption is 25 to 50% AI-generated lines. Engineers who use the tools daily for completions and simple generation. They review what the AI gives them before accepting. They show up in the dashboards consistently. We set this as the floor. Nobody should be below Basic.

Intermediate Adopter is 50 to 75%. These engineers have moved past autocomplete. They write prompts with real context, use multi-file features, generate tests and docs with AI. They iterate on outputs instead of taking the first suggestion. They can tell you why the AI’s approach works or doesn’t for a given situation.

Power Adopter is 75 to 90%. This is where it gets interesting. Agentic workflows. Multi-step tasks delegated to the AI. The engineer’s job shifts from writing code to scoping the problem, validating assumptions, and verifying the AI’s work before shipping. Still fully accountable for everything, but day-to-day work looks different.

AI Native is 90% and above. These engineers make everyone around them better. They build custom workflows, run workshops, push AI into CI/CD, shape how the team thinks about the tools. We don’t expect many people here. Staff+ and AI Champions, mostly.

The names are deliberately non-hierarchical. “AI Native” sounds better than “L4” and doesn’t carry the same weight of failure if you’re at a lower level. Basic Adoption isn’t a failing grade. It’s where most engineers start, and it’s fine for juniors to stay there a while.

What changes when you measure this way

Our “leader” streams suddenly looked less impressive. Motors was at 84% weekly active but the actual AI-generated line percentage was middling. Having Copilot open all day doesn’t mean you’re using it for anything substantial.

Real Estate, our worst stream on weekly active, actually had a few individual engineers well into Power Adopter territory. They just happened to be surrounded by non-adopters dragging the average down. That’s a completely different problem than “the whole stream is behind” and needs a different fix.

We could also compare across work types for the first time. Mobile engineers, front-end folks, back-end service teams, data engineers. Do they adopt AI at the same rate? Does it help them equally? We couldn’t answer any of that with weekly active. Early data suggests back-end service work sees the highest AI-generated percentages and mobile is lower. We’re still figuring out if that’s a tooling gap or something about the nature of the work.

And we finally had something to tie to performance expectations. “You should be at Intermediate by Q2” is specific. A manager can look at the data and see where someone stands without making a subjective call about whether behaviour X qualifies as “context-aware prompting.” Qualitative assessment still matters in calibrations, but the number does the heavy lifting.

People will game it

Yeah, probably. Someone will regenerate code a few times to bump their percentage. Someone will accept every suggestion without reading it. But people game every metric. Story points. Lines of code. Coverage percentages. If you wait for one that can’t be gamed, you’ll wait forever and measure nothing.

We handle it by not making the metric the only thing that matters. It’s the primary quantitative signal. The calibration process also looks at code review quality, peer feedback, whether you can explain your approach. If someone’s at 85% AI-generated lines but their reviews are a disaster and they can’t explain why they chose a given approach, that’s going to surface.

There’s a cultural piece too. We keep hammering the message that you own every line you ship, AI-generated or not. If you blindly accept suggestions and one causes an incident, that’s on you. Not on the AI. Not on your manager for pushing adoption. On you. That makes people think twice about inflating numbers at the expense of understanding their code.

Where to start if you want to try this

Don’t overthink the setup. If you’re running Copilot or Cursor, you already have data. Both tools emit telemetry you can correlate with commits. Get someone on your platform or devex team to build a dashboard showing % AI-generated lines per engineer, per team, per stream. Doesn’t have to be pretty. Just has to exist.

Then look at it for a few weeks. Don’t set targets yet. Don’t announce anything. Just look at the distribution. You’ll find surprises. Some of your best engineers will be lower than expected. Some quiet junior devs will be high. The distribution tells you a story about your org you can’t get any other way.

Once you understand the baseline, pick your segment boundaries. Ours are 25/50/75/90 but yours might be different. If most of your org is below 30%, a “Basic” floor of 25% makes sense. If you’re already averaging 60%, set more aggressive thresholds. Base it on your data, not on what you read in a blog post.

Map segments to seniority expectations with a timeline that isn’t insane. We gave juniors a quarter to reach Basic, mid-levels a quarter to reach Intermediate. Seniors get more time for Power Adopter because the agentic tooling they need is still rolling out. Nobody goes from zero to AI Native in three months.

One thing I’d really recommend: run a dry cycle before it counts. Assess everyone against the framework during one performance review, surface the gaps, have the conversations, but don’t attach consequences. Let people see where they stand. The anxiety this removed from our hard rollout was significant. People who were behind got a heads-up. Managers who weren’t sure how to assess it got practice. Worth the wait.

Coming up next

I’m writing more posts on this topic. Still working through the outline but roughly:

• Why “you own every line you ship” and “writing code is the easy part” turned out to be the most important things we said during the rollout
• How to plug AI proficiency into performance reviews without breaking your calibration process
• What we’re learning about AI adoption differences between mobile, front-end, back-end, and data engineering teams
• Our tooling strategy, and why you don’t need the expensive tools to get most of the value
• How the rollout went across 600+ engineers, including the parts that didn’t work
• Everything that went wrong or almost went wrong

Drop me a line if you’re working on something similar. I want to know how other orgs handle the measurement piece.