On 1 June, GitHub moved Copilot to usage-based billing. Premium requests gave way to AI credits, consumed by input, output and cached tokens at each model's published API rate. Code completion and Next Edit suggestions stay included, and everything else, agent sessions among them, draws credits.
So the question a budget owner asks this autumn, "how many seats?", needs a second number next to it: how much usage, on what kind of work, measured on your own codebase.
Benchmarks run on someone else's code
Vendor benchmarks use public repositories with clean tests and well-known libraries. Your code has internal frameworks nobody outside the company has seen, and conventions that live in reviewers' heads. Those decide whether an agent's change merges or gets rewritten.
Security is the other gap. A Cloud Security Alliance research note from April pulls together studies in which 45% to 70% of AI-generated code samples failed security tests, with broken authorization among the most common flaws. So the trial needs a security scan on every change.
A two-week trial on closed tickets
Pick 20 to 30 tickets your team closed last quarter. Mix the kinds of work you do: small bug fixes, a few features that touch more than one module, some test-writing, a refactor or two. You already know what a good solution looks like for each, because one shipped.
Reset a branch to the commit before each fix and give the agent the ticket as it was written. Let two or three developers run it, ideally people who'd be sceptical of the result.
For each ticket, record:
- whether the agent's change would have merged as it was or needed rework
- how long review took compared with the original change
- what the security scan and a senior reviewer found
- what the ticket cost in usage
With usage billing, a ticket that took the agent forty minutes of back-and-forth costs more than one it solved in a single pass, and your rollout budget depends on the mix.
Read the results by kind of work
Break the results down by the kind of ticket. A single success rate hides which kinds of work the agent handles on your code and which it doesn't.
Roll out first on the ticket types with a high merge rate. For the rest, look at why the changes needed rework. Missing context, like a convention nobody wrote down, can often be fixed with a short document the agent reads.
Keep review where it was
Whatever the trial shows, the change goes through the same review as a person's change, and through security scanning before merge. Look hardest at authorization logic. A missing role check doesn't show up in a diff, so ask where the server checks the role.
Before the renewal
If a coding agent contract or seat renewal is on next year's budget, run the trial before you sign.
Bring the per-ticket costs to the negotiation, multiplied out over the tickets your team closes in a quarter, so the seat quote sits next to a real usage estimate.
Frequently asked questions
How should we evaluate AI coding agents?
Run them on your own repositories against tickets your team already closed, so you know what a good result looks like. Measure how often the agent's change merges without rework, how long review takes, what the security scan and a senior reviewer found, and what each ticket cost in usage.
Why not just compare vendor benchmarks?
Benchmarks run on public code with clean tests. Your codebase has internal libraries nobody outside has seen and conventions that live in reviewers' heads, and those decide whether an agent's output is usable.
What does a coding agent cost per developer?
It depends on usage. GitHub moved Copilot to usage-based AI credits on 1 June 2026, metered by tokens at each model's API rate, so heavy agent use costs more than the seat price suggests. Measure spend per ticket during the trial.
Do coding agents write secure code?
Not reliably. A Cloud Security Alliance research note from April 2026 summarises studies in which 45% to 70% of AI-generated code samples failed security tests, with authorization flaws among the most common. Keep human review and security scanning in place.