AI pilots tend to fade out. The demo goes well, the first few weeks of real data are messier than expected, someone suggests another round of prompt changes, and six months later the pilot is still "nearly there" and still holding its budget.
In August 2025, MIT's NANDA initiative reported that most enterprise generative AI pilots delivered little or no measurable impact on profit and loss. In June of the same year, Gartner predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing "escalating costs, unclear business value, and inadequate risk controls."
Stopping at week six frees the budget months earlier than stopping at month nine.
Decide what failure looks like on day one
Before a pilot starts, write down two sets of conditions and get the budget owner to sign them.
The first set ends the pilot. Make each one a number and a date. By week six, for example, the agent still handles fewer than half of real cases without a person stepping in, or the cost per case, review time included, is still above what the work costs today.
The second set moves it forward. By week ten, the agent handles most real cases unaided and the cost per case, review included, is below today's. Hitting those gets a production budget.
Results between the two are a judgement call for the budget owner.
Measure on real cases from the start
Pilots that run on hand-picked examples usually look good. Feed the agent a sample of real cases from last month, including the messy ones, and keep a set the agent never sees during tuning so you can check it at the end.
For every case, record whether the agent's output was used as it was or needed rework, how long the review took, and what the case cost to run. Those numbers are what the criteria get checked against, so collect them from the first week.
Give the pilot a hard end date
Put the decision meeting in the calendar when the pilot starts. On that date, the team brings the numbers and the budget owner makes one of three calls: stop, fund the next stage, or extend once, and only once, with a named reason and a new date.
Write down what you learned either way
A stopped pilot still produces a record of what the agent got wrong and which cases were hard. Write that up on a page and keep it where the next team trying something similar will find it.
And if the pilot passes, the same numbers become the production targets. The agent that handled most cases unaided in the pilot should keep doing so live, so keep the week-one measurement running after launch.
Frequently asked questions
What are kill criteria for an AI pilot?
Written conditions, agreed before the pilot starts, that end it. For example, the agent fails to handle at least a set share of real cases unaided by week six, or the cost per case stays above the manual cost. They sit next to the criteria for moving to production.
Why do AI pilots stall instead of ending?
Nobody agreed what failure looked like, so there's always one more tweak to try. MIT's NANDA initiative reported in 2025 that most enterprise generative AI pilots delivered little or no measurable P&L impact, and Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027.
Who should set the kill criteria?
The person who owns the budget for the next stage, with the team running the pilot. If the sponsor didn't agree the numbers up front, they'll argue with the result at the end.
Is killing a pilot a failure?
No. The expensive outcome is a pilot that runs for a year and holds budget that could have funded something else.