I tested five ideas and closed every one. That was the result

Earlier this year I shut down a trading research programme.

Five independent ideas went through the process. Four faced locked quantitative gates. A fifth stopped earlier, when a read-only probe showed that the premise was structurally impossible. None produced a tradeable result. I documented the verdicts and retired the project.

It was some of the most useful research I have done. Not because it found the answer I wanted, but because it made stopping an evidence-based decision.

That discipline is missing from too many AI pilots.

Pre-registration sounds academic, but the practical version is simple. Before you see any results, write down what you expect to happen, the number that would show it happened, the number that would mean it did not, and what you will do in either case. Then run the test and read the result against what you wrote.

One of my tests looked for a persistent directional bias in a market. Success required the lower bound of the measured win rate to clear 55 per cent. After 272 settled trials, the observed win rate was 49.3 per cent and the lower bound was 43.38 per cent. The test had not passed. I closed it.

Without the gate, those same numbers would have invited a story. Perhaps one more week would change the trend. Perhaps a particular slice looked better. Perhaps the model only needed tuning. The locked threshold made all of that irrelevant. The question had been asked and answered.

Another lane found a measurable signal in historical data. It still closed. The signal was too small to survive the cost of executing it in the real market. That distinction mattered: the pattern existed, but it was not a business. A chart can be statistically interesting and commercially useless at the same time. Because the gate measured the result after costs, the programme could tell the difference.

Now consider a three-month AI pilot.

At the review, someone says the tool “showed promise.” According to what? Staff say it feels faster. One department is enthusiastic. A slide shows hours saved, calculated from a survey of the same people who selected the tool. Nobody is being dishonest, but the pilot could not have failed because failure was never defined.

So it gets extended. More licences are bought. The team keeps collecting evidence after the original decision point, and the standard moves to wherever the latest data landed.

The fix costs one meeting held before the pilot begins. In it, write down four things:

The single existing business number that must move.
How much it must move to justify the cost.
By when.
What the company will do if it does not.

The fourth item gives the exercise teeth. An experiment with no consequence for failing is a purchase that has already been made.

The same discipline turns a vendor demo from theatre into information. Ask the person selling the tool: what result would convince you that this does not work for our company?

A useful answer names the conditions under which the tool fits badly, the data it needs, and the point at which continuing would waste money. If no possible result counts against the product, the conversation is about belief rather than evidence.

There is also an organisational benefit. Stopping an initiative can become politically expensive when a senior person sponsored it and careers are attached to the outcome. Pre-agreed kill criteria change the meaning of the decision. If the test misses the gate, stopping is not a personal defeat. It is the plan working.

My research programme produced a clear scoreboard, reusable research tools, and five paths I no longer needed to fund. The real-money cost of reaching those verdicts was $1.06. That is a better outcome than protecting an ambiguous success and building on it for another year.

Most companies planning an AI pilot are one meeting away from defining how they will know whether it worked. The meeting happens before the spending, and its output is a short sheet naming the result, threshold, deadline, and stop decision.

If your organisation is planning an AI pilot and nobody has written down what would make you stop it, that is the gap I would close first. It is also part of the AI Opportunity Audit: before deciding what is worth doing, agree how you will know.