I tested five ideas and closed every one. That was the result

Earlier this year I shut down a trading research project.

Five separate ideas went through it. Four were tested against pass marks I’d fixed in advance. The fifth stopped earlier, when a simple check showed the idea couldn’t work the way the market is set up. None of them produced something worth trading. I wrote down the results and closed the project.

It was some of the most useful research I’ve done, because when it came time to stop, the numbers made the decision, not my mood.

A lot of AI pilots don’t have that.

The idea sounds academic, but in practice it’s simple. Before you see any results, write down what you expect to happen, the number that would show it happened, the number that would show it didn’t, and what you’ll do either way. Then run the test and compare the result with what you wrote.

One of my tests was checking whether a market kept leaning in one direction. To pass, the lowest believable win rate had to be above 55 per cent. After 272 completed trials the win rate was 49.3 per cent, and the lowest believable figure was 43.38 per cent. It didn’t pass, so I closed it.

Without that pass mark, the same numbers would have tempted me to tell myself a story. Maybe one more week would turn it around. Maybe one slice of the data looked better. Maybe it just needed tuning. Having the number fixed in advance meant none of that mattered. I’d asked the question and got the answer.

Another one found a real pattern in the historical data and still got closed. The pattern was too small to cover the cost of actually trading it. So the pattern was real, but you couldn’t make money from it. A chart can be interesting and still useless for business. Because the test measured the result after costs, I could see that.

Now think about a three-month AI pilot.

At the review, someone says the tool “showed promise”. Compared to what? Staff say it feels faster. One department loves it. There’s a slide showing hours saved, worked out from a survey of the same people who chose the tool. Nobody is lying, but the pilot couldn’t fail, because nobody ever said what failing would look like.

So it gets extended. More licences get bought. The team keeps gathering evidence past the original decision date, and the bar moves to wherever the latest numbers landed.

Fixing this takes one meeting before the pilot starts. In it, write down four things:

The one business number you already track that has to move.
How much it has to move to be worth the cost.
By when.
What the company will do if it doesn’t.

The fourth one is what makes it real. If nothing happens when the pilot fails, you haven’t run an experiment. You’ve already made the purchase.

The same thinking helps in vendor demos. Ask the person selling the tool: what result would convince you this doesn’t work for our company?

A good answer tells you when the tool is a bad fit, what data it needs, and at what point carrying on would waste money. If no result could ever count against the product, you’re being asked to believe, not shown evidence.

It helps inside the company too. Stopping a project gets political when a senior person backed it and careers are tied to it. If everyone agreed the stopping point in advance, stopping isn’t anyone’s personal failure. It’s just the plan doing what it said.

My research project left me with a clear set of results, research tools I can use again, and five ideas I no longer had to spend money on. Reaching those answers cost $1.06 of real money. That’s better than holding on to a maybe and building on it for another year.

Most companies planning an AI pilot are one meeting away from knowing how they’ll judge it. That meeting happens before the spending starts, and what comes out of it is a short page naming the result, the target, the deadline and the decision if it misses.

If your company is planning an AI pilot and nobody has written down what would make you stop it, that’s the first gap I’d close. It’s also part of the AI Opportunity Audit: before deciding what’s worth doing, agree how you’ll know.