How I run a test

Method

This is how an idea becomes a result. It is about the process, not about any one signal. The ideas that failed are written up in full on theLedger. The one that passed is not, because I am still testing it.

Everything above runs by itself on one machine. The step that matters is the fourth one, and the only thing that makes it work is that the pass mark was written down before the data reached it.

1. The data

Two sets of data, both built rather than bought. One records the full order book for a fixed list of stocks every trading day and grows every session. The other is about ten years of daily prices, rebuilt from the exchange's own published files.

That second one matters more than it sounds. Building it up from daily exchange files means companies that were delisted, merged or went bust are still in there. Those are exactly the names missing from an easy dataset, and leaving them out quietly makes every backtest look better than it was. Every split and bonus is handled through a table I checked by hand, with one rule I test for: a real crash has to survive the adjustment. If my cleaning smooths out a genuine collapse, I have not cleaned the data, I have damaged it.

2. Write the test down first

The idea goes into a dated commit before I run anything. It says what I am measuring, over what time horizon, what pass mark it has to beat, what account size the costs are based on, how consistent it has to be, and what would make the test invalid.

That last part matters. A test is invalid if the signal fires so rarely that the result means nothing, or so often that it is not really picking anything. Two ideas here ended that way without ever getting a result, which is correct. You cannot get there if you write the rules afterwards.

3. Work out the real trading cost

Every pass mark is written as a multiple of what one round trip actually costs at a stated account size: brokerage, government charges, and the real gap between buy and sell prices taken from my own recorded data rather than a round number. The same cost model runs in the study and in the simulated trading, so the two cannot quietly disagree.

This is where most home-built research fools itself. If an effect is real but smaller than what it costs to trade it, there is nothing there at all. One effect on my list came out positive on every single day I tested and is still useless to me. Depending on account size it is somewhere between a twentieth and a third of what one round trip costs.

4. Run it once

The idea is scored once against the pass mark I wrote down. I keep some days aside without looking at them, and I do not take a quick peek even when it would be useful. Once you have looked at data it is no longer fresh, and it never becomes fresh again.

The result is failed, thrown out, or passed with caveats. I record it with the date and I do not go back to it. If I set a threshold badly enough to deserve a retry, I get exactly one, I say so in advance, and the second result is final for that whole family of ideas.

5. Then test it forward

An idea that passes does not get money. It gets a written test plan, published before the first day of the test, and it has to survive on data that did not exist when I set the pass marks. The current plan and its dates are on the Protocol page.

What this has produced so far

23 ideas tested and 22 of them finished, one that passed and is still waiting for its first real result, and no strategies built on anything that had not passed. Whether that is a good outcome depends on what you think the alternative would have produced.

The full list is on the Ledger. The counts on this page are calculated from it, so they cannot drift apart.