How to read a back-test without being fooled

Almost every strategy you will ever see online comes with a chart that goes up and to the right. The chart is usually accurate about the past. The question is whether it says anything about the future. These eight questions help you decide, and we ask them of our own results too.

1. When do trades happen compared with the signal?

A strategy that sees a signal at the close and trades at that same close is using information it could not have acted on. We tested a popular community strategy that returned roughly 490% a year when trades happened at the same close as the signal. With a one-day delay it returned about 160% a year, with a worst drawdown of about −63%. Same rules, same data, very different story. Always ask: if I could only trade the next day, what happens?

2. Are costs included?

Trading is not free. Spreads, slippage and fees take a bite every time the portfolio changes. In our tests, a daily-rebalanced Keel gave back much more to costs than a weekly one. A result with zero costs is a best case, not an expectation. Ask what cost per trade was assumed, and how the result changes if it doubles.

3. How many versions were tried?

Try enough variations and one will look brilliant by luck. Ask how many settings were tuned and how many versions were tested before this one was chosen. When we searched for the best group of core ETFs on the first half of our history, how well that ranking held up on the second half was close to nothing (a correlation of about 0.19). The winner of one half told us almost nothing about the other.

4. Was any data held back?

A fair test designs the rules on one part of the history and judges them on a part the designer never touched. If a result only exists on the data used to build it, treat it as a description of the past, not a test.

5. Does it still work if the settings move a little?

If a 100-day average works but 90 or 110 breaks it, the result is luck. A good rule works across a neighborhood of settings, which we call a plateau. A result that works at exactly one setting is a cliff. We look for plateaus and drop cliffs.

6. Does one event explain it?

Much of the gain in our own Keels against the S&P 500 comes from stepping aside in the 2020 crash. If you remove a few weeks and the advantage disappears, you are looking at one event, not a method. Ask what the result looks like with the single best period taken out.

7. Does the day it trades matter?

For a weekly Keel, the five weekdays gave results that differed by a lot. Monday and Tuesday looked best, partly by luck. A result that depends on trading on a lucky day is weaker than it looks. We treat the five-day average as the honest number.

8. What did it put you through?

Annual return is only half the picture. Ask for the worst drawdown, how long it took to recover, and a measure that captures both, such as UPI. A strategy that earns 20% a year with a −45% drawdown is a different thing from one that earns 20% with −12%.

A short checklist

Our own results

We apply these tests to our own Keels, and we publish the limits next to the numbers. Our simulated results are prepared with hindsight and partly fitted to past data, they lean on the 2020 crash, and they can trail the market in sharp rebounds. That is not a reason to ignore them. It is a reason to read them the way you now know how.

What we learned from testing more than 40 indicators · Join the waitlist

Illustrative guide using figures from our research. Not investment advice. Hypothetical results are prepared with hindsight. Past performance, real or simulated, does not predict future results. You can lose money.

Join the waitlist