Skip to content
AmiBroker TA Course

Out-of-Sample and Walk-Forward Testing

The previous part left you with a parameter region rather than a parameter, and with a result measured on the same data that chose it. That result is not evidence. It is the value of the best draw from a search, and the more thoroughly you searched, the higher it climbs regardless of whether anything real is there.

This part is about the two repairs the industry has settled on. The first is a holdout: set data aside, do not look at it, and spend it once. The second is walk-forward analysis: re-estimate the parameters on a rolling schedule using only information that existed at the time, and evaluate only what the procedure produced on data it had not yet seen.

Neither repair is magic, and both are easier to break than to run. Most of the effort here goes into the breaking.

What this part actually claims

Section titled “What this part actually claims”

Walk-forward analysis answers one narrow question well: if I had re-optimised on a schedule, using only data available at each point, what would that procedure have produced on this history, under these cost assumptions? That is worth knowing. It is not a forecast, it does not measure the future, and a good out-of-sample result does not license a claim about what a strategy will do.

Take AmiBroker’s own framing seriously, because it is unusually blunt for vendor documentation: in-sample results are always good, or should be, so they tell you nothing; what matters is out-of-sample performance; if that is poor, do not trade the system. The User’s Guide says this in almost those words. It is the right attitude, and the four lessons here try to make it operational.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • state, for any test you run, exactly which data influenced which decision — and notice when the answer is “all of it”;
  • design a holdout and describe the ways yours could already be contaminated;
  • explain the difference between anchored and rolling walk-forward in terms of the assumption each one makes about markets, not in terms of which checkbox is ticked;
  • set up a walk-forward run in AmiBroker using the actual fields on the actual tab, read the Preview list before pressing anything, and find the out-of-sample results afterwards;
  • decide in advance how much degradation you would accept, and recognise the several situations where catastrophic degradation means your test was broken rather than your idea;
  • produce a written record of a run that somebody else could reproduce and disagree with.

What you need first

Section titled “What you need first”

You need Part 28 and Part 31: the portfolio backtester, and what Optimize() does. The lab here optimises two parameters over a watch list, which will be slow if you have not thought about search-space size, and meaningless if you have not thought about costs.

Everything runs on free end-of-day data. Nothing in this part needs the Professional edition, a real-time feed or a paid subscription. The User’s Guide places no edition restriction on optimization or walk-forward; what the Professional edition buys you here is throughput — 32 threads per Analysis window against the Standard edition’s 2 — and a walk-forward run is a sequence of full optimizations, so throughput is felt.

A warning about the shape of this part

Section titled “A warning about the shape of this part”

Three of these five pages are, in effect, about ways to be wrong. That is deliberate. The mechanics of running a walk-forward test in AmiBroker occupy one lesson and about fifteen minutes of clicking. The reason walk-forward analysis is worth a part of this course is that almost everybody who runs it draws a conclusion the run does not support — usually because the holdout leaked, or because the out-of-sample segments contain nine trades between them, or because a poor result was read as a verdict on the idea when it was a verdict on the settings.

Part32 of 36Level4 — Trading System ResearcherPages5Estimated time2.9 hours

0 / 5 lessons in this part completed