Skip to content
AmiBroker TA Course

Backtesting Mistakes

A backtest is a machine for producing convincing numbers. Point it at a rule, a universe and a date range, and it will hand you an equity curve, an annual return, a drawdown figure and a trade list, all rendered with the same authority whether the rule was sound or the formula was reading tomorrow’s prices. The engine has no opinion. It executes what you wrote, under the settings you left in the dialog, on whatever symbols happened to be in the database, and it reports the result without a word about whether any of it could have happened.

That is the whole problem this part exists to solve. Parts 28 and 29 taught you to run a portfolio backtest and to read the report AmiBroker produces. This part teaches you to distrust it — specifically, precisely, and with a procedure rather than a mood.

What makes this part different from a list of warnings

Section titled “What makes this part different from a list of warnings”

You have almost certainly read a list of backtesting biases before. Look-ahead bias, survivorship bias, overfitting, unrealistic fills: the names are common currency, and knowing them changes nothing, because the names are not where the damage happens. The damage happens in a specific line of AFL, or a specific checkbox, that nobody looked at.

So every lesson here works the same way. It names the error, then shows you the mechanism: the exact expression that reads a bar that had not printed, the exact combination of trade price and delay that fills a decision at a price from earlier that morning, the exact sizing line that compounds into a position larger than everything that traded on the day. Where a mechanism can be measured, there is a formula that measures it on your own data.

Six families, and why they are ordered this way

Section titled “Six families, and why they are ordered this way”

How a backtest result gets inflated

  1. The formula knows things it could not have knownLook-ahead bias — future bars, end-of-period aggregates, same-bar fills
  2. The universe was chosen after the factSurvivorship and selection bias — the losers were never in the database
  3. The rule was chosen after seeing the answerData snooping and overfitting — many attempts, one reported result
  4. The fills were never availableUnrealistic prices, absent costs, positions the market could not absorb
  5. The sizing compounds the fictionConcentration, margin, and equity curves no account could have followed
  6. The evidence was never enoughToo few trades, one regime, conclusions the sample cannot carry
Each stage inflates the result produced by the stage before it. They compound.

The order is deliberate. Look-ahead comes first because it is the only error on the list that can make a rule with no merit at all look flawless, and because it hides inside constructions that look completely ordinary. Survivorship comes second because it is invisible in the formula entirely — nothing in your AFL is wrong; the database is. Overfitting comes third because it is the one you commit while doing everything else correctly. The last three are the arithmetic of execution: prices, size, and sample.

They compound, and they compound multiplicatively rather than additively. A rule that leaks a small amount of future information, on a universe with the failures removed, tuned over forty parameter combinations, filled at prices nobody could have got, sized into positions nobody could have filled, over one bull market, does not produce a result that is somewhat too good. It produces a result that has no relationship to anything.

What you need before starting

Section titled “What you need before starting”

Part 28 for the backtester itself: signals, trade prices, delays, SetPositionSize(), ApplyStop() and portfolio mode. Part 29 for the report vocabulary — this part uses AmiBroker’s own metric names throughout, so Annual Return % rather than CAGR, Max. system % drawdown rather than max DD, and Sharpe Ratio of trades rather than Sharpe.

You will also want Part 14’s multi-timeframe material, because two of the most common look-ahead leaks live in TimeFrameExpand() and TimeFrameGetPrice(), and Part 2’s data-quality work, because a padded or gap-filled series produces fills that were never available regardless of how careful your formula is.

Nothing here needs the Professional edition or a real-time feed. Every formula runs on end-of-day data, and the two challenges are designed to be completed with a free end-of-day database.

The two challenges

Section titled “The two challenges”

The part closes with two exercises that reverse the usual direction of teaching. Instead of showing you a correct system and explaining why it is correct, they hand you broken ones and ask you to find out why.

The first is a single spectacular result — an illustrative report claiming an annual return most people would not question closely enough, alongside a drawdown that should have been the giveaway. You get the formula, the settings and the evidence, and a task before any hint appears. The hints are staged, the full solution sits behind clearly separated headings further down the page, and the root causes are ranked by how much of the result each one was responsible for.

The second is a collection of six independent faults in one system: a future data leak, missing costs, survivorship, repeated signals, a sizing error and a multi-timeframe misalignment. Each comes with symptoms, evidence you can gather, a task, staged hints, a full solution and the root cause. They ship as one AFL file with a fault switch, so you can turn each defect on and off and watch exactly which report metrics move.

What changes by the end

Section titled “What changes by the end”

You should finish this part able to take any backtest — yours or somebody else’s — and run a fixed audit over it: what did the formula know and when; what was in the universe and how was it chosen; how many things were tried before this one; what price was assumed and could it have been obtained; how large was the position relative to the market; and how many genuinely independent observations support the conclusion. You should also be comfortable reaching the answer “this evidence does not support a conclusion”, writing that down, and moving on.

Part30 of 36Level4 — Trading System ResearcherPages8Estimated time4.7 hours

0 / 8 lessons in this part completed