Level 4 · Trading System ResearcherLessonPart 33 · page 4 of 426 min

26Minutes

3AFL functions

4Sources

StandardRequires

AFL functions taught here3

The Limits of Simulation

A Monte Carlo page adds five percentile columns, two distribution charts and a straw-broom plot to a backtest report. It adds no information about the market. Every number on it was computed from the trade list the backtest produced, which was computed from the price data you supplied, under the fill assumptions you chose, using parameters you selected while looking at the answer.

That is the argument of this lesson, and it applies with equal force to walk-forward analysis, to parameter-surface plateaus, to sensitivity sweeps and to every other technique in Parts 31 to 33. They are all downstream of the same object. If the backtest is wrong, they do not detect it — they restate it with greater apparent precision, which is worse than saying nothing.

Confident garbage

Precision and accuracy are different properties, and simulation improves only one of them. Running five thousand realizations instead of five hundred narrows the sampling noise around the distribution. It does not move the distribution towards the truth. If the trade list is biased, the distribution is a tight, well-estimated picture of a biased thing.

What a robustness tool actually sees

Your databaseWhatever survivorship, splits, bad prints and gaps it contains
Your rules and parametersChosen after looking at results on this same data
The backtestFills at prices you specified, costs you assumed, liquidity you did not check
The trade listThe only thing the simulator ever sees
Percentiles, CDFs, straw broomsPrecision applied to whatever arrived

The simulator has no access to anything above the trade list, and no way to detect a problem there.

Here is the list of defects that pass through untouched, each of them covered in Part 30:

Look-ahead bias. A rule that used tomorrow’s close produces a trade list that is simply wrong. Resampling it produces five thousand wrong trade lists with a comforting spread.
Survivorship bias. Symbols that were delisted are missing from most free databases. The bootstrap cannot draw a trade in a company that went to zero if no such trade exists.
Optimistic fills. Filling at the close of the signal bar, or exactly at a stop level inside the bar, inflates every trade in the pool by a similar amount. The distribution shifts up as a block and its shape barely changes, so nothing looks wrong.
Uncharged costs. Same effect, same invisibility.
Overfitting. Parameters selected because they produced the best result on this data produce a trade list that is a description of this data. The simulator has no notion of how many parameter combinations you tried before this one.
Too few trades. A distribution built by resampling twenty trades is dominated by which of those twenty get drawn. It will still print nine percentile rows to two decimal places.

The failures that arrive together

Resampling destroys correlation by construction. That is not a side effect; it is the mechanism. Drawing trades independently from a pool is precisely an assertion that they are independent, and the resulting distribution is the distribution you would get if they were.

Real portfolio losses do not arrive independently. Three distinct mechanisms make them cluster, and none of them is visible to the simulator:

Positions share exposure. Ten holdings selected by the same momentum rule in the same market are ten expressions of one bet. When the market turns, they turn together. The Monte Carlo procedure treats their outcomes as ten separate draws.
Conditions persist. Whatever made the last three trades fail — a range-bound market for a breakout system, a trending one for a mean-reversion system — is still in force when the next signal fires. Losses queue.
Overlapping trades get serialised. In trade-list mode this is explicit and documented: bootstrapped trades are replayed one after another, so six positions that fell together in reality become six separate small dents. The reported drawdown comes out too small.

The independence check from the first lesson exists to make the second of those measurable on your own results rather than assertable in general. Run it on the equity curve you are about to bootstrap. If the runs z-score is clearly negative and the absolute-change autocorrelation is clearly positive, then the series is not exchangeable, and the drawdown percentiles you are about to read are a floor rather than an estimate.

Regime change

A bootstrap draws from a pool. The pool is your realised trades. It follows, with no qualification at all, that no realization can contain an event that is not already in the sample.

If your test period contains no episode in which liquidity vanished and correlations went to one, then none of your five thousand simulated futures contains one either. The distribution is not merely uncertain about such an event; it assigns it probability zero. The most important number on the Monte Carlo page is therefore one that is not printed anywhere: the probability that the future is drawn from a different distribution altogether.

There is a second, quieter regime problem. Your rules were selected because they worked on the sample, and the sample had a character. A system built on data from a long, low-volatility advance has been fitted to that character as surely as if you had optimised a parameter for it — and no amount of resampling that data will reveal the dependence, because every realization inherits the same character.

Walk-forward analysis, from Part 32, addresses part of this and only part. It re-estimates parameters on data the estimation had not seen, which tests whether the fitting procedure survives contact with new data. It does not manufacture a regime the record does not contain. If your entire history is one long advance, walk-forward gives you many overlapping tests of a single market condition, and the honest conclusion is that the evidence covers one regime.

Liquidity evaporation

Every trade in the pool was filled. That is what a backtest does: it assumes the order was executed, at the price you nominated, in the size your sizing rule asked for. The bootstrap inherits the assumption without ever examining it.

In the conditions that produce the drawdowns you are trying to estimate, that assumption is at its weakest. Spreads widen exactly when volatility rises. Depth thins. Auctions gap through levels overnight, so a stop that a backtest filled at its trigger price fills far below it. Some instruments halt outright, and a position you had modelled as exitable is not exitable at any price for a period you do not control. None of this is in the price series, and none of it can be recovered by resampling the price series.

The related trap is that thin instruments produce the most attractive backtest statistics, because the largest percentage moves live there. A system that ranks by recent return is selecting, among other things, for illiquidity. Its trade list is therefore built disproportionately from the fills least likely to have been achievable, and every robustness statistic computed from it is doubly optimistic.

Perturbing fills, as a technique

There is no dialog in AmiBroker that randomises trade prices. What you can do is write the perturbation yourself and see how much of the conclusion survives it:

Fragment — not a complete formula

// Randomised fills as a sensitivity test. This is your code, not a feature:
// AmiBroker documents no built-in slippage randomiser.
SlipRangePct = 0.30;                             // worst case each way
Jitter = ( 2 * mtRandomA() - 1 ) * SlipRangePct / 100;

BuyPrice  = Open * ( 1 + abs( Jitter ) );        // always against us
SellPrice = Open * ( 1 - abs( Jitter ) );

// AmiBroker adjusts trade prices into the bar's High-Low range unless
// PriceBoundChecking is turned off, so a jitter wider than the bar cannot
// push a fill outside it. Leave the check on and understand the ceiling.
SetOption( "PriceBoundChecking", True );

Wrap that in the repeat-counter pattern from the randomization technique — an Optimize() call used purely to run the backtest many times — and you get a distribution over fill luck rather than over trade ordering. It is a genuinely different and genuinely useful experiment.

It is also still bounded by imagination. A uniform jitter of 0.3 per cent models a slightly noisy normal market. It does not model a gap, a halt, or the day your size was larger than the whole book. To model those you would have to decide how often they happen, and that decision, not the simulation, is where the answer comes from.

What robustness testing is genuinely good for

None of the above makes the exercise pointless. It makes it a specific tool with a specific range, and inside that range it is valuable.

It sizes the drawdown you have not yet seen. Your single backtest showed one worst stretch. The simulated distribution says how much worse a plausible rearrangement of the same results could have been. Read the answer as a floor, add a generous margin for the clustering the bootstrap removed, and use that number for the decision it is fit for: whether you could sit through it.

It exposes dependence on a handful of trades. If removing the possibility of drawing your three best trades collapses the distribution, the system’s record is a story about three trades. That is worth knowing before you commit capital to it, and it is very hard to see any other way.

It supports a sizing decision. The ruin laboratory’s shape — growth rising, peaking and falling as the risked fraction increases, with the ruin probability climbing throughout — is robust to a great deal of misspecification. The specific numbers are synthetic; the trade-off is real.

It compares candidates as distributions rather than points. Two systems whose CAR/MaxDD figures differ by ten per cent, whose simulated distributions overlap almost completely, are not meaningfully different, and treating them as ranked is how a research programme starts chasing noise.

It kills systems. This is the use with the highest expected value and the least appeal. If the fifth percentile of drawdown is beyond what you could tolerate, the correct action is to stop, and no further work on the entry rule changes that.

What it cannot do, ever: forecast a return, establish that a system is sound, substitute for data the system has not been tested on, or convert a bad backtest into a good one.

Before you quote a Monte Carlo number

Every one of these has to be answerable, and the number does not leave your notes until they all are:

Which backtest produced the trade list, on what universe, over what date range?
What costs and what fill assumptions were charged, and where did those numbers come from?
Which sampling mode was used, and does it match whether the system holds overlapping positions?
What position-sizing method did the simulator use, and is it the one the guide recommends?
How many realizations, and does the answer move if you halve or double them?
Is the drawdown column signed positive or negative, and does the reading you are giving match the setting?
What did the independence check say about the equity series being resampled?
Which known biases of the underlying backtest have you audited, and which have you not?

If that list looks disproportionate for a single percentile, that is the correct impression. The number is cheap to produce and expensive to justify, which is exactly the sort of number that should make you cautious.

Recap

Robustness tools inherit everything. Look-ahead bias, survivorship, optimistic fills, uncharged costs, overfitting and a thin sample all pass through the simulator unchanged and come out the other side with percentile columns attached. Precision improves; accuracy does not.

Bootstrapping asserts independence, and the failures that matter are correlated: positions that share exposure, conditions that persist, and — in trade-list mode — overlapping trades that get serialised into separate smaller dents. Nothing in the procedure can contain an event absent from the sample, so regime change is assigned probability zero by construction, and liquidity evaporation is invisible because every trade in the pool was, by assumption, filled.

Used as a sensitivity analysis, resampling tells you how much of your result depended on ordering and on a handful of trades, helps you size positions, prevents you from ranking systems that are statistically indistinguishable, and occasionally tells you to stop. Used as a forecast, it is a way of being wrong with more decimal places. Part 34 takes the sizing question and makes it concrete; Part 35 puts the whole research process together with the human decision left where it belongs.

Sources for this lesson

4 verified · checked 2026-08-31

01AmiBroker User's Guide — Monte Carlo simulationamibroker.com/guide/h_montecarlo.html2026-08-31
02AFL Function Reference — SetOption§ PriceBoundCheckingamibroker.com/guide/afl/setoption.html2026-08-31
03AFL Function Reference — mtRandomamibroker.com/guide/afl/mtrandom.html2026-08-31
04AmiBroker User's Guide — Walk-forward testingamibroker.com/guide/h_walkforward.html2026-08-31

Every technical claim on this page was checked against the official AmiBroker documentation on the date shown. Where the course disagrees with folklore, the source is how you can tell which one to trust.