What Monte Carlo Can and Cannot Tell You

Your backtest produced eighty-three trades in a particular order and an equity curve with a particular shape. Change nothing about the rules, the universe or the period, and ask a question the backtest cannot answer: if those same eighty-three trades had arrived in a different order, how different would the ride have been, and would you still be trading at the end of it?

That question is answerable, cheaply, and AmiBroker answers it as part of every portfolio backtest. By the end of this lesson you should be able to say exactly what is being resampled and what is not, name the statistical assumption the whole procedure depends on, measure whether your own results violate it, and read a percentile without turning it into a prediction.

One draw from a distribution you never saw

An equity curve is a sample of size one. That is the whole difficulty. You cannot compute a standard error from one observation, and the human eye is very willing to read personality into a single line — “it recovers well from drawdowns”, “it is smooth”, “the bad patch was early” — when what it is looking at is one arrangement of a set of outcomes.

Bootstrapping is a way of manufacturing more observations from the one you have. You treat your realised results as if they were the population, draw new samples from them at random, and look at how much the answers move. The spread you get back is not the spread of possible futures. It is the spread produced by re-arranging your own past, which is a much smaller and much better behaved thing — and confusing the two is the central error this part exists to prevent.

What AmiBroker actually randomises

The User’s Guide describes the procedure step by step, and it is short enough to hold in your head.

AmiBroker's bootstrap, as documented

1. 1Run the backtestProduces the original set of N trades
2. 2Draw N trades at random, with replacementSome originals are skipped, some appear twice or more. This is one "realization".
3. 3Replay them sequentiallyApply gain and loss in the drawn order, using the configured position sizing, to build an equity curve
4. 4Record the equity curveIts final value, its annual return, its deepest drawdown, its lowest point
5. 5Repeat 1,000 times or moreThen turn the recorded results into distributions

Steps B.1 to C.1 of the procedure set out on the Monte Carlo page of the User's Guide.

Two things are worth pulling out of that. The first is with replacement. A realization is not a shuffle of your trade list; it is a fresh draw of the same length from the same pool, so some trades are missing and others are repeated. Sampling with replacement is what makes the exercise statistical rather than combinatorial, and it is why the guide can point out that with N input trades there are N to the power N distinguishable realizations — enough, past about a hundred trades, that drawing the original sequence again is not something you need to worry about.

The second is that AmiBroker offers two different things to resample, and the choice matters more than any other setting on the tab:

Mode	What gets drawn	Use it when
Simulate using trade list	Individual closed trades from the backtest’s own trade list, with the position sizing you configure	The system holds one position at a time, so trades never overlap
Simulate using portfolio equity changes	Bar-by-bar percentage changes in portfolio equity, expressed as ratios — a bar that gained 10 per cent is 1.1 — multiplied together cumulatively	The system holds several positions at once, which is most portfolio systems

The reason is mechanical and the guide states it plainly: bootstrapped trades are replayed sequentially. If your original system held six positions simultaneously and they all fell together, that shared drawdown happened once, all at the same time. Replayed one after another, the same six losses become six small separate dents instead of one large hole, and the reported drawdown comes out smaller than the drawdown you actually lived through. A portfolio system bootstrapped in trade-list mode is being flattered by the arithmetic.

✕Common mistake

Leaving a portfolio system on “Simulate using trade list” is the most common Monte Carlo error, and the symptom is seductive rather than alarming: the simulated drawdown percentiles come back better than the drawdown your backtest report shows. If the Monte Carlo page says your system’s median maximum drawdown is milder than the one you can see on the equity chart, you are almost certainly in the wrong sampling mode.

The list of things it does not randomise

This is short and it is exhaustive, and it is the part most often got wrong:

• It does not randomise prices. No synthetic price series is generated anywhere.
• It does not randomise your parameters. A 20/100 moving-average pair stays 20/100.
• It does not randomise costs, slippage or fills. Whatever the backtest charged, the bootstrap inherits unchanged.
• It does not change the universe, the date range, or the rules.
• It does not invent a trade that did not occur. Every drawn trade is one of yours.

That last point has a precise and useful consequence. In trade-list mode, no realization can contain a single trade worse than your worst actual trade, because there is no such trade in the pool to draw. The tail of the simulated distribution is built entirely out of unlucky orderings and repetitions of losses you have already seen. If your sample never contained a catastrophe, neither will any of the five thousand futures you just simulated.

Exchangeability: the assumption that carries all the weight

Resampling with replacement is legitimate when the observations are exchangeable: when the order in which they arrived carries no information, so that any rearrangement is as plausible as the one that happened. Coin flips are exchangeable. Draws from an urn are exchangeable. Trades are not.

Four reasons, all of them structural rather than accidental:

1. Trades share a regime. A breakout system does well while breakouts persist and badly while they do not, and those conditions arrive in blocks lasting months. The losses do not sprinkle themselves evenly across the record; they queue up.
2. Concurrent positions share exposure. In a portfolio system the trades open at the same time are frequently in correlated instruments, so they win and lose together. They are not independent draws even in principle.
3. Sizing links one trade to the next. Any percent-of-equity rule makes each trade’s size a function of everything before it. That is serial dependence deliberately built into the system.
4. The rules themselves have memory. Trailing stops, regime filters, ranking by recent strength — all of these make the availability and the character of the next trade depend on what just happened.

The consequence is specific: a bootstrap breaks up the clusters. It takes a run of six consecutive losses that happened because the market was in one particular state and scatters them across the sequence, which produces a smoother equity curve and a shallower drawdown than the world that generated them. The simulated drawdown distribution is therefore biased towards the optimistic side, and the bias grows with how strongly your results cluster.

∴Research note

That is a claim you can check rather than accept. The rest of this section builds a measurement of clustering in your own equity curve, using the exact series AmiBroker resamples in equity-changes mode. If the measurement comes back unremarkable, you have learned something specific about your system. If it comes back strongly clustered, you know to read the Monte Carlo drawdown column as an underestimate.

Goal: measure the clustering the bootstrap is about to destroy

We want a number, not an impression. Two standard statistics do the job, and both operate on the bar-by-bar percentage changes in portfolio equity — the series AmiBroker publishes as the composite symbol ~~~EQUITY after every portfolio backtest, and the same series it draws from in “Simulate using portfolio equity changes” mode.

The first is a runs test. Count the unbroken blocks of same-signed changes. If the signs were independent, the expected number of blocks for u up-bars and d down-bars is 2ud/(u+d) + 1, with a known variance, so the observed count converts to a z-score. Fewer, longer runs than expected — a negative z — is exactly what clustering looks like.

The second is lag-1 autocorrelation, computed both on the changes themselves and on their absolute values. The autocorrelation of the absolute changes is the more revealing of the two: it measures whether large moves are followed by large moves, which is the volatility clustering that makes drawdowns arrive in one piece rather than in instalments.

The complete formula

Complete runnable AFL

equity-independence-check.afl

// equity-independence-check.afl
// Part 33 - Monte Carlo and Robustness
//
// The built-in Monte Carlo simulator resamples your results with replacement.
// That is only meaningful if the things being resampled are exchangeable -
// if the order they occurred in carried no information. This formula tests
// that assumption against the very series AmiBroker bootstraps in
// "Simulate using portfolio equity changes" mode: the bar-by-bar percentage
// changes of portfolio equity, published by the last portfolio backtest as the
// composite symbol ~~~EQUITY.
//
// HOW TO USE IT
//   1. Run a portfolio Backtest (mc-baseline-system.afl will do). AmiBroker
//      writes portfolio equity into ~~~EQUITY as it finishes.
//   2. Run THIS formula as an Exploration, applied to the current symbol, over
//      the same range. It reads ~~~EQUITY, not the symbol you are standing on.
//   3. Read the z-score and the two autocorrelations.
//
// WHAT THE NUMBERS MEAN
//   Runs z-score .... The Wald-Wolfowitz runs test. Count how many unbroken
//                     blocks of same-signed changes the series contains. Under
//                     independence the expected count is
//                       E = 2*u*d/(u+d) + 1
//                     with variance
//                       V = 2*u*d*(2*u*d - u - d) / ((u+d)^2 * (u+d-1))
//                     for u up-bars and d down-bars. A z far below zero means
//                     fewer, longer runs than independence predicts: winning
//                     stretches and losing stretches, which is clustering.
//   Autocorr(1) ..... Lag-1 autocorrelation of the changes themselves.
//   |Autocorr|(1) ... Lag-1 autocorrelation of the ABSOLUTE changes. This is
//                     the one that usually bites. Big days follow big days.
//
// LIMITS - state them with any number this produces:
//   * A single test statistic on a single equity curve is one observation.
//     A z-score near zero is not proof of independence.
//   * Bars on which equity did not move at all are excluded from the runs
//     test, because a zero has no sign. A system that is often flat therefore
//     gets tested on a shorter series than you might expect.
//   * The test says nothing about WHY the series clusters. Regime is the usual
//     answer, but this formula cannot demonstrate that.

Eq = Foreign( "~~~EQUITY", "C" );

Chg    = Cum( 0 );
AbsChg = Cum( 0 );
Valid  = Cum( 0 );

// ---- Pass 1: collect the changes and count sign runs ---------------------
ValidCount = 0;
SumChg     = 0;
SumAbs     = 0;
UpBars     = 0;
DownBars   = 0;
RunCount   = 0;
PrevSign   = 0;

for ( i = 1; i < BarCount; i++ )
{
    if ( Eq[ i ] > 0 AND Eq[ i - 1 ] > 0 )
    {
        Change = Eq[ i ] / Eq[ i - 1 ] - 1;

        Chg[ i ]    = Change;
        AbsChg[ i ] = abs( Change );
        Valid[ i ]  = 1;

        ValidCount = ValidCount + 1;
        SumChg     = SumChg + Change;
        SumAbs     = SumAbs + abs( Change );

        // A zero change has no sign, so it takes no part in the runs test and
        // does not break the run either.
        if ( Change > 0 )
        {
            UpBars = UpBars + 1;
            if ( PrevSign != 1 )
            {
                RunCount = RunCount + 1;
            }
            PrevSign = 1;
        }
        if ( Change < 0 )
        {
            DownBars = DownBars + 1;
            if ( PrevSign != -1 )
            {
                RunCount = RunCount + 1;
            }
            PrevSign = -1;
        }
    }
}

MeanChg = 0;
MeanAbs = 0;
if ( ValidCount > 0 )
{
    MeanChg = SumChg / ValidCount;
    MeanAbs = SumAbs / ValidCount;
}

// ---- Pass 2: variance and lag-1 covariance -------------------------------
VarChg  = 0;
VarAbs  = 0;
CovChg  = 0;
CovAbs  = 0;

for ( i = 1; i < BarCount; i++ )
{
    if ( Valid[ i ] == 1 )
    {
        DevChg = Chg[ i ] - MeanChg;
        DevAbs = AbsChg[ i ] - MeanAbs;

        VarChg = VarChg + DevChg * DevChg;
        VarAbs = VarAbs + DevAbs * DevAbs;

        // Only consecutive bars form a lag-1 pair.
        if ( Valid[ i - 1 ] == 1 )
        {
            CovChg = CovChg + DevChg * ( Chg[ i - 1 ] - MeanChg );
            CovAbs = CovAbs + DevAbs * ( AbsChg[ i - 1 ] - MeanAbs );
        }
    }
}

AutoChg = 0;
AutoAbs = 0;
if ( VarChg > 0 )
{
    AutoChg = CovChg / VarChg;
}
if ( VarAbs > 0 )
{
    AutoAbs = CovAbs / VarAbs;
}

// ---- The runs test --------------------------------------------------------
SignedBars   = UpBars + DownBars;
ExpectedRuns = 0;
RunsZ        = 0;

if ( UpBars > 0 AND DownBars > 0 AND SignedBars > 1 )
{
    Product      = UpBars * DownBars;
    ExpectedRuns = 2 * Product / SignedBars + 1;
    RunsVariance = 2 * Product * ( 2 * Product - SignedBars ) /
                   ( SignedBars * SignedBars * ( SignedBars - 1 ) );
    if ( RunsVariance > 0 )
    {
        RunsZ = ( RunCount - ExpectedRuns ) / sqrt( RunsVariance );
    }
}

// ---- Output ---------------------------------------------------------------
Filter = Status( "lastbarinrange" ) AND ValidCount > 0;

SetOption( "NoDefaultColumns", True );

AddTextColumn( "~~~EQUITY", "Series", 1.0 );
AddColumn( ValidCount,   "Bars used",        1.0 );
AddColumn( UpBars,       "Up bars",          1.0 );
AddColumn( DownBars,     "Down bars",        1.0 );
AddColumn( RunCount,     "Runs observed",    1.0 );
AddColumn( ExpectedRuns, "Runs expected",    1.1 );
AddColumn( RunsZ,        "Runs z-score",     1.2 );
AddColumn( AutoChg,      "Autocorr lag 1",   1.3 );
AddColumn( AutoAbs,      "Abs autocorr lag 1", 1.3 );

Download equity-independence-check.afl168 lines

How it works

The formula reads ~~~EQUITY through Foreign() rather than using the symbol it happens to be standing on — the instrument you select supplies nothing but a place to put the output row. It then makes two passes over the array.

The first pass walks the bars, and for every bar where both the current and the previous equity values are positive it records the percentage change, accumulates the sums it will need for the means, and updates the runs count. A change of exactly zero is deliberately skipped by the runs logic: zero has no sign, so counting it as either would corrupt the test, and treating it as a break would inflate the run count on any system that spends time in cash.

The second pass computes the sums of squared deviations and the lag-1 cross-products. Doing this in a second pass rather than a single one means both statistics use the full-sample mean, which is what the textbook definition of the autocorrelation calls for. The lag-1 term is only accumulated when the previous bar was itself valid, so a gap in the equity series does not silently create a pair out of two bars that were never adjacent.

The runs arithmetic at the end is the Wald–Wolfowitz expression written out in full, with a guard for the degenerate cases — an equity curve that only ever went up, or one with fewer than two signed bars — where the variance is undefined.

Key functions

• Foreign( "~~~EQUITY", "C" ) — reads the closing values of another symbol into the current formula. The ~~~EQUITY composite is written by AmiBroker at the end of every portfolio-level backtest and holds that backtest’s portfolio equity.
• Cum( 0 ) — returns an array of zeros with one element per bar. It is the idiomatic way to create a working array you are going to fill by index.
• Status( "lastbarinrange" ) — true on the final bar of the analysis range. It reduces the exploration to a single summary row instead of one row per bar.
• SetOption( "NoDefaultColumns", True ) — suppresses the automatic Ticker and Date/Time columns, which would be misleading here because the output does not describe the ticker.

Expected result

One row, with the number of bars used, the up and down counts, the observed and expected run counts, the z-score and the two autocorrelations.

Read the z-score first, and read its sign before its magnitude. A negative z means fewer runs than independence predicts, which is to say longer stretches of consecutive gains and consecutive losses. Values within roughly two standard deviations of zero are unremarkable on any single sample and should not be talked about as a finding.

The absolute-change autocorrelation is usually the more clearly non-zero of the two, and a positive value there is the ordinary condition of financial time series rather than a peculiarity of your system. It is also the number with the most direct bearing on what you are about to do: if large moves cluster, then large adverse moves cluster, and the drawdown distribution you are about to read has been built by pulling them apart.

Test it

Run the formula twice on the same completed backtest and confirm you get identical output — there is no randomness in it, so any variation means the composite changed underneath you. Then run a portfolio backtest with a deliberately different character, such as one that holds a single position and spends most of its time in cash, and check that the bars-used count falls accordingly. Finally, sanity-check the arithmetic: the observed and expected run counts should both be somewhere near half the number of signed bars for a series with no strong structure, and u + d should equal the bars used minus the count of exactly-flat bars.

Common errors

• No ~~~EQUITY symbol. The composite only exists after a portfolio-level backtest has been run in this database. Run one first; the exploration will produce no rows otherwise.
• A stale composite. ~~~EQUITY holds the last portfolio backtest, whatever that was. Run the backtest you mean to analyse immediately before the exploration.
• Applying it to every symbol in a watch list. The result is identical for every symbol because it never reads the symbol, so you get hundreds of identical rows. Apply it to the current symbol.
• Reading a single z-score as proof. One statistic on one equity curve is one observation, and the runs test detects only one particular kind of dependence.

Extension

Extend the second pass to compute the autocorrelation at lags 2 to 10 as well, and print the whole set. A single significant lag-1 term can come from mechanical causes such as a multi-day position being marked to market; a decaying tail of positive terms across many lags is a stronger signal that the equity series has a persistent structure the bootstrap cannot see.

Reading the two distributions that matter

The simulation records several quantities. Two of them carry most of the decision weight.

Terminal equity

The distribution of final equity across all realizations answers: how much of where I ended up was a consequence of the order in which my trades happened, rather than of the trades themselves?

For a system sized as a fixed percentage of equity with no other constraints, the answer is often “surprisingly little” — and there is a reason, examined properly in the sequence-risk lesson. Multiplying a fixed set of growth factors together gives the same product in any order. Where the terminal distribution genuinely widens is where the sampling with replacement bites: a realization that happens to draw your three best trades twice each and skip your worst finishes somewhere your backtest never did.

Drawdown

The drawdown distribution is the one worth your attention, because drawdown is order-dependent even when terminal equity is not. The same trades in a different sequence produce a genuinely different worst peak-to-valley decline, and that number is the one that decides whether a real person keeps following the rules.

Treat it as a floor rather than as an estimate. It is a floor for two separate reasons: the bootstrap has scattered your clustered losses, and, in trade-list mode, it has serialised positions that originally overlapped. Both errors point the same way.

What a percentile is a statement about

A percentile is a statement about the simulated sample, and nothing else. The 10th percentile of maximum drawdown means that ten per cent of the realizations experienced a drawdown at least that bad. It is a description of five thousand rearrangements of your own history. It is not a probability attached to next year.

!Important

Write the sentence out in full before you use the number anywhere: “In ten per cent of five thousand bootstrap resamples of the eighty-three trades this system took on this universe between these dates, under these cost assumptions, the maximum drawdown was at least this deep.” Every clause in that sentence is load-bearing. If you cannot state the number that way, you are not entitled to state it at all.

Why none of this is a forecast

Three reasons, in increasing order of seriousness.

The bootstrap treats your sample as the population. Every draw comes from the trades you took. Whatever the sample failed to contain — a liquidity crisis, a gap through your stop, a change in the market’s character — cannot appear in any realization, no matter how many you run. Running ten thousand instead of one thousand narrows the sampling noise around a distribution that is centred on the wrong thing, if the sample is unrepresentative.

The trades themselves came from a backtest, not from a market. They inherit every optimistic fill, every uncharged cost, every symbol your database quietly dropped after it was delisted, and every parameter you chose after looking at the answer. The simulator has no way to detect any of this and no interest in it. Feed it a curve-fitted system and it will produce a beautifully tight distribution of curve-fitted outcomes.

The independence assumption is false in the direction that matters. Not “approximately true”, not “true enough for practical purposes” — false, structurally, and false in a way that makes the risk numbers look better than they should. That is the worst kind of modelling error, because it is invisible in the output.

None of this makes the exercise worthless. It makes it a sensitivity analysis rather than a prediction: a way of finding out how much of your result rode on the particular ordering you happened to get. That is a real question with a real answer, and it is the only question the procedure answers.

Recap

Monte Carlo simulation in AmiBroker means bootstrapping — sampling with replacement — either the realised trade list or the bar-by-bar percentage changes in portfolio equity, and replaying the result a thousand or more times to build distributions of final equity, annual return, drawdown and lowest equity. Nothing else is randomised: not prices, not parameters, not costs, not the universe.

The procedure assumes the resampled units are exchangeable. Trades are not: they cluster by regime, they share exposure when they overlap, and percent-of-equity sizing links each one to its predecessors. Because clustering is what the bootstrap destroys, the simulated drawdown distribution reads optimistically, and you now have a formula that measures the clustering in your own equity curve rather than leaving it as an assertion.

A percentile from the simulation is a description of rearrangements of your own past. Used as a sensitivity analysis it is informative; used as a forecast it is worse than useless, because it comes with a spurious air of precision. The next lesson takes you through the feature itself — where it lives, what every setting does, and how to read the report page in the direction it is actually written.

Check your understanding

Question 1. A portfolio system holds up to ten positions at once. Its backtest report shows a maximum system drawdown of 31 per cent. The Monte Carlo page, run in "Simulate using trade list" mode, reports a median maximum drawdown of 19 per cent. What is the most likely explanation?

• The system is more robust than the single backtest suggested
• Trade-list mode replays overlapping trades one after another, so simultaneous losses become separate smaller ones
• The number of runs was too low
• The report and the simulation use different definitions of drawdown

Show the answer and why

Answer: Trade-list mode replays overlapping trades one after another, so simultaneous losses become separate smaller ones

The User’s Guide is explicit that bootstrapped trades are replayed sequentially. Six positions that fell together in reality become six separate dents in the simulation, so the drawdown comes out too small. For a system with overlapping positions the documented choice is "Simulate using portfolio equity changes".

Question 2. In trade-list mode, which of these could appear in a bootstrap realization? Select all that apply.

• A single trade that lost more than the worst trade in the original list
• The same original trade occurring three times
• A trade on a symbol the backtest never traded
• A sequence of losses longer than any losing streak in the original list

Show the answer and why

Answer: The same original trade occurring three times, A sequence of losses longer than any losing streak in the original list

Sampling is with replacement from the original trades, so repetition is expected and unusually long streaks can be assembled by chance. But the pool contains only trades that actually occurred, so no realization can contain a worse individual trade, or a trade on a symbol that was never traded. The tail of the distribution is built from bad orderings, not from bad events.

Question 3. What does "exchangeable" mean in the context of bootstrapping a trade list?

• Every trade has the same expected profit
• Any reordering of the trades is as plausible as the order that actually occurred
• Trades can be swapped between different systems
• The trades are all the same size

Show the answer and why

Answer: Any reordering of the trades is as plausible as the order that actually occurred

Exchangeability is about order carrying no information. It is the assumption that licenses resampling. Real trades violate it because market conditions persist, so losses arrive in blocks — which is precisely the structure the bootstrap discards.

Question 4. The Monte Carlo page reports a 5th-percentile maximum drawdown of 42 per cent. Which statement is defensible?

• There is a 5 per cent chance the system loses 42 per cent next year
• Five per cent of the resamples of this backtest’s own results reached a drawdown at least that deep
• The system will not exceed a 42 per cent drawdown 95 per cent of the time
• The 42 per cent figure is the worst case for this system

Show the answer and why

Answer: Five per cent of the resamples of this backtest’s own results reached a drawdown at least that deep

A percentile describes the simulated sample. It is a statement about rearrangements of a particular set of trades on a particular universe over a particular period under particular cost assumptions — not a probability attached to the future, and certainly not a worst case, since the sample can only contain events that already happened.

Question 5. Why does the independence check read the composite symbol ~~~EQUITY rather than the price series of the symbol it is run on?

• Because Foreign() cannot read the current symbol
• Because ~~~EQUITY holds the portfolio equity of the last portfolio backtest, which is the series the equity-changes bootstrap resamples
• Because composites are faster to read than price arrays
• Because the price series has too many bars

Show the answer and why

Answer: Because ~~~EQUITY holds the portfolio equity of the last portfolio backtest, which is the series the equity-changes bootstrap resamples

The point of the test is to interrogate the exact series the simulator is about to treat as exchangeable. AmiBroker writes portfolio equity to ~~~EQUITY at the end of every portfolio-level backtest, so reading it puts the test on the same data the bootstrap uses.

Check my answersClear