Components 7 and 8: Backtest and Robustness Evaluation

The order these must be done in

Validation before results, every time

1. 11. Run the portfolio backtestOver the stated universe and range, with the report's trade list turned on. Do not read the summary yet.
2. 22. Hand-check three tradesA winner, a loser, and one that exited by stop. Reconstruct each bar by bar against the chart.
3. 33. Run the execution auditWas every trade physically possible? Same-bar leak, participation, gaps through the stop.
4. 44. NOW read the summaryExposure first, then Risk Adjusted Return, then drawdown, then the return figure.
5. 55. Parameter sensitivityThe shape of the surface, not the best cell.
6. 66. Out-of-sampleOn data you have genuinely not looked at. Once.
7. 77. Walk-forwardTesting the process rather than a parameter set.
8. 88. Assemble the evidenceEvery number attached to its assumptions.

Steps 2 and 3 come before step 4 for a reason that is easy to state and hard to obey: once you have seen a good number, you will not look for the reason it is wrong.

Component 7: the portfolio backtest

Procedure

1. Settings check. Periodicity daily. Initial equity matching the formula. Margin 100. Interest rate 0. Max open positions matching PosQty. Delays 1 on all four. “Include trade list in the report” on.
2. Save the configuration as an .APX. The Analysis window’s Apply to and Range live in a dialog and appear nowhere in the formula. They are the two settings most likely to differ from what you believe, and an .APX is self-contained.
3. Run it. Once, at the specification’s parameters.
4. Do not read the summary. Go to step 2 of the list above.

Hand-checking three trades

Pick a winner, a loser, and one that exited by stop — different faults hide in each. For every one, verify in order:

Check	What must be true
Signal bar	The entry condition was true on that bar, including the regime and liquidity gates
Entry bar	The next bar. If it is the same bar, your delays are not what you think
Entry price	That bar’s open × (1 + slippage), unless the open was the bar’s high — in which case exactly the high, because of price-bound clamping
Share count	Position value ÷ entry price, whole shares, value = one PosQty-th of that bar’s equity
Exit	Which fired — the rule or the stop? If the stop, a prior bar’s range went below the level and the exit is on the following bar
Commission	CommissionPct of each leg’s value

If all six agree on all three trades, the machinery is doing what you believe. If any one disagrees, the same discrepancy is in every other trade too.

The execution audit

Complete runnable AFL

strategy-audit.afl

// strategy-audit.afl
// Capstone Component 7 - the evidence that the backtest was physically possible
//
// GOAL
//   A backtest report says what the engine decided. This exploration says
//   whether those decisions could have been executed. Run it over the same
//   universe and range as the capstone backtest, and attach the output to the
//   research report.
//
//   It answers five questions, one column group each:
//     1. Was the entry decided on a bar and filled on a LATER one?
//     2. How much would a same-bar fill have handed the system for free?
//     3. How large is the intended position against what actually traded?
//     4. How often would the stop have been gapped through?
//     5. What does the round trip cost, as a percentage?
//
// HOW TO RUN
//   Analysis -> Apply to: the capstone universe.  Periodicity: Daily.
//   Range: the capstone range.  Press EXPLORE.
//   Sort by "Participation % of bar volume" descending and read the top rows.
//
// ASSUMPTIONS
//   Account size    stated, not read from a live equity curve. A compounding
//                   backtest will want LARGER positions than these as equity
//                   grows, so treat these figures as the optimistic case.
//   Ref( ..., 1 )   reads the bar after the signal. Legitimate in an AUDIT,
//                   which looks backwards at what followed. The identical
//                   expression inside a Buy rule is a bug.
//   Scope           this checks execution realism. It says nothing about
//                   whether the rules have an edge.

SetBarsRequired( sbrAll, sbrAll );

BenchSymbol      = ParamStr( "Benchmark symbol", "" );
AccountSize      = Param( "Assumed account size", 100000, 10000, 10000000, 10000 );
PosQty           = Param( "Max open positions", 10, 1, 40, 1 );
RegimePeriod     = Param( "Benchmark regime average", 200, 20, 400, 10 );
TrendPeriod      = Param( "Symbol trend average", 200, 20, 400, 10 );
BreakoutPeriod   = Param( "Entry look-back (bars)", 50, 5, 250, 5 );
StopAtrMult      = Param( "Stop distance (x ATR)", 3, 0.5, 10, 0.5 );
AtrPeriod        = Param( "ATR period", 20, 2, 100, 1 );
LiquidityPeriod  = Param( "Liquidity look-back (bars)", 50, 5, 400, 5 );
CommissionPct    = Param( "Commission per trade (%)", 0.10, 0, 1.00, 0.01 );
SlippagePct      = Param( "Slippage per fill (%)", 0.15, 0, 2.00, 0.01 );

// ------------------------------------------------ the rule under audit
// Identical to strategy.afl's entry. If you change the strategy, change this.
Turnover      = Median( Close * Volume, LiquidityPeriod );
TrendMa       = MA( Close, TrendPeriod );
BreakoutLevel = Ref( HHV( High, BreakoutPeriod ), -1 );

HaveBench = StrLen( BenchSymbol ) > 0;

if( HaveBench )
{
    BenchClose = Foreign( BenchSymbol, "C" );
    RegimeOpen = NOT IsNull( BenchClose ) AND BenchClose > 0
             AND BenchClose > MA( BenchClose, RegimePeriod );
}
else
{
    RegimeOpen = True;
}

EntrySignal = RegimeOpen
          AND Close > TrendMa
          AND Cross( Close, BreakoutLevel )
          AND Volume > 0;

// ------------------------------------------ 1 and 2. the fill assumption
FillSameBarOpen  = Open;
FillSameBarClose = Close;
FillNextBarOpen  = Ref( Open, 1 );

// The gap between deciding on the close and being filled at that same bar's
// open. Positive means the leak paid the system before the trade began.
SameBarLeakPct = 100 * SafeDivide( Close - Open, Open, 0 );

// What the honest assumption costs relative to the leaky one, on this bar.
OvernightGapPct = 100 * SafeDivide( FillNextBarOpen - Close, Close, Null );

// ------------------------------------------------ 3. size and absorption
IntendedValue  = AccountSize / PosQty;
FillPrice      = FillNextBarOpen * ( 1 + SlippagePct / 100 );
IntendedShares = SafeDivide( IntendedValue, FillPrice, Null );

EntryBarVolume   = Ref( Volume, 1 );
ParticipationBar = 100 * SafeDivide( IntendedShares, EntryBarVolume, Null );
ParticipationMed = 100 * SafeDivide( IntendedValue, Turnover, Null );

// ---------------------------------------------------- 4. the stop realism
StopDistance = StopAtrMult * ATR( AtrPeriod );
StopLevel    = FillPrice - StopDistance;

// Did the bar AFTER the fill open below where the stop would have sat? If so, a
// backtest that exits "at the stop level" is claiming a price that was never
// available on that bar.
NextNextOpen   = Ref( Open, 2 );
GapThroughStop = NOT IsNull( NextNextOpen ) AND NextNextOpen < StopLevel;

// ------------------------------------------------------------- 5. costs
RoundTripCostPct = 2 * CommissionPct + 2 * SlippagePct;

// A cost that is large relative to the stop distance means the system is paying
// a meaningful fraction of its own risk budget just to open and close.
StopPct       = 100 * SafeDivide( StopDistance, FillPrice, Null );
CostVsRiskPct = 100 * SafeDivide( RoundTripCostPct, StopPct, Null );

// -------------------------------------------------------------- output
Filter = EntrySignal AND Status( "barinrange" ) AND NOT IsNull( FillNextBarOpen );

AddColumn( DateTime(), "Signal bar", formatDateTimeISO );
AddColumn( Close, "Decision close", 1.3 );

AddColumn( FillSameBarOpen,  "Fill: same-bar open",  1.3 );
AddColumn( FillSameBarClose, "Fill: same-bar close", 1.3 );
AddColumn( FillNextBarOpen,  "Fill: next-bar open",  1.3 );
AddColumn( SameBarLeakPct,   "Same-bar leak %",      1.2 );
AddColumn( OvernightGapPct,  "Overnight gap %",      1.2 );

AddColumn( IntendedValue,    "Intended position", 1.0 );
AddColumn( IntendedShares,   "Intended shares",   1.0 );
AddColumn( EntryBarVolume,   "Entry bar volume",  1.0 );
AddColumn( ParticipationBar, "Participation % of bar volume", 1.2 );
AddColumn( ParticipationMed, "Participation % of median turnover", 1.2 );

AddColumn( StopLevel,        "Stop level", 1.3 );
AddColumn( StopPct,          "Stop distance %", 1.2 );
AddColumn( GapThroughStop,   "Gapped through stop", 1.0 );

AddColumn( RoundTripCostPct, "Round-trip cost %", 1.2 );
AddColumn( CostVsRiskPct,    "Cost as % of risk budget", 1.1 );

// COUNT and AVERAGE. The average same-bar leak and the average participation
// are the two numbers to quote in the report.
AddSummaryRows( 2 | 8 | 16, 1.2 );

Download strategy-audit.afl136 lines

Run as an Exploration over the same universe and range. It prints one row per entry signal with the numbers that decide whether the backtest was physically possible.

=Expected result

Three columns to read, in this order.

Average “Same-bar leak %” — what a same-bar fill would have handed the system for free. Your backtest uses next-bar fills, so this is the size of the error you avoided. On a breakout rule it is often larger than the whole edge, which is a useful thing to be able to quote.

“Participation % of bar volume”, sorted descending. If the top rows exceed a few percent, those positions could not have been established at that size — and any profit from them is fiction. This is the number that decides whether your participation cap is set sensibly.

“Gapped through stop” — how often the bar after the fill opened below where the stop would have sat. Every one of those is a loss larger than your nominal risk per trade, and a backtest using ExitAtStop = 1 would have hidden all of them.

Also worth quoting: “Cost as % of risk budget”. A round trip that consumes a large fraction of the stop distance means the system is paying much of its own risk allowance simply to open and close.

Component 8: robustness

Parameter sensitivity

Complete runnable AFL

strategy-sensitivity.afl

// strategy-sensitivity.afl
// Capstone Component 8 - parameter sensitivity
//
// GOAL
//   The same strategy as strategy.afl, with the four parameters most likely to
//   matter exposed to the optimiser. Its purpose is NOT to find the best cell.
//   It is to see the SHAPE of the surface, which is the only thing an
//   optimisation can honestly tell you on its own.
//
//   Broad plateau  neighbouring parameter pairs behave alike. Encouraging: the
//                  result looks like a property of the idea rather than of two
//                  numbers.
//   Single spike   one cell is good and its neighbours are not. That is what
//                  curve fitting looks like from outside.
//   Flat and dull  everything behaves alike and much like the benchmark. Also
//                  a result, and a common one.
//
// WHAT YOU MUST NOT DO WITH THIS FILE
//   Report the best cell as your capstone result. You have just run hundreds of
//   tests against one dataset, and the best of hundreds looks good partly
//   because it is the best of hundreds. The number you report is the one from
//   the specification you wrote down in advance; this file tells you how much
//   confidence that number deserves.
//
// HOW TO RUN
//   1. Analysis -> Optimize, over the SAME universe, range and costs as
//      strategy.afl. Record the whole result table, not the top row.
//   2. Sort by CAR/MaxDD and look at where the good cells SIT relative to each
//      other, not at how good the best one is.
//   3. Then run it once as an ordinary Backtest at the specification defaults,
//      which is your actual result.
//
// ASSUMPTIONS
//   Identical to strategy.afl in every respect except that four Param() calls
//   have become Optimize() calls. If you change anything else, the sensitivity
//   study is measuring the change instead of the parameters.

BenchSymbol      = ParamStr( "Benchmark symbol", "" );
PosQty           = Param( "Max open positions", 10, 1, 40, 1 );
StartingEquity   = Param( "Starting equity", 100000, 10000, 10000000, 10000 );

// ------------------------------------------------- the four under test
// Ranges are wide enough to show the surface and coarse enough to finish. A
// finer grid does not make the study more informative; it makes it slower and
// increases the number of specifications you have to admit to having tried.
BreakoutPeriod = Optimize( "Entry look-back",   50,  20, 100, 10 );
ExitPeriod     = Optimize( "Exit look-back",    25,  10,  60,  5 );
StopAtrMult    = Optimize( "Stop (x ATR)",       3, 1.5,   6, 0.5 );
TrendPeriod    = Optimize( "Symbol trend average", 200, 100, 300, 50 );

RegimePeriod     = Param( "Benchmark regime average", 200, 20, 400, 10 );
AtrPeriod        = Param( "ATR period", 20, 2, 100, 1 );
LiquidityPeriod  = Param( "Liquidity look-back (bars)", 50, 5, 400, 5 );
MinTurnover      = Param( "Minimum median turnover", 2000000, 0, 100000000, 250000 );
MinPrice         = Param( "Minimum close", 2, 0, 500, 0.5 );
ParticipationPct = Param( "Max % of median turnover", 1, 0.05, 25, 0.05 );
CommissionPct    = Param( "Commission per trade (%)", 0.10, 0, 1.00, 0.01 );
SlippagePct      = Param( "Slippage per fill (%)", 0.15, 0, 2.00, 0.01 );

// ------------------------------------------------------------ account
SetOption( "InitialEquity",           StartingEquity );
SetOption( "MaxOpenPositions",        PosQty );
SetOption( "AllowPositionShrinking",  True );
SetOption( "MinPosValue",             1000 );
SetOption( "AccountMargin",           100 );
SetOption( "InterestRate",            0 );
SetOption( "CommissionMode",          1 );
SetOption( "CommissionAmount",        CommissionPct );
SetOption( "AllowSameBarExit",        False );
SetOption( "UsePrevBarEquityForPosSizing", True );

RoundLotSize = 1;
SetBacktestMode( backtestRegular );
SetPositionSize( 100 / PosQty, spsPercentOfEquity );
SetTradeDelays( 1, 1, 1, 1 );

BuyPrice   = Open * ( 1 + SlippagePct / 100 );
SellPrice  = Open * ( 1 - SlippagePct / 100 );
ShortPrice = Open * ( 1 - SlippagePct / 100 );
CoverPrice = Open * ( 1 + SlippagePct / 100 );

// ------------------------------------------------------------ universe
Turnover    = Median( Close * Volume, LiquidityPeriod );
BaseDollars = StartingEquity / PosQty;
CapDollars  = ( ParticipationPct / 100 ) * Turnover;

Tradeable = Turnover >= MinTurnover
        AND Close >= MinPrice
        AND Volume > 0
        AND BaseDollars <= CapDollars;

// -------------------------------------------------------------- regime
HaveBench = StrLen( BenchSymbol ) > 0;

if( HaveBench )
{
    BenchClose = Foreign( BenchSymbol, "C" );
    RegimeOpen = NOT IsNull( BenchClose )
             AND BenchClose > 0
             AND BenchClose > MA( BenchClose, RegimePeriod );
}
else
{
    RegimeOpen = True;
}

// --------------------------------------------------------------- rules
TrendMa       = MA( Close, TrendPeriod );
BreakoutLevel = Ref( HHV( High, BreakoutPeriod ), -1 );
ExitLevel     = Ref( LLV( Low, ExitPeriod ), -1 );

RequiredBars = Max( Max( TrendPeriod, RegimePeriod ),
                    Max( BreakoutPeriod, LiquidityPeriod ) ) + 1;
Ready        = BarIndex() >= RequiredBars;

// A combination whose exit window is longer than its entry window is a
// different kind of system, not a variant of this one. Silencing it keeps
// meaningless rows out of the table rather than leaving you to spot them.
if( ExitPeriod >= BreakoutPeriod )
{
    Buy  = 0;
    Sell = 0;
}
else
{
    Buy = IIf( Ready,
               RegimeOpen AND Tradeable
               AND Close > TrendMa
               AND Cross( Close, BreakoutLevel ),
               False );

    Sell = Cross( ExitLevel, Close );
}

Short = False;
Cover = False;

StopDistance = StopAtrMult * ATR( AtrPeriod );
StopAtEntry  = Nz( Ref( StopDistance, -1 ), 0 );
ApplyStop( stopTypeLoss, stopModePoint, StopAtEntry, 2, False, 0, 0, -1 );

PositionScore = Turnover;

Download strategy-sensitivity.afl142 lines

Identical to the strategy in every respect except that four Param() calls have become Optimize() calls. If you change anything else, the study measures the change instead of the parameters.

Run it as an Optimization over the same universe, range and costs. Then look at the shape:

Shape	What it means
Broad plateau	Neighbouring parameter pairs behave alike. Encouraging — the result looks like a property of the idea rather than of two numbers.
Single spike	One cell good, its neighbours not. This is what curve fitting looks like from outside.
Flat and dull	Everything behaves alike, and much like the benchmark. Also a result, and a common one.

⚠Warning

Do not report the best cell as your capstone result. You have just run hundreds of tests against one dataset, and the best of hundreds looks good partly because it is the best of hundreds.

The number you report is the one from the specification you wrote down in advance. The optimisation tells you how much confidence that number deserves — which is a completely different use of the same output.

Out-of-sample

The rule is simple and the discipline is not: hold back a period you have genuinely not looked at, test on it once, and report what happened.

Practically:

• Reserve the most recent 20–30% of your range, or an early period if your data allows.
• Do all of Component 7 and the sensitivity study on the remaining part.
• Then run the specification’s parameters on the held-out part. Once.

✕Common mistake

The moment you look at the out-of-sample result and go back to change the model, that data has joined the training set. You cannot get it back, and extending the period does not recover it.

Doing this twice converts walk-forward analysis into an elaborate form of in-sample fitting. The research log is the only record of how many times you have looked, which is why it matters more than any single technique here.

Walk-forward

Out-of-sample tests a parameter set. Walk-forward tests the process: optimise on a window, apply the winner to the next window, roll forward, repeat, and evaluate only the concatenated out-of-sample segments.

The workflow, the settings and the interpretation are Part 32’s subject — see the AmiBroker walk-forward workflow and interpreting degradation. For the capstone you need three things from it:

1. The out-of-sample equity curve, concatenated across steps.
2. The degradation — how much worse out-of-sample is than in-sample, expressed as a ratio you can quote.
3. Whether the selected parameters were stable across steps. Parameters that jump to a different corner of the grid at every step are telling you the optimiser is fitting noise, and that is often more informative than the performance figures.

Monte Carlo, where appropriate

AmiBroker’s Monte Carlo facility resamples your trade sequence to produce a distribution of outcomes rather than one path. Part 33 covers it and its limits.

∴Research note

Two limitations decide whether it belongs in your report.

It resamples the trades you got. If those trades came from one market regime, every simulated path is a rearrangement of that regime. Monte Carlo cannot generate an environment your sample did not contain — so it addresses sequence risk, not regime risk.

Standard resampling assumes trades are independent. Yours are not: they cluster in time and correlate across symbols. That makes the simulated distribution narrower than reality.

Include it if you understand those two, and state them. Leave it out rather than quoting a “95% confidence” figure whose assumptions you cannot defend.

Assembling the evidence

Every number goes into the report with six clauses attached: what rules, over what universe, over what period, at what costs, with what fill assumption, from what data source. A figure without those six is not evidence.

Evidence	From	What it establishes
Backtest summary	Component 7	What the simulation produced
Trade list, three hand-checked	Component 7	That the engine did what you believe
Execution audit table	Component 7	That the trades were physically possible
Parameter surface	Component 8	Whether the result is a property or a coincidence
Out-of-sample result	Component 8	Whether it survives data you did not fit to
Walk-forward summary and degradation	Component 8	Whether the process survives
Parameter stability across steps	Component 8	Whether the optimiser is fitting noise
Specification count from the log	Research log	How many tries produced the reported number

Validation checklist

Before you write Component 9, confirm every one of these.

1. The .APX reproduces the backtest exactly.
2. Three trades hand-checked, all six checks each.
3. The execution audit was run over the same universe and range.
4. Exposure was read before any return figure.
5. The parameter sensitivity study changed only the four parameters.
6. The out-of-sample data was looked at once, and the log records the date.
7. The walk-forward result reports out-of-sample segments only, not the in-sample ones.
8. Every quoted number has its six clauses.
9. The research log has a count of specifications evaluated.
10. You can state what result would have refuted the hypothesis, and whether it occurred.

▷Exercise

Before running anything in Component 8, write down in the log — dated — what parameter surface you expect. Plateau, spike or flat?

Then run it. Comparing your prediction with the result is a cheap and surprisingly uncomfortable calibration exercise, and it is the closest thing to a controlled experiment on your own judgement that this capstone contains.

Common errors

Reading the summary before validating. The most consequential error in the whole capstone, and the hardest to undo, because you cannot un-see a good number.

Comparing return figures across different exposures. Read Exposure % first; use Risk Adjusted Return % when exposures differ.

Reporting the optimisation’s best cell. See the warning above.

Changing the model after seeing the out-of-sample result. The hold-out is spent.

Walk-forward reported including in-sample segments. The whole point is the out-of-sample concatenation. Check what the report is summarising.

A Monte Carlo confidence interval quoted without its assumptions. Independence and regime coverage are both violated. State them or omit the figure.

The audit run over a different range than the backtest. Then it is auditing a different set of signals. Use the same .APX-recorded range.

Sensitivity study with something else changed. A different universe, a different cost, a different slot count. The surface then shows the effect of that change, not of the parameters.

What to record for the report

Everything in the evidence table, plus:

• The date the out-of-sample data was first examined.
• The number of specifications evaluated, from the log.
• Your prediction of the parameter surface’s shape, dated before you ran it.

Check your understanding

Question 1. Why must the hand-checks and the execution audit come before reading the backtest summary?

• Because AmiBroker recalculates the summary after an exploration
• Because once you have seen a good number you will not look for the reason it is wrong
• Because the audit changes the trade list
• Because the summary is only valid after an exploration has been run

Show the answer and why

Answer: Because once you have seen a good number you will not look for the reason it is wrong

It is a discipline against a known bias in yourself, not a technical requirement. A disappointing result prompts a hunt for errors; a spectacular one prompts a celebration — which is exactly why the validation has to be unconditional and first.

Question 2. The parameter surface shows one good cell surrounded by mediocre neighbours. What does that indicate and what should you report?

• That the good cell is the correct parameter choice and should be reported
• A single spike — the signature of a fit to noise. Report the number from the pre-registered specification, and report the spike as evidence about how much confidence it deserves
• That the grid step size is too large
• That the universe is too small

Show the answer and why

Answer: A single spike — the signature of a fit to noise. Report the number from the pre-registered specification, and report the spike as evidence about how much confidence it deserves

Neighbouring parameters describe nearly the same rule and should behave nearly the same way. When they do not, the good cell is far more likely to be noise. The optimisation is diagnostic; the reported number still comes from the specification written in advance.

Question 3. Which columns of the execution audit belong in the report? Select all that apply.

• Average same-bar leak %, as the size of the error the next-bar fill assumption avoided
• The largest participation percentages, as evidence the positions were establishable
• How often the bar after the fill gapped through the stop level
• The number of rows, as the sample size for the strategy

Show the answer and why

Answer: Average same-bar leak %, as the size of the error the next-bar fill assumption avoided, The largest participation percentages, as evidence the positions were establishable, How often the bar after the fill gapped through the stop level

The row count is a count of entry signals, not of independent observations, and quoting it as a sample size would repeat exactly the error the insufficient-evidence lesson warns about. The other three each answer a specific question about whether the simulation was physically possible.

Question 4. You examine the out-of-sample result, find it disappointing, adjust the exit rule, and re-test on the same held-out data. What is the status of that data?

• Still out-of-sample, since the entry rule did not change
• It has joined the training set and can no longer serve as a hold-out — and extending the period does not recover it
• Valid provided the change is documented
• Valid if the change improves the in-sample result too

Show the answer and why

Answer: It has joined the training set and can no longer serve as a hold-out — and extending the period does not recover it

A hold-out is only a hold-out while it is unlooked-at. Once a result from it has influenced the model, it is training data, whichever part of the model changed. This is why the log must record the date it was first examined.

Question 5. Which limitations must accompany a Monte Carlo figure in this report? Select all that apply.

• It resamples the trades you actually got, so it cannot generate a market regime your sample did not contain
• Standard resampling assumes trades are independent, which they are not — they cluster in time and correlate across symbols
• It addresses sequence risk rather than regime risk
• It requires the Professional edition

Show the answer and why

Answer: It resamples the trades you actually got, so it cannot generate a market regime your sample did not contain, Standard resampling assumes trades are independent, which they are not — they cluster in time and correlate across symbols, It addresses sequence risk rather than regime risk

The first three are what make a confidence figure narrower than reality. Whether to include Monte Carlo at all is a judgement: include it if you can state those limitations, and omit it rather than quote an interval whose assumptions you cannot defend.

Check my answersClear