Challenge Collection: Six Broken Backtests

The previous challenge gave you one report with everything wrong at once. This one is the diagnostic exercise: one system, six faults, exactly one enabled at a time.

Because nothing else changes between runs, every difference in the report is caused by the fault you selected — and the pattern of which metrics move is as informative as the size of the move. Learning those patterns is what lets you diagnose a backtest you did not write.

The formula

Complete runnable AFL

break-fix-collection.afl

// break-fix-collection.afl
// Part 30 - Challenge Collection: Six Broken Backtests
//
// ####################################################################
// #  ONE SYSTEM, SIX SWITCHABLE FAULTS. Set "Fault to enable" to 0    #
// #  for the correct version, then to 1..6 to introduce exactly one   #
// #  defect at a time. Nothing else changes, so every difference in   #
// #  the report is caused by the fault you selected.                  #
// ####################################################################
//
//   0  no fault - the reference version
//   1  future data leak        entry confirmed by the NEXT bar's close
//   2  missing costs           commission and slippage set to zero
//   3  survivorship            symbols without full history are excluded
//   4  repeated signals        raw multi mode, redundant entries kept
//   5  sizing error            one position, 100% of equity, no liquidity cap
//   6  MTF misalignment        weekly high read with TimeFrameGetPrice defaults
//
// Run each one over the same universe and the same date range, and record
// Net Profit %, Annual Return %, Max. system % drawdown, the number of trades
// and the winners percentage in a table. The pattern of which metrics move is
// as informative as the size of the move.
//
// ============================ ASSUMPTIONS (fault 0) ===================
//   Universe        whatever the Analysis window is applied to. For fault 3 to
//                   mean anything the universe must contain symbols that
//                   listed late or stopped trading during the range.
//   Periodicity     Daily.
//   Decision bar    the bar's Close.
//   Fill            the next bar's Open, delays 1 on all four signals,
//                   plus/minus SlippagePct.
//   Commission      CommissionPct of trade value on each leg.
//   Liquidity       50-bar average turnover at the decision bar must clear
//                   MinTurnover, and the intended position must not exceed
//                   ParticipationPct of it.
//   Higher frame    the PREVIOUS completed weekly high, via an explicit
//                   negative shift.
//   Stops           max loss, checked on High-Low, executed on the NEXT bar at
//                   the regular trade price.
// ======================================================================
//
// Nothing this formula produces is a forecast, and the broken variants are not
// results at all. They are demonstrations of specific defects.

FaultNo = Param( "Fault to enable (0 = none)", 0, 0, 6, 1 );

AccountSize      = Param( "Assumed account size", 100000, 10000, 10000000, 10000 );
PosQty           = Param( "Max open positions", 10, 1, 50, 1 );
BreakoutPeriod   = Param( "Breakout lookback (bars)", 50, 10, 250, 5 );
ExitPeriod       = Param( "Exit lookback (bars)", 20, 5, 100, 5 );
StopPct          = Param( "Max loss stop (%)", 10, 1, 30, 0.5 );
CommissionPct    = Param( "Commission per trade (%)", 0.10, 0, 1.00, 0.01 );
SlippagePct      = Param( "Slippage per fill (%)", 0.15, 0, 2.00, 0.01 );
MinTurnover      = Param( "Min 50-bar turnover", 1000000, 0, 50000000, 250000 );
ParticipationPct = Param( "Max % of average daily turnover", 1, 0.05, 25, 0.05 );
RequiredBars     = Param( "Fault 3: bars of history demanded", 750, 50, 5000, 50 );

// Each fault is a single scalar switch, so the code below reads as one correct
// system with six clearly marked wounds.
UseFutureConfirm = FaultNo == 1;
NoCosts          = FaultNo == 2;
SurvivorsOnly    = FaultNo == 3;
RawMultiSignals  = FaultNo == 4;
ConcentratedSize = FaultNo == 5;
LeakyWeekly      = FaultNo == 6;

// ---------------------------------------------------------------------
// 1. Engine mode. SetBacktestMode belongs near the top of the formula.
//    backtestRegular is the default: redundant entry signals between an entry
//    and its matching exit are removed exactly as ExRem() would remove them.
// ---------------------------------------------------------------------
if( RawMultiSignals )
{
    // FAULT 4. Redundant entries are kept AND several positions per symbol may
    // be open at once. One symbol in a long trend can consume the whole book.
    SetBacktestMode( backtestRegularRawMulti );
}
else
{
    SetBacktestMode( backtestRegular );
}

// ---------------------------------------------------------------------
// 2. Costs
// ---------------------------------------------------------------------
if( NoCosts )
{
    // FAULT 2. Both cost assumptions removed. Nothing else changes, so the
    // difference between this run and fault 0 is the whole cost bill.
    CommissionPct = 0;
    SlippagePct   = 0;
}

SetOption( "InitialEquity", AccountSize );
SetOption( "AllowPositionShrinking", True );
SetOption( "AllowSameBarExit", False );
SetOption( "CommissionMode", 1 );
SetOption( "CommissionAmount", CommissionPct );
SetOption( "UsePrevBarEquityForPosSizing", True );

SetTradeDelays( 1, 1, 1, 1 );

BuyPrice   = Open * ( 1 + SlippagePct / 100 );
SellPrice  = Open * ( 1 - SlippagePct / 100 );
ShortPrice = Open * ( 1 - SlippagePct / 100 );
CoverPrice = Open * ( 1 + SlippagePct / 100 );

// ---------------------------------------------------------------------
// 3. Universe membership
// ---------------------------------------------------------------------
if( SurvivorsOnly )
{
    // FAULT 3. Every symbol that does not carry a full history is thrown out.
    // That sounds like data hygiene. What it actually removes is everything
    // that listed late and, far more damagingly, everything that stopped
    // trading - which is the population whose returns were worst.
    Exclude = BarCount < RequiredBars;
}

// ---------------------------------------------------------------------
// 4. Higher-timeframe context
// ---------------------------------------------------------------------
if( LeakyWeekly )
{
    // FAULT 6. TimeFrameGetPrice defaults to shift = 0 and mode = expandFirst.
    // The whole week's high is therefore written onto the week's FIRST bar, so
    // a Monday comparison already knows Friday's high.
    WeeklyHigh = TimeFrameGetPrice( "H", inWeekly );
}
else
{
    // -1 means the previous COMPLETED weekly bar.
    WeeklyHigh = TimeFrameGetPrice( "H", inWeekly, -1 );
}

// ---------------------------------------------------------------------
// 5. The rule
// ---------------------------------------------------------------------
Turnover      = MA( Close * Volume, 50 );
BaseDollars   = AccountSize / PosQty;
CapDollars    = ( ParticipationPct / 100 ) * Turnover;

BreakoutLevel = Ref( HHV( High, BreakoutPeriod ), -1 );
ExitLevel     = Ref( LLV( Low, ExitPeriod ), -1 );

Tradable = Turnover >= MinTurnover
       AND Volume > 0
       AND BaseDollars <= CapDollars;

Setup = Close > BreakoutLevel
    AND Close > 0.98 * WeeklyHigh;

if( UseFutureConfirm )
{
    // FAULT 1. "Only take the breakout if it followed through." The follow
    // through is measured on the bar AFTER the decision bar, which had not
    // printed when the decision was made. Ref() with a positive shift is
    // documented as referencing the future; here it is simply a bug.
    Setup = Setup AND Ref( Close, 1 ) > Close;
}

Buy  = Setup AND Tradable;
Sell = Close < ExitLevel;

// ---------------------------------------------------------------------
// 6. Sizing
// ---------------------------------------------------------------------
if( ConcentratedSize )
{
    // FAULT 5. One position at a time, the whole account in it, and the
    // liquidity condition dropped. Every winner compounds into the next
    // position, and nothing stops the requested share count exceeding the
    // entire volume that traded on the entry bar.
    SetOption( "MaxOpenPositions", 1 );
    PositionSize = -100;
    Buy = Setup AND Volume > 0;
}
else
{
    SetOption( "MaxOpenPositions", PosQty );
    PositionSize = -100 / PosQty;
}

PositionScore = Turnover;

ApplyStop( stopTypeLoss, stopModePercent, StopPct, 2 );

_SECTION_BEGIN( "Break-fix view" );
Plot( Close, "Close", colorDefault, styleCandle );
Plot( BreakoutLevel, "Breakout level", colorGreen, styleLine );
Plot( WeeklyHigh, "Weekly high as used", colorRed, styleLine );
PlotShapes( Buy * shapeUpArrow, colorGreen, 0, Low );
PlotShapes( Sell * shapeDownArrow, colorRed, 0, High );
_SECTION_END();

Download break-fix-collection.afl194 lines

Set “Fault to enable” to 0 for the correct reference version, then to 1 through 6 for one defect at a time.

How to run the exercise

1. Run fault 0 over your universe and range. Record the reference figures.
2. Run faults 1 to 6 in turn, over the identical universe and range.
3. For each, record: Net Profit %, Annual Return %, Max. system % drawdown, number of trades, Winners %, Exposure % and Avg. Bars Held.

For fault 3 to mean anything, the universe must contain symbols that listed late or stopped trading during the range. On a clean current-membership index list it will do almost nothing, which is itself worth seeing.

	Trades	Net Profit %	Ann. Ret %	Max DD %	Winners %	Exposure %	Avg Bars
0 — reference
1
2
3
4
5
6

Your task

Before reading the solutions, work down the table and for each fault answer three questions:

• Which metrics moved, and in which direction?
• What single mechanism would produce exactly that pattern?
• What evidence, beyond the summary, would confirm it?

The third question is the one that matters professionally. A hypothesis you cannot test is a guess.

▷Exercise

Cover the solutions and try to identify at least four of the six from the metric pattern alone, without looking at the code. The formula is deliberately structured so that each fault is a single scalar switch, so once you have a hypothesis, confirming it in the code takes seconds.

The six faults, with symptoms and solutions

Fault 1 — future data leak

Fragment — not a complete formula

// "Only take the breakout if it followed through."
Setup = Setup AND Ref( Close, 1 ) > Close;

Symptoms. Winners % rises sharply, often to 70% or more. Average loss shrinks. The equity curve becomes conspicuously smooth. Trade count falls, because the extra condition filters signals.

Diagnosis. Ref() with a positive shift references the future — the documentation says so explicitly. The follow-through is measured on the bar after the decision bar, which had not printed when the decision was made.

How you would confirm it. Two ways. Search the formula for any positive Ref() shift, which is a five-second check. Or run the entry rule as an Exploration and look at how many signals it produces on the last bar of the data: a rule that depends on the next bar cannot fire on the final bar, so the signal count drops to zero at the right-hand edge in a way an honest rule’s does not.

The fix. Delete the condition, or express the follow-through with a negative shift and accept that the entry is one bar later.

Why it is seductive. “Wait for confirmation” is genuinely sound advice. The fault is not the idea — it is measuring the confirmation on a bar that had not happened.

Fault 2 — missing costs

Fragment — not a complete formula

CommissionPct = 0;
SlippagePct   = 0;

Symptoms. Net profit rises by an amount proportional to the trade count. Winners % rises slightly — trades that were marginally negative after costs become marginally positive. Drawdown improves modestly. Trade count and exposure are unchanged.

Diagnosis. The unchanged trade count is the giveaway. A fault that changes what the system does moves the trade count; a fault that changes what each trade costs does not.

How you would confirm it. Read Total commissions paid in the report. If it is zero, or a rounding error against gross profit, the costs are not set. Then divide the profit difference by the trade count: it should be close to the round-trip cost.

The fix. Set both, at figures you can point to in a broker schedule, and run the sensitivity sweep from the costs lesson.

Scale. Proportional to turnover. A 100-bar-hold system barely notices; a 5-bar-hold system is destroyed. This is why Avg. Bars Held belongs in your table.

Fault 3 — survivorship

Fragment — not a complete formula

Exclude = BarCount < RequiredBars;

Symptoms. Depends entirely on the universe. On a list containing delisted and late-listing symbols, expect fewer symbols traded, a higher win rate, and a materially better result. On a clean current-membership list, almost no change — because the bias is already baked in.

Diagnosis. Exclude is a documented reserved variable: a true value excludes the current symbol from a scan, exploration or backtest, and excluded symbols are also left out of buy-and-hold calculations.

Excluding symbols with short histories sounds like data hygiene. What it actually removes is everything that listed late and — far more damagingly — everything that stopped trading, which is precisely the population whose returns were worst.

How you would confirm it. Count the symbols that produced trades in fault 0 versus fault 3. Then list the excluded ones and look at what they are. If they are mostly companies that ceased to exist, you have your answer.

The fix. Do not filter on history length. If some symbols genuinely lack enough warm-up, handle that per bar with a BarIndex() guard rather than by removing the symbol.

The deeper problem. Even fault 0 carries survivorship if the watch list is a current-membership list. That is not fixable in the formula — it needs a point-in-time universe. Say so in the write-up and treat the result as an upper bound.

Fault 4 — repeated signals

Fragment — not a complete formula

SetBacktestMode( backtestRegularRawMulti );

Symptoms. Trade count rises sharply. Exposure rises. Diversification collapses — the account ends up holding several positions in the same symbol. Drawdown gets worse, sometimes dramatically.

Diagnosis. The documented behaviour: redundant raw entry signals are not removed, and multiple positions per symbol will be open if the entry signal is true for more than one bar and there are free funds. Sell/Cover exit all open positions on that symbol at once.

So a symbol in a sustained trend, whose entry condition stays true for twenty bars, can consume the entire book by itself.

How you would confirm it. Sort the trade list by symbol. If the same symbol appears with several simultaneously open entries, that is the mode.

The fix. backtestRegular for one position per symbol with redundant entries stripped, or backtestRegularRaw for one position per symbol with entry signals retained. RawMulti is a deliberate design choice for pyramiding systems, not a default.

Why it is not simply “wrong”. All three modes are legitimate; they model different behaviours. The fault here is using a mode whose behaviour the author did not intend and did not check.

Fault 5 — sizing error

Fragment — not a complete formula

SetOption( "MaxOpenPositions", 1 );
PositionSize = -100;
Buy = Setup AND Volume > 0;   // liquidity condition dropped

Symptoms. Net profit may go up enormously or down enormously — it is entirely path-dependent. Drawdown gets much worse. Trade count falls. And the distribution changes shape: a few trades account for nearly all of the result.

Diagnosis. Three faults in three lines. One position at a time removes all diversification. Full size means every winner compounds into the next position. Dropping the participation condition means nothing stops the requested share count exceeding the entire volume that traded on the entry bar.

How you would confirm it. Sort the trade list by share count descending and compare the top entries against the volume printed on their entry bars. The audit exploration does this systematically.

The fix. Restore the slot count, the fractional sizing and the participation cap.

The instructive part. This is the only fault whose direction is not predictable. It can look spectacular or catastrophic depending on which symbol the concentrated bet happened to land on — which is itself the argument against concentration.

Fault 6 — multi-timeframe misalignment

Fragment — not a complete formula

WeeklyHigh = TimeFrameGetPrice( "H", inWeekly );   // shift 0, expandFirst

Symptoms. Very similar to fault 1: a high win rate, a smooth equity curve, an implausibly good result. Trade count may go either way.

Diagnosis. TimeFrameGetPrice( pricefield, interval, shift = 0, mode = expandFirst ). With the defaults, the whole week’s high is written onto the week’s first bar. A Monday comparison already knows Friday’s high.

The documentation is explicit: “if shift = 0 compressed data may look into the future ( weekly high can be known on monday ). If you want to write a trading system using this function please make sure to reference PAST data by using negative shift value.”

How you would confirm it. Plot the weekly high alongside price and look at where it steps. If the level changes at the start of each week and the price then rises to meet it, the value was published before it was established. On a correct version the line steps at the start of the week to the previous week’s completed value and never moves during the week.

The fix. TimeFrameGetPrice( "H", inWeekly, -1 ).

Why it is the hardest of the six to spot. Faults 1 and 6 are the same category — reading data that had not happened — but fault 1 is visible as a positive Ref() shift, while fault 6 is invisible: an omitted argument. The code looks completely ordinary. This is why every higher-timeframe call in a trading system deserves an explicit shift, even when it is the one the default would have given you.

The diagnostic procedure

Generalised from the six, this is the order to work in on an unfamiliar backtest.

∴Research note

1. Compare trade counts. Did the fault change what the system does or what each trade costs? Unchanged count points at costs, fills or sizing. Changed count points at the rules.

2. Look at the win rate. A win rate that is high for the strategy type is the signature of look-ahead. Trend systems usually win under half their trades; an apparent 75% needs an explanation.

3. Look at the drawdown against the stop distance. A drawdown smaller than a few stops’ worth means either the stops are being filled at impossible prices or the losses are being avoided by information the system should not have.

4. Look at the trade distribution. If a handful of trades produce nearly all the profit, the result is those trades — check their sizes, their symbols and their dates.

5. Search the formula for positive Ref() shifts. Five seconds, catches an entire category.

6. Check every higher-timeframe call for an explicit shift. The dangerous version is the one with an argument missing.

7. Read Total commissions paid. Zero means no costs.

8. Count the symbols that produced trades, and ask what is not in the universe.

What the six have in common

Five of the six make the result better. Fault 5 is the exception, and only because it is path-dependent — its expected effect on a reported result is also favourable, because a concentrated bet that went badly would have been abandoned rather than published.

That asymmetry is the thing to carry away. As in the previous challenge: the faults that survive into a reported backtest are overwhelmingly the ones that flattered it, because the others prompted their author to go looking.

Check your understanding

Question 1. A fault leaves the trade count and exposure unchanged but raises net profit and slightly raises the win rate. Which category is it?

• A look-ahead in the entry rule
• Something that changes what each trade costs rather than what the system does — costs, fills or sizing
• A survivorship problem in the universe
• A backtest-mode change

Show the answer and why

Answer: Something that changes what each trade costs rather than what the system does — costs, fills or sizing

An unchanged trade count means the rules produced identical decisions. The win rate rising slightly is the signature of marginal trades crossing zero once the cost is removed. Reading Total commissions paid confirms it in seconds.

Question 2. Why is Exclude = BarCount < RequiredBars a survivorship error rather than data hygiene?

Exclude = BarCount < RequiredBars;

• Because Exclude is not a valid AFL variable
• Because it removes everything that listed late and, more damagingly, everything that stopped trading — the population whose returns were worst
• Because BarCount is not available in a backtest
• Because it excludes symbols from buy-and-hold calculations only

Show the answer and why

Answer: Because it removes everything that listed late and, more damagingly, everything that stopped trading — the population whose returns were worst

Exclude is documented: a true value removes the symbol from the scan, exploration or backtest, and from buy-and-hold calculations too. The problem is what "short history" correlates with. Warm-up should be handled per bar with a BarIndex() guard, not by deleting symbols.

Question 3. What does backtestRegularRawMulti do that backtestRegularRaw does not?

• It keeps redundant exit signals as well as entries
• It allows MULTIPLE positions per symbol to be open when the entry signal is true on more than one bar and funds are free
• It disables position ranking
• It removes redundant entry signals

Show the answer and why

Answer: It allows MULTIPLE positions per symbol to be open when the entry signal is true on more than one bar and funds are free

Both keep raw entry signals. RawMulti additionally permits several simultaneous positions in the same symbol, with Sell/Cover exiting all of them at once. It is a legitimate mode for pyramiding designs and a disaster when selected by accident — one trending symbol can consume the whole book.

Question 4. Faults 1 and 6 are the same category of error. Why is fault 6 harder to spot?

• Because TimeFrameGetPrice is undocumented
• Because fault 1 is a visible positive Ref() shift while fault 6 is an omitted argument — the code looks completely ordinary
• Because fault 6 only occurs on weekly data
• Because fault 6 does not change the results

Show the answer and why

Answer: Because fault 1 is a visible positive Ref() shift while fault 6 is an omitted argument — the code looks completely ordinary

Both read data that had not happened. One is a wrong value you can search for; the other is a missing value you cannot. This is why every higher-timeframe call in a trading system should carry an explicit shift, even when the default would have been correct.

Question 5. Which observations should make you suspect look-ahead before you read any code? Select all that apply.

• A win rate far above what the strategy type usually produces
• A maximum drawdown smaller than a few stop distances
• An unusually smooth equity curve
• A high number of trades

Show the answer and why

Answer: A win rate far above what the strategy type usually produces, A maximum drawdown smaller than a few stop distances, An unusually smooth equity curve

All three of the first are consequences of avoiding losses using information the system should not have had. Trade count on its own says nothing about look-ahead — it is a clue about which category a fault belongs to, not about which fault.

Check my answersClear