Lab: Run a Walk-Forward Analysis

The deliverable of this lab is not a number. It is one page of writing that another person could read, disagree with, and reproduce — plus the files that let them do it. The walk-forward run is about fifteen minutes of clicking and a wait; everything else here is the part that makes the run mean anything.

Budget an hour, of which perhaps half is machine time you can spend doing something else.

What you will produce

By the end you should have six artefacts:

1. A saved Analysis project (.APX) for the walk-forward run.
2. A saved Analysis project for the fixed-parameter control run.
3. An exported result list from each.
4. A chart of ~~~OSEQUITY, captured or exported before the next run overwrites it.
5. A run log: dates, geometry, target, universe, costs, and what you expected beforehand.
6. A written conclusion of five to ten sentences that states what the evidence supports and what it does not.

⚠Warning

Nothing in this lab produces a system to trade. It produces a worked example of a procedure. Every figure you obtain will be specific to your database, your universe, your date range and your cost assumptions, and none of it is an estimate of future returns.

The system under test

Goal

We need a system with exactly two optimisable parameters, enough structure to be realistic, and no cleverness that could distract from the methodology. A long-only breakout with a trend gate and a liquidity floor fits: it is well understood, it trades often enough to produce a usable number of out-of-sample trades, and both of its parameters have an obvious meaning, so parameter instability across steps is easy to read.

The complete formula

Complete runnable AFL

wf-breakout-portfolio.afl

// wf-breakout-portfolio.afl
// Part 32 - Lab: Run a Walk-Forward Analysis
//
// A long-only portfolio breakout system with exactly two optimisable
// parameters, written so that one file charts, backtests, optimises and
// walk-forward tests without a single edit.
//
// ============================ ASSUMPTIONS ==================================
// Read these before the rules. Every number this formula produces is a
// consequence of them, and none of them is measured from your broker.
//
//   Data          Daily bars, split- and dividend-adjusted, end of day.
//   Universe      A fixed watch list selected in the Analysis window. It is
//                 NOT point-in-time: symbols delisted along the way are simply
//                 absent, so results carry survivorship bias. Nothing in this
//                 file removes that.
//   Signals       Evaluated on the close of the signal bar.
//   Fills         The NEXT bar's open. SetTradeDelays( 1, 1, 1, 1 ), with
//                 BuyPrice and SellPrice built from Open.
//   Commission    0.15% of trade value, each way (CommissionMode 1).
//   Slippage      A flat SlippagePct of the open, against us on both entry and
//                 exit. It is an assumption, not a measurement, and it does not
//                 widen in a falling market the way real slippage does.
//   Liquidity     A symbol trades only if 50-day average turnover clears
//                 MinTurnover. With 100,000 of capital in at most 10 positions,
//                 a full position is about 10,000, which is roughly 0.5% of a
//                 2,000,000 average day. There is no modelling of what happens
//                 when everyone wants out at once.
//   Stops         A percentage maximum-loss stop, checked against the bar's
//                 high-low range but EXITING NEXT BAR at the open
//                 (ExitAtStop = 2). The test therefore never awards itself an
//                 intrabar fill at the exact stop price.
//   Capital       100,000 units of account currency, no margin, 10 positions
//                 maximum, 10% of equity per position.
//   Not modelled  Borrow costs, dividends as cash, taxes, corporate actions
//                 other than the adjustment already in the data, and any limit
//                 on how much of a day's volume you could really have taken.
// ===========================================================================

_SECTION_BEGIN( "Walk-forward breakout" );

// --- The two parameters a walk-forward run re-estimates at every step -------
// Optimize() returns its DEFAULT (second argument) in every mode except
// optimization, so charting and plain backtests use 55 and 20 unchanged.
// 17 x 9 = 153 combinations per optimization step: (100-20)/5+1 by (50-10)/5+1.
BreakoutPeriod = Optimize( "Breakout period", 55, 20, 100, 5 );
ExitPeriod     = Optimize( "Exit period",     20, 10,  50, 5 );

// --- Fixed by design. Every one of these is a decision, not a default -------
TrendPeriod    = 200;        // regime gate, deliberately not optimised
TurnoverPeriod = 50;
MinTurnover    = 2000000;    // in the instrument's own currency
StopPercent    = 15;
SlippagePct    = 0.05;
PositionPct    = 10;

// --- Account and cost model -------------------------------------------------
SetOption( "InitialEquity",           100000 );
SetOption( "MaxOpenPositions",        10 );
SetOption( "CommissionMode",          1 );
SetOption( "CommissionAmount",        0.15 );
SetOption( "AllowPositionShrinking",  True );
// Stops resolved AFTER regular signals, so cash freed by a stop is not
// available to open a new position on the same bar.
SetOption( "ActivateStopsImmediately", True );
SetTradeDelays( 1, 1, 1, 1 );

BuyPrice  = Open * ( 1 + SlippagePct / 100 );
SellPrice = Open * ( 1 - SlippagePct / 100 );

SetPositionSize( PositionPct, spsPercentOfEquity );

// --- Liquidity gate ----------------------------------------------------------
Turnover    = Close * Volume;
AvgTurnover = MA( Turnover, TurnoverPeriod );
Liquid      = AvgTurnover > MinTurnover;

// --- Regime gate and the breakout itself ------------------------------------
UpTrend = Close > MA( Close, TrendPeriod );

// Ref( ..., -1 ) so the level being broken is yesterday's, computed from bars
// that had already closed when the signal was evaluated. Without the shift the
// highest high of the last N bars includes today, and today's close can never
// exceed a high it is part of - the system would take no trades at all.
BreakoutLevel = Ref( HHV( High, BreakoutPeriod ), -1 );
ExitLevel     = Ref( LLV( Low,  ExitPeriod     ), -1 );

Buy  = Liquid AND UpTrend AND Close > BreakoutLevel;
Sell = Close < ExitLevel;

Buy  = ExRem( Buy,  Sell );
Sell = ExRem( Sell, Buy  );

// --- Ranking when there are more candidates than money ----------------------
// The backtester ranks on the ABSOLUTE value of PositionScore by default, so a
// bare momentum figure would rank a -60% collapse above a +10% advance. ROC
// cannot fall below -100, so the offset makes every score non-negative and the
// ordering means what it looks like it means.
PositionScore = 100 + ROC( Close, 100 );

// --- Risk control ------------------------------------------------------------
ApplyStop( stopTypeLoss, stopModePercent, StopPercent, 2 );

_SECTION_END();

Download wf-breakout-portfolio.afl104 lines

How it works

The file has five sections, in the order the backtester cares about.

Assumptions come first, in a comment block, because every number the run produces is a consequence of them. They are not decoration: the survivorship note, the slippage figure and the liquidity floor each place a boundary on what the result can be used to claim.

The two optimisable parameters are declared with Optimize() at the top, once each, as the documentation requires — each call generates its own optimisation loop, so calling one inside a condition or a loop corrupts the search space. Their defaults are 55 and 20, which is what every non-optimising mode uses, so the same file charts and backtests without edits.

The account and cost model sets initial equity, the position cap, commission, and the trade delays. SetTradeDelays( 1, 1, 1, 1 ) with BuyPrice and SellPrice built from Open means a signal computed on today’s close is filled at tomorrow’s open, moved against us by the slippage assumption. Nothing here is filled at a price the signal itself could have influenced.

The rules gate on liquidity and on a 200-bar trend, then require the close to exceed the highest high of the previous BreakoutPeriod bars — with Ref( ..., -1 ), so the level being broken was fixed before today’s bar existed. Exit is a close below the lowest low of the previous ExitPeriod bars. ExRem() reduces the repeated signals to one entry and one exit per swing.

Ranking and risk finish the file. PositionScore decides who gets the money when more symbols signal than there is capital, and ApplyStop() adds a percentage maximum-loss stop that exits on the next bar’s open rather than at an assumed intrabar price.

Key functions

Optimize( "description", default, min, max, step ) returns default in every mode except optimization, where it returns successive values from min to max inclusive. The count of values is (max - min) / step + 1, so 20 to 100 by 5 is 17 values, not 16.

ApplyStop( type, mode, amount, exitatstop, ... ) — the fourth argument is not a Boolean. 0 checks only the trade price; 1 checks the high–low range and exits intrabar at the stop level; 2 checks the high–low range but exits on the next bar at the regular trade price. This formula uses 2, which is the conservative choice: it never awards itself a fill at exactly the stop price on a bar that gapped through it.

PositionScore is ranked on its absolute value by default, which is why the formula adds 100 to a rate-of-change figure that can be negative. Without the offset, a symbol down 60 per cent would outrank one up 10 per cent.

What a plain backtest should show

Before optimising anything, run a single ordinary portfolio backtest over your whole date range with the defaults. You are checking plausibility, not performance:

=Expected result

Trades exist, and there are enough of them to work with — hundreds rather than dozens over a decade on a reasonable watch list. No more than ten positions are ever open at once. Entry dates are one bar after the bar where the close first exceeded the breakout level. Exits carry a mix of reasons, including some from the stop. Nothing is traded in a symbol whose average turnover is below the floor.

If any of those is untrue, stop and fix it. A walk-forward run on a formula that does not behave is sixteen times the wasted effort.

Checking it before you trust it

Pick one trade from the trade list and reproduce it by hand on a chart. Find the bar where the close first exceeded the previous BreakoutPeriod-bar high, confirm the entry is the following bar, and confirm the entry price is that bar’s open plus the slippage percentage. Then find the exit and check it against either the exit level or the stop.

Then run a deliberate falsification: set MinTurnover to an absurdly large number and confirm the backtest produces no trades at all. A liquidity filter that cannot be made to bite is a liquidity filter that is not connected.

Common errors

No trades at all. Usually the liquidity floor against a database whose volume field is in different units, or an initial equity too small for a ten-position portfolio. The detailed log result mode shows, bar by bar, why a candidate was not entered.

Far too many trades in one symbol. ExRem() missing or applied in the wrong order.

The optimisation runs for hours. Count first: 17 by 9 is 153 combinations per step, and each combination is a full portfolio backtest over your entire watch list.

Extension

Once the walk-forward run is done, add a third Optimize() call for StopPercent and observe two things: the run time multiplies by the number of stop values, and the in-sample results improve while the out-of-sample results are unlikely to improve as much. That is the bias from the previous lesson, made visible for the price of an afternoon.

Step 1 — Set up the database and the universe

Use daily, split- and dividend-adjusted end-of-day data. Free sources are sufficient; nothing in this lab needs a subscription, a real-time feed or the Professional edition.

Assemble a watch list of at least forty liquid symbols with history covering your whole intended span. Forty is not a statistical threshold, it is a practical one: fewer, and the portfolio rarely has ten candidates to choose between, so PositionScore never does anything and the run tells you less than it appears to.

Write down what the list is and how you built it. If you built it from symbols that are liquid today, write that down too — it is survivorship bias, it inflates results, and walk-forward analysis does not touch it.

Step 2 — Design the windows, and justify them

Do the arithmetic before you open the dialog.

Run the plain backtest from the previous step and note the number of trades and the span in years. That gives you an approximate trade rate. Now choose:

• an in-sample length long enough to contain enough trades that 153 parameter combinations can be distinguished from one another — a few hundred trades is a reasonable aim, and if your rate makes that impossible, say so rather than pretending;
• a step equal to how often you would genuinely re-optimise in practice, which for a system holding positions for weeks is usually a year;
• anchored or rolling, decided on the assumption you are prepared to defend, not on which produces a better result.

A concrete worked design, for a database spanning 2004 to 2024:

The lab design: rolling, four-year in-sample, one-year step

Step 1

IS 2004-2007OOS 2008

Step 2

IS 2005-2008OOS 2009

Step 3

IS 2006-2009OOS 2010

…

and so on to 2024

2004201020162024

Twenty years of history, a four-year training window and a one-year step gives sixteen out-of-sample years.

The justification, written out, might read: rolling rather than anchored, because the liquidity and volatility characteristics of the universe changed materially over the span and I am not prepared to argue that 2004 should govern a 2023 parameter choice; four years, because it is the shortest window that contains enough trades for the two parameters to be estimated rather than guessed; a one-year step, because annual re-optimisation is a schedule I would actually keep.

That paragraph is worth more than the result. Write yours before you run.

▷Exercise

Compute your own step count now: ( span_years - in_sample_years ) / step_years. Then compute the run cost: steps multiplied by 153 combinations multiplied by the number of symbols. Write both numbers down and compare them with what actually happens.

Step 3 — Configure the Walk-Forward tab

Open an Analysis window, load the formula, then press Settings and go to the Walk-Forward tab.

1. Select Easy mode (EOD). This is daily data, and Easy mode derives the out-of-sample dates from the in-sample ones so that the geometry cannot be invalid.
2. Set the in-sample Start and End to the first training window from your design.
3. Set Last to the final date you want the walk to reach.
4. Leave Use today unticked. This run has to be reproducible.
5. Untick Anchored for a rolling window, or tick it for an anchored one — whichever your written justification says.
6. Set Step to 1 year.
7. Set Optimization target to CAR/MDD unless your justification says otherwise. It is the default, and it prefers a smoother equity path to a larger endpoint.
8. Read every row of the Preview list. Confirm the number of OOS rows equals your computed step count, that each OOS row starts where its IS row ends, and that no IS row overlaps its own OOS row.
9. Use the tab’s Save button to store the settings, then save the whole Analysis project as an .APX file. The Batch documentation describes an .APX as self-contained: formula, options, settings, apply-to and range selections.

Also confirm on the other tabs that initial equity, commission and trade delays match the formula’s assumptions. Where the formula sets an option with SetOption(), the formula wins, but a dialog that disagrees with the code is a trap for the next person — including you in three months.

Step 4 — Run it

Click the drop-down arrow on the Optimize button and choose Walk-Forward.

When the progress dialog appears, press MINIMIZE on it. That lets you watch results accumulate in the Walk Forward tab at the bottom of the Analysis window, step by step. If the first two steps look obviously wrong — no trades, absurd figures, parameters pinned to a range boundary — stop and diagnose rather than waiting for all sixteen.

Expect this to take a while. The run is a sequence of complete optimizations, and the Standard edition gives you 2 threads per Analysis window against the Professional edition’s 32, so the same run can differ by a large factor between machines and editions.

Step 5 — Run the control

This step is not optional, and it is the one most people skip.

Complete runnable AFL

wf-fixed-baseline.afl

// wf-fixed-baseline.afl
// Part 32 - Lab: Run a Walk-Forward Analysis (the control)
//
// This is wf-breakout-portfolio.afl with the two Optimize() calls replaced by
// constants, and NOTHING else changed. It is the control the walk-forward
// result has to beat.
//
// Why it exists: a walk-forward run tells you what the re-optimisation
// PROCEDURE produced. On its own that number is uninterpretable, because you
// do not know what the same rules would have produced with no re-optimisation
// at all. Run this over the same span, the same watch list and the same
// settings, and you have something to compare against.
//
// The parameter values below are the mid-points of the two optimisation
// ranges, chosen before either test was run so that the comparison is not
// itself a selection. Do not "improve" them after seeing a result: the moment
// you tune the control, it stops being a control.
//
// Every assumption from wf-breakout-portfolio.afl applies here unchanged:
// daily adjusted bars, a non-point-in-time watch list carrying survivorship
// bias, signals on the close, fills at the next open with 0.05% slippage each
// way, 0.15% commission each way, a 15% maximum-loss stop exiting next bar at
// the open, 100,000 of capital, 10 positions of 10% of equity, no margin.

_SECTION_BEGIN( "Fixed-parameter baseline" );

// --- Fixed in advance, at the mid-point of the optimisation ranges ----------
BreakoutPeriod = 60;         // range was 20..100 step 5
ExitPeriod     = 30;         // range was 10..50  step 5

TrendPeriod    = 200;
TurnoverPeriod = 50;
MinTurnover    = 2000000;
StopPercent    = 15;
SlippagePct    = 0.05;
PositionPct    = 10;

SetOption( "InitialEquity",            100000 );
SetOption( "MaxOpenPositions",         10 );
SetOption( "CommissionMode",           1 );
SetOption( "CommissionAmount",         0.15 );
SetOption( "AllowPositionShrinking",   True );
SetOption( "ActivateStopsImmediately", True );
SetTradeDelays( 1, 1, 1, 1 );

BuyPrice  = Open * ( 1 + SlippagePct / 100 );
SellPrice = Open * ( 1 - SlippagePct / 100 );

SetPositionSize( PositionPct, spsPercentOfEquity );

Turnover    = Close * Volume;
AvgTurnover = MA( Turnover, TurnoverPeriod );
Liquid      = AvgTurnover > MinTurnover;

UpTrend = Close > MA( Close, TrendPeriod );

BreakoutLevel = Ref( HHV( High, BreakoutPeriod ), -1 );
ExitLevel     = Ref( LLV( Low,  ExitPeriod     ), -1 );

Buy  = Liquid AND UpTrend AND Close > BreakoutLevel;
Sell = Close < ExitLevel;

Buy  = ExRem( Buy,  Sell );
Sell = ExRem( Sell, Buy  );

PositionScore = 100 + ROC( Close, 100 );

ApplyStop( stopTypeLoss, stopModePercent, StopPercent, 2 );

_SECTION_END();

Download wf-fixed-baseline.afl70 lines

Load it into a second Analysis window, set the range to cover exactly the span of your out-of-sample segments — from the start of the first OOS row to the end of the last — and run a plain Backtest. Same universe, same settings, same costs. Save this as its own .APX.

You now have the comparison that makes the walk-forward number interpretable: what the same rules did over the same period with no re-optimisation at all.

∴Research note

The control’s parameters were fixed at the mid-points of the optimisation ranges before either run. That ordering matters. If you choose the control’s parameters after seeing the walk-forward result, the control is no longer a control — it is a second search, and a rigged one.

Step 6 — Collect the results

Four things to gather, in this order.

Per step, from the Walk Forward tab: the in-sample value of your target, the out-of-sample value of the same metric, the number of out-of-sample trades, and the parameter values chosen. Copy them into a table. With a New Analysis window active, File → Export HTML/CSV writes the result list out.

The aggregate, from the out-of-sample summary report: open the Report Explorer from the drop-down arrow on the Report button and take the last entry, of type PS. Record Net Profit %, Annual Return %, Max. system % drawdown, CAR/MaxDD, the number of trades and Exposure %. Use AmiBroker’s labels verbatim, so that nobody has to guess which statistic you meant.

The control’s report, the same fields, from the plain backtest.

The composite equity, charted before you run anything else — ~~~ISEQUITY and ~~~OSEQUITY are overwritten by the next walk-forward run.

Step 7 — Read the out-of-sample equity

Apply the plotting formula from the previous lesson to any symbol whose history spans the run.

Four questions, in this order, and none of them is about the endpoint:

Where did it go flat, and for how long? A twenty-month flat stretch is twenty months of paying costs and watching. Measure it in months and ask whether you would have kept going.

Is the shape driven by one stretch? Cover the best-looking twelve months with your hand. If what remains is a horizontal line, the result is one market episode rather than sixteen observations.

What is the worst drawdown, and where? Compare it with Max. system % drawdown in the summary report; they should tell the same story. If they do not, you are looking at the wrong composite or a stale one.

How does it compare with the control? Both curves over the same span is the single most informative picture this lab produces.

!Important

The composite is concatenated and scaled to keep the equity line continuous, and the User’s Guide states that this assumes you are generally compounding profits. Read the shape; do not quote a precise terminal value from it as if it were a measured result. The summary report is the place for numbers.

Step 8 — Write the conclusion

Five to ten sentences, covering: what was tested, over what universe and period, under what assumptions; the geometry and the target; the out-of-sample outcome in AmiBroker’s own metric names; the comparison against the control; the number of out-of-sample trades; and what the result does not license you to claim.

Here is the shape, with illustrative figures that are not results from any run — they exist to show the form of the sentence, and you must replace every one of them:

∴Research note

Illustrative example, not a measured result. Tested a long-only breakout with a 200-bar trend gate on a 45-symbol watch list of liquid shares, daily adjusted bars, 2004-01-01 to 2024-12-31, 0.15% commission and 0.05% slippage each way, ten positions of 10% of equity, survivorship bias present because the list was built from current constituents. Rolling walk-forward, four-year in-sample window, one-year step, sixteen steps, target CAR/MDD. Illustrative out-of-sample outcome: CAR/MaxDD of about 0.6 against an in-sample median of about 1.9, Max. system % drawdown around −34%, 212 out-of-sample trades, positive in nine of sixteen steps. The fixed-parameter control over the same span produced a similar CAR/MaxDD, so the annual re-optimisation added nothing detectable. Conclusion: the rules were not falsified, the optimisation step was not justified, and the result says nothing about future returns; the next test would be the same rules, fixed parameters, on a universe I have not touched.

Notice what that paragraph does not do. It does not round anything up, it does not describe the result as validation, it does not quote a figure without its universe and assumptions, and it reports the control even though the control was unflattering to the exercise.

Verification checklist

Work through this before you file the run. Any “no” invalidates the conclusion until fixed.

1. Does the Preview list’s OOS row count equal the step count you computed by hand?
2. Does each OOS segment begin exactly where its IS segment ends, on the EOD convention?
3. Is Use today unticked, and is Last the date you intended?
4. Does the Optimization target in the dialog match the target named in your run log?
5. Did you record whether Anchored was ticked?
6. Do initial equity, commission and trade delays in the settings agree with the formula’s assumption block?
7. Did you hand-verify at least one trade from entry signal to fill price?
8. Does the liquidity filter actually bite when you set the floor absurdly high?
9. How many out-of-sample trades are there in total? Is that enough for you to have changed your mind either way?
10. Did any single step produce most of the out-of-sample result?
11. Did the chosen parameters oscillate between the extremes of the range, or sit pinned at a boundary?
12. Did you run the fixed-parameter control over exactly the out-of-sample span?
13. Did you chart ~~~OSEQUITY before starting any further run?
14. Are both .APX files and both exports saved somewhere you will find them?
15. Does your written conclusion state, in one sentence, what the result does not support?

What changed

You have now run the full procedure once, which is enough to discover that the run itself is the easy part. The parts that took effort — writing the justification before the run, building the control, hand-checking a trade, and stating the conclusion in a form somebody could attack — are the parts that make the difference between a result and a screenshot.

Keep the run log. The next part resamples what a backtest produced, and it will ask you for exactly the same discipline about what the simulation can and cannot know.

Check your understanding

Question 1. Your walk-forward run finished and you want the aggregate out-of-sample figures. Where do you look?

• The last row of the Walk Forward tab
• The out-of-sample summary report in the Report Explorer, of type PS
• The optimisation result list, sorted by CAR/MDD
• The terminal value of the ~~~OSEQUITY composite

Show the answer and why

Answer: The out-of-sample summary report in the Report Explorer, of type PS

The summary report covers all out-of-sample steps with metrics computed properly across them. The Walk Forward tab gives per-step rows, and the composite is a scaled curve for reading shape rather than quoting values.

Question 2. Why does the lab insist on running the fixed-parameter control over exactly the out-of-sample span, using parameters chosen in advance?

• Because AmiBroker requires a baseline before it will generate a summary report
• To give the walk-forward result a denominator: what the same rules did with no re-optimisation
• To find better parameters than the optimisation found
• Because a plain backtest is faster and can replace the walk-forward run

Show the answer and why

Answer: To give the walk-forward result a denominator: what the same rules did with no re-optimisation

A walk-forward number on its own cannot be interpreted. The control answers "compared with what?". Choosing its parameters after seeing the result would turn the control into a second search.

Question 3. The optimisation declares Optimize( "Breakout period", 55, 20, 100, 5 ). How many values does that parameter take during an optimisation, and what does it return during a plain backtest?

BreakoutPeriod = Optimize( "Breakout period", 55, 20, 100, 5 );

• 16 values; returns 20 in a plain backtest
• 17 values; returns 55 in a plain backtest
• 16 values; returns 55 in a plain backtest
• 17 values; returns 100 in a plain backtest

Show the answer and why

Answer: 17 values; returns 55 in a plain backtest

The range is inclusive, so the count is (100 − 20) / 5 + 1 = 17. Outside optimization, Optimize() hands back its second argument, which is why the same file charts and backtests unchanged.

Question 4. Which of these findings would make your written conclusion weaker, even if the out-of-sample composite looked good? Select all that apply.

• Twelve of the sixteen steps were flat or negative
• The universe was built from symbols that are liquid today
• Total out-of-sample trades numbered 38
• The chosen breakout period was 45, 50, 45 and 55 across four consecutive steps

Show the answer and why

Answer: Twelve of the sixteen steps were flat or negative, The universe was built from symbols that are liquid today, Total out-of-sample trades numbered 38

One dominant step, a survivorship-biased universe and a tiny trade count each undermine the composite. Parameters settling in a neighbourhood is the encouraging pattern, not a warning.

Check my answersClear