Lab: Optimise Without Fooling Yourself

The deliverable of this lab is not a number. It is a written parameter decision that somebody sceptical could read, disagree with, and re-run. You will search a two-parameter grid, lay it out as a surface, identify a region rather than a peak, measure how stable that region is, lock the choice into a formula by hand, and finish by writing down everything the exercise has not established — which, at the end of Part 31, is nearly everything that matters.

Budget about an hour. Most of it is reading and writing; the machine time is small.

What you will produce

Four things, and the lab is not complete without all four:

1. An optimization result list, with the universe, period, costs and grid recorded alongside it.
2. A surface — a grid of your chosen metric with the two parameters on the axes.
3. A neighbourhood sensitivity run around your candidate, with its spread recorded.
4. A decision record naming the region, the chosen value, the evidence, and the outstanding tests.

The lab, end to end

1. 1Question and acceptance ruleWritten down before anything runs
2. 2SetupFormula, universe, period, costs, delays, settings
3. 3Count the gridOn paper, before pressing anything
4. 4OptimizeOne exhaustive run
5. 5Build the surfaceGrid, not ranking
6. 6Choose a regionContiguous, interior, adequately traded
7. 7Stress itNeighbourhood, costs, both halves of the period
8. 8Lock and documentHand-edited formula plus a decision record

Before you start

Check each of these. A missing one does not make the lab harder — it makes the result meaningless in a way that is hard to notice afterwards.

• A working portfolio backtest from Part 28, with costs, delays and a liquidity filter in place.
• A universe you can name: a watch list, not “all symbols” in a database whose contents you have not audited.
• Enough history that the shortest and longest parameter settings both have room to trade.
• Part 29’s vocabulary to hand, because from here on metrics are named as AmiBroker names them.

⚠Warning

Nothing in this lab produces evidence that a strategy is worth trading. Every run happens on the same history the parameters are chosen from. A parameter choice made this way and then traded is the exact error the whole part is written to prevent; Parts 32 and 33 are the prerequisites for that step, and even they do not turn an in-sample choice into an expectation of future returns.

Step 1 — Write the question and the acceptance rule first

Two sentences, before AmiBroker is open, in whatever file you keep research notes in.

The question. Not “what are the best moving-average periods” — that question presupposes the answer is a pair of numbers. Better: does this two-average rule have a broad region of parameter values that behave alike on this universe and period, or only isolated good cells? That question has an answer that could be “no”, which is what makes it worth asking.

The acceptance rule. What would you have to see to call a region acceptable, stated before you see anything? Write it in this shape:

Pseudocode — not valid AFL

a region is acceptable if:
    it is a contiguous block of at least nine cells
    every cell in it clears the metric threshold I set now
    it does not touch any edge of the grid I searched
    every cell in it has at least the minimum trade count I set now

Then fill in the two numbers. There are no universal correct values; the point is that you chose them in advance and can be held to them. Writing “the threshold will be whatever the region I like turns out to clear” is the failure mode this step exists to block.

Step 2 — Set up the run

The formula

Complete runnable AFL

optimizable-trend-system.afl

// optimizable-trend-system.afl
// Part 31 - Optimization
//
// A deliberately plain two-parameter trend system, written so that the SAME
// file runs as a chart, a scan, a backtest and an optimization without being
// edited. Its job in this part is not to be a good system. Its job is to give
// us a parameter space whose shape we can look at.
//
// ASSUMPTIONS - every number this file produces inherits all of them:
//   - Daily bars. Portfolio backtest: one account, cash only, no margin.
//   - Signals are computed from the close of the signal bar. Every entry and
//     exit is delayed one bar and filled at the NEXT bar's open, so nothing is
//     ever filled at a price that was not yet known when the signal formed.
//   - Costs: commission is charged as a percentage of trade value, per side,
//     and is set from the formula so that it is recorded with the run. The
//     figure below is a stand-in for YOUR broker's commission plus an
//     allowance for the spread you would have crossed. Raise it, do not
//     lower it.
//   - Liquidity: a symbol may not be bought unless its average turnover over
//     the past 50 bars clears the floor below. Without this filter the
//     optimizer finds its best cells in symbols you could never have
//     transacted in, and the surface you read is largely fiction.
//   - No survivorship correction is applied. If the database contains only
//     companies that still exist, every result is optimistic by an amount
//     this file cannot measure.
//
// Both optimized variables are declared exactly ONCE, at the top, with unique
// description strings. The User's Guide requires both: each call generates its
// own optimization loops, and duplicated description strings produce garbage.

_SECTION_BEGIN( "Optimizable Trend System" );

// --- The parameter space -------------------------------------------------
// Optimize( "description", default, min, max, step ) hands back "default" in
// every mode except Optimization. The two numbers below are therefore also the
// parameters this file trades with when you press Backtest, Scan or Explore.
FastPeriod = Optimize( "Fast MA", 20, 5, 60, 5 );
SlowPeriod = Optimize( "Slow MA", 120, 60, 260, 10 );

// Combinations where the "fast" average is not actually faster are not a
// system. Exclude suppresses their statistics so they do not clutter the
// result list or waste report time. It does not reduce the number of
// combinations the optimizer enumerates.
Exclude = FastPeriod >= SlowPeriod;

// --- Fixed assumptions ---------------------------------------------------
StartingEquity = 100000;
MaxPositions   = 10;
CommissionPct  = 0.30;      // percent of trade value, charged per side
MinTurnover    = 2000000;   // in the currency of the database
TurnoverPeriod = 50;

SetOption( "InitialEquity",    StartingEquity );
SetOption( "MaxOpenPositions", MaxPositions );
SetOption( "CommissionMode",   1 );            // 1 = percent of trade value
SetOption( "CommissionAmount", CommissionPct );
SetOption( "AllowPositionShrinking", True );

// Ten slots, so each entry asks for a tenth of current portfolio equity.
SetPositionSize( 100 / MaxPositions, spsPercentOfEquity );

// Signals form on the close; trades happen on the following open.
SetTradeDelays( 1, 1, 1, 1 );
BuyPrice  = Open;
SellPrice = Open;

// --- Liquidity filter ----------------------------------------------------
// MA() is Null until it has enough bars, and any comparison with Null is Null,
// so the filter is written to be explicitly false during warm-up rather than
// silently undefined.
Turnover     = Close * Volume;
AvgTurnover  = MA( Turnover, TurnoverPeriod );
LiquidEnough = NOT IsNull( AvgTurnover ) AND AvgTurnover >= MinTurnover;

// --- The rules -----------------------------------------------------------
FastAvg = MA( Close, FastPeriod );
SlowAvg = MA( Close, SlowPeriod );

Buy  = Cross( FastAvg, SlowAvg ) AND LiquidEnough;
Sell = Cross( SlowAvg, FastAvg );

// When more symbols signal on one bar than there is capital for, the
// backtester needs a rule for choosing between them. Ranking by how far price
// sits above its slow average is one choice among many, and it is part of the
// system being optimized - not a neutral detail.
PositionScore = 100 * ( Close - SlowAvg ) / SlowAvg;

// --- Chart output, so the same file can also be read visually -------------
Plot( Close,   "Close",                         colorDefault, styleCandle );
Plot( FastAvg, "Fast MA(" + FastPeriod + ")",   colorBlue,    styleLine );
Plot( SlowAvg, "Slow MA(" + SlowPeriod + ")",   colorRed,     styleLine );

_SECTION_END();

Download optimizable-trend-system.afl93 lines

Two Optimize() calls, each appearing exactly once at the top with a distinct description string, and an Exclude line that suppresses statistics for every combination where the fast average is not actually faster. Everything else — equity, position limit, commission, sizing, delays, fill prices, liquidity floor — is set from the formula rather than the Settings dialog, so the run travels with the file.

The settings checklist

Confirm each of these before the first run, and copy the values into your notes.

Setting	Where	What to confirm
Apply to	Analysis window	Your named watch list, via Filter — not “All symbols” by accident
Range	Analysis window	From-To dates, chosen in the next step
Initial equity	Formula, SetOption( "InitialEquity", … )	Matches what your notes say
Max. open positions	Formula, SetOption( "MaxOpenPositions", … )	Matches your sizing
Commission	Formula, SetOption( "CommissionMode", 1 ) and "CommissionAmount"	Percentage of trade value, per side, and pessimistic
Trade delays	Formula, SetTradeDelays( 1, 1, 1, 1 )	One bar on every array
Trade prices	Formula, BuyPrice/SellPrice	The next bar’s open
Periodicity	Settings → General	Daily, matching your data
Warn on long optimizations	Settings → Report tab	Left switched on

The range and the warm-up

Set Range to From-To dates, and start the “From” date after the longest warm-up in your grid. The slow average runs up to 260 bars, so a cell using it produces no signal for the first 260 bars of data. If your range begins at the first bar in the database, cells with short slow averages are judged on a longer period than cells with long ones, and the surface you build is partly a map of that difference.

Pick the end date deliberately too, and write both down. “All quotations” changes meaning every time you update the database, which quietly makes two runs incomparable.

Step 3 — Count the grid before you run it

On paper, before pressing anything:

• Fast: 5 to 60 in steps of 5. That is (60 − 5)/5 + 1 = 12 values.
• Slow: 60 to 260 in steps of 10. That is (260 − 60)/10 + 1 = 21 values.
• Total: 12 × 21 = 252 combinations, each a full backtest of the whole watch list.

Write the number down. It is not administrative trivia: it is the size of the selection process you are about to run, and in the next lesson-length step it is what tells you how sceptical to be about the winner. Two hundred and fifty-two draws is enough for a comfortably impressive maximum to appear out of noise alone.

▷Exercise

Before you continue, work out what happens if you add a third parameter with ten values. Then work out how long that run would take at the speed the 252-cell run turns out to take. Do the arithmetic even though you are not going to run it — the habit of estimating before launching is worth more than any single result in this lab.

Step 4 — Run the optimization

Send the formula to a New Analysis window, set Apply to and Range, then press Optimize. The drop-down arrow on that button also offers Walk-Forward, the 3D Optimization chart and Individual Optimize; you want the plain command.

While it runs, note two things you will need later. The Info tab reports per-phase timings. And backtest reports are disabled during optimization by default, which is why you get a result list rather than 252 HTML reports — a default worth keeping.

When it finishes, the list arrives sorted by Net % profit. Resist reading the top row. Before anything else, click the All trades column header and look at the extremes: how many combinations produced very few trades, and are any of them near the top of the profit ranking? That single check disposes of more spurious optima than anything else in this lab.

Step 5 — Build the surface

Copy the result list into a spreadsheet and pivot it: fast period down the rows, slow period across the columns, your chosen metric in the cells. Twelve rows by twenty-one columns.

Do it twice. Once with the metric you are judging on, and once with All trades. The second grid is a mask: any region of the first grid where the trade count collapses is a region you cannot interpret, however good it looks.

If you would rather look at it in three dimensions, remember the rule: sort the result list by the column you want plotted — a blue arrow appears in the header — and only then open the 3D Optimization chart from the Optimize button’s drop-down. Without that, you are looking at Net profit. Use Page Up and Page Down to raise the water level until roughly a tenth of the surface remains above it, and note whether what remains is one shape or several.

Step 6 — Choose a region

Apply the acceptance rule you wrote in Step 1, in this order:

1. Threshold the metric grid. Mark every cell at or above your threshold.
2. Mask by trade count. Unmark every cell below your minimum.
3. Find contiguous blocks of marked cells. Diagonal-only contact does not count.
4. Discard blocks touching an edge of the grid. An optimum on the boundary means the surface was still moving when you stopped looking; widen that range and re-run rather than reasoning about it.
5. Take the largest surviving block and read off its bounds in both parameters.
6. Take the geometric centre of that block, rounded to a value on your grid, and prefer a round number where the block contains one.

=Expected result

A common and perfectly respectable outcome is that no block survives step 4, or that the only surviving block has three cells in it. That is a result: on this universe, over this period, with these costs, the rule does not have a stable region. Record it and stop. The lab is complete — a negative finding obtained cheaply is exactly what a research process is for, and it costs you nothing but the hour.

Step 7 — Test the neighbourhood

Take the centre you chose and measure the spread around it.

Complete runnable AFL

neighbourhood-sensitivity.afl

// neighbourhood-sensitivity.afl
// Part 31 - Optimization
//
// The same trend rules as optimizable-trend-system.afl, restricted to the
// immediate neighbourhood of ONE candidate parameter set.
//
// WHY THIS IS A SEPARATE FILE
// After a full optimization you have a candidate cell. The question that
// decides whether you may use it is not "how good is this cell" - you already
// know it is the best one, which is why you are looking at it - but "how good
// are the cells around it". This file answers that in 25 backtests, and it
// reports OFFSETS rather than absolute periods, so the optimization result
// list reads as a map of the neighbourhood with the candidate at (0, 0).
//
// HOW TO USE IT
//   1. Set FastCentre and SlowCentre to your candidate.
//   2. Set FastStepSize and SlowStepSize to the step sizes you optimized with,
//      so one offset unit here means one cell of the original surface.
//   3. Run Optimize. You get 5 x 5 = 25 rows.
//   4. Sort by the column you are judging on and read the SPREAD across all
//      25 rows: the worst, the median and the best. The top row on its own
//      tells you nothing you did not already know.
//
// The offset ranges are written as literal numbers rather than computed from a
// "reach" variable, so that what the optimizer will enumerate is legible at the
// point where it is declared. Widen them by editing the two lines.
//
// ASSUMPTIONS - identical to optimizable-trend-system.afl, and repeated here
// on purpose, because a file that produces numbers must carry its own
// assumptions: the file is what gets copied, not the lesson.
//   - Daily bars, portfolio backtest, cash account, no margin.
//   - One-bar delay on every signal; fills at the next bar's open.
//   - Commission as a percentage of trade value per side, set from the formula.
//   - A turnover floor applied before any entry is allowed.
//   - No survivorship correction.

_SECTION_BEGIN( "Neighbourhood Sensitivity" );

// --- The candidate under examination -------------------------------------
FastCentre   = 20;    // the fast period you are considering trading
SlowCentre   = 120;   // the slow period you are considering trading
FastStepSize = 5;     // the step used in the original optimization
SlowStepSize = 10;    // the step used in the original optimization

// --- The neighbourhood ---------------------------------------------------
// Two steps either way, in both dimensions. Offset 0 is the candidate itself.
FastOffset = Optimize( "Fast offset (steps)", 0, -2, 2, 1 );
SlowOffset = Optimize( "Slow offset (steps)", 0, -2, 2, 1 );

// Max() clamps rather than trusting the centre and the reach to be sensible
// together: a moving-average period below 2 is not a moving average, and the
// formula still runs for combinations that Exclude will later suppress.
FastPeriod = Max( 2, FastCentre + FastOffset * FastStepSize );
SlowPeriod = Max( 3, SlowCentre + SlowOffset * SlowStepSize );

Exclude = FastPeriod >= SlowPeriod;

// --- Fixed assumptions ---------------------------------------------------
StartingEquity = 100000;
MaxPositions   = 10;
CommissionPct  = 0.30;      // percent of trade value, charged per side
MinTurnover    = 2000000;   // in the currency of the database
TurnoverPeriod = 50;

SetOption( "InitialEquity",    StartingEquity );
SetOption( "MaxOpenPositions", MaxPositions );
SetOption( "CommissionMode",   1 );
SetOption( "CommissionAmount", CommissionPct );
SetOption( "AllowPositionShrinking", True );

SetPositionSize( 100 / MaxPositions, spsPercentOfEquity );

SetTradeDelays( 1, 1, 1, 1 );
BuyPrice  = Open;
SellPrice = Open;

// --- Liquidity filter ----------------------------------------------------
Turnover     = Close * Volume;
AvgTurnover  = MA( Turnover, TurnoverPeriod );
LiquidEnough = NOT IsNull( AvgTurnover ) AND AvgTurnover >= MinTurnover;

// --- The rules, unchanged ------------------------------------------------
FastAvg = MA( Close, FastPeriod );
SlowAvg = MA( Close, SlowPeriod );

Buy  = Cross( FastAvg, SlowAvg ) AND LiquidEnough;
Sell = Cross( SlowAvg, FastAvg );

PositionScore = 100 * ( Close - SlowAvg ) / SlowAvg;

_SECTION_END();

Download neighbourhood-sensitivity.afl91 lines

Set FastCentre and SlowCentre to your chosen values and FastStepSize and SlowStepSize to the steps of your grid — 5 and 10 here. Run Optimize again: 25 rows, with the two offset columns running from −2 to +2.

First, verify the run. Find the row where both offsets are zero and check that its figures match the same cell from Step 4 exactly. If they differ, one of the two runs used a different range, a different watch list, or settings that changed in between, and the discrepancy is more important than anything else on the screen.

Then record four numbers from the 25 rows, in your notes:

• the worst value of your metric,
• the median,
• the best,
• and the value at offset (0, 0) — your candidate.

The candidate close to the neighbourhood median is what you want. A candidate near the top of its own neighbourhood is a warning that step 6 recovered a spike rather than a region, most often because the threshold was set generously.

Step 8 — Test the cost assumption

Add one line to the neighbourhood formula, replacing the fixed commission:

Fragment — not a complete formula

CommissionPct = Optimize( "Commission %", 0.30, 0.20, 0.60, 0.10 );
SetOption( "CommissionMode",   1 );
SetOption( "CommissionAmount", CommissionPct );

Five cost levels across the 25-cell neighbourhood is 125 runs — still a short job. Two things to read from it.

Does the region survive? If your metric falls below the threshold everywhere once costs double, the result was being paid for by an assumption rather than by the rules.

Does the region move? Regions that drift toward longer holding periods as costs rise are behaving exactly as they should, since turnover is what costs punish. Regions that jump somewhere unrelated are telling you the surface is noise-dominated.

Step 9 — Look at both halves of the period

Split your date range in two and repeat the full 252-cell optimization on each half. Build both surfaces and put them side by side.

The question is not whether the same cell wins — it will not, and that would prove nothing if it did. The question is whether the regions overlap: is there a block of parameter values acceptable in both halves?

!Important

This is a diagnostic, not an out-of-sample test. You have already looked at all of this data, and you will be tempted to choose the region that works in both halves — which is a choice made with knowledge of both halves, and therefore in-sample. It is worth doing because a region that appears in one half and vanishes in the other is disqualifying, and finding that out now is cheap. Part 32’s holdout and walk-forward procedures are the real test, and they require data you have not looked at.

Step 10 — Lock the choice

AmiBroker does not write optimum values back into your formula. You type them in — and the fact that it is manual is what makes it a decision rather than a side effect.

Complete runnable AFL

chosen-parameters.afl

// chosen-parameters.afl
// Part 31 - Optimization, Lab
//
// The end product of the lab: the same system with the chosen parameters
// written in as ordinary constants, and the decision that produced them
// recorded in the file rather than in somebody's memory.
//
// AmiBroker never writes optimum values back into a formula. After an
// optimization you type them in yourself. This file exists so that the typing
// happens exactly once, into a file that also says why those numbers and not
// their neighbours.
//
// =====================================================================
// DECISION RECORD
// ---------------------------------------------------------------------
// Every field below records a CHOICE, not a result. There is no performance
// figure anywhere in this file, and none belongs here: a number copied out of
// an in-sample optimization and pasted into a formula header becomes, within a
// week, something you half-remember as an expectation. Replace every line with
// the details of your own run before you rely on this file for anything.
//
//   Question ......... Does this two-average rule have a broad region of
//                      parameters that behave alike, or only one good cell?
//   Universe ......... The liquidity-filtered watch list built in Part 12.
//                      Turnover floor: 2,000,000 per day averaged over 50 bars.
//   Period ........... 1 January 2010 to 31 December 2024, daily bars.
//   Data source ...... End-of-day data imported as described in Part 2.
//                      Known defect: no delisted symbols, so survivorship
//                      bias is present and its size is unmeasured.
//   Costs assumed .... 0.30% of trade value per side, covering commission and
//                      an allowance for the spread crossed.
//   Fills assumed .... One-bar delay, filled at the next bar's open.
//   Grid searched .... Fast 5..60 step 5 (12 values);
//                      Slow 60..260 step 10 (21 values); 252 combinations,
//                      of which the Exclude rule suppressed the ones where
//                      fast >= slow.
//   Judged on ........ CAR/MDD, set in Settings -> Walk-Forward tab ->
//                      Optimization target.
//   Region chosen .... Fast 15 to 30, Slow 100 to 140 - a contiguous block
//                      whose cells were all acceptable, not touching any edge
//                      of the searched grid.
//   Centre chosen .... Fast 20, Slow 120. The centre of the region, not its
//                      best cell. The best cell was rejected precisely because
//                      it was the best cell of 252 tries.
//   Sensitivity ...... 25-cell neighbourhood run with
//                      neighbourhood-sensitivity.afl; the spread of the target
//                      across those cells is recorded in the research log,
//                      not here.
//   Still untested ... Out-of-sample holdout (Part 32), walk-forward
//                      (Part 32), sequence risk (Part 33), behaviour across
//                      market regimes (Part 30), participation limits against
//                      real daily turnover, and every execution assumption
//                      listed above.
//
//   This file is a parameter choice with its reasoning attached. It is not
//   evidence that the system works, and nothing in this part produces such
//   evidence.
// =====================================================================

_SECTION_BEGIN( "Chosen Parameter Set" );

// --- The decision, as code -----------------------------------------------
FastPeriod = 20;
SlowPeriod = 120;

// --- Fixed assumptions, unchanged from the optimization run --------------
StartingEquity = 100000;
MaxPositions   = 10;
CommissionPct  = 0.30;
MinTurnover    = 2000000;
TurnoverPeriod = 50;

SetOption( "InitialEquity",    StartingEquity );
SetOption( "MaxOpenPositions", MaxPositions );
SetOption( "CommissionMode",   1 );
SetOption( "CommissionAmount", CommissionPct );
SetOption( "AllowPositionShrinking", True );

SetPositionSize( 100 / MaxPositions, spsPercentOfEquity );

SetTradeDelays( 1, 1, 1, 1 );
BuyPrice  = Open;
SellPrice = Open;

// --- Liquidity filter ----------------------------------------------------
Turnover     = Close * Volume;
AvgTurnover  = MA( Turnover, TurnoverPeriod );
LiquidEnough = NOT IsNull( AvgTurnover ) AND AvgTurnover >= MinTurnover;

// --- The rules, unchanged ------------------------------------------------
FastAvg = MA( Close, FastPeriod );
SlowAvg = MA( Close, SlowPeriod );

Buy  = Cross( FastAvg, SlowAvg ) AND LiquidEnough;
Sell = Cross( SlowAvg, FastAvg );

PositionScore = 100 * ( Close - SlowAvg ) / SlowAvg;

// --- Chart output --------------------------------------------------------
// The title states which parameter set produced what you are looking at. A
// backtest report and a chart that disagree about their parameters is one of
// the easiest mistakes to make and one of the hardest to notice.
Plot( Close,   "Close",                       colorDefault, styleCandle );
Plot( FastAvg, "Fast MA(" + FastPeriod + ")", colorBlue,    styleLine );
Plot( SlowAvg, "Slow MA(" + SlowPeriod + ")", colorRed,     styleLine );

Title = Name() + "  |  chosen set: fast " + FastPeriod + " / slow " + SlowPeriod
      + "  |  commission " + CommissionPct + "% per side  |  next-open fills";

_SECTION_END();

Download chosen-parameters.afl110 lines

The file carries the parameter values as ordinary constants, a decision record in its header, and a Title line that prints which parameter set produced whatever you are looking at. The record in the header is a worked example of the form: every field in it is a choice — the universe, the period, the costs, the grid, the region, the centre — and there is deliberately no performance figure anywhere in the file. Replace each field with your own before you rely on it.

Now run a single Backtest with this file and confirm the report matches the corresponding cell of your optimization exactly. It should, because nothing else changed. If it does not, find out why before you go any further: the usual culprits are a Range that has moved, a watch list that has gained a symbol, or a setting changed in the Settings dialog rather than in the file.

Step 11 — Write the decision record

Everything above becomes one page. The fields:

Field	What goes in it
Question	The one from Step 1, unedited
Acceptance rule	The thresholds you set before looking
Universe	Named watch list, with the liquidity filter stated
Period	Exact From and To dates, and the warm-up allowance
Data source	Where the bars came from, plus known defects such as missing delisted symbols
Costs	The percentage per side and what it is meant to cover
Fills	Delay and price, in words as well as code
Grid	Ranges, steps, and the combination count
Metric	Spelled as AmiBroker spells it in the place you typed it
Region	Bounds in both parameters; whether it touched an edge
Chosen value	The centre, with the statement that it is not the best cell
Neighbourhood	Worst, median, best, and the candidate’s position among them
Cost sensitivity	Whether the region survived and whether it moved
Period halves	Whether the regions overlapped
Still untested	The list below

Write the “chosen value” line in the register the last lesson recommended: on this universe and period, with these costs, anything in this block behaves alike; I chose the centre; the best single cell was better and I did not take it.

What still has to be tested

Put this list in the record verbatim, because it is the part that stops a parameter choice turning into a belief.

• Out-of-sample behaviour. Everything here used data you have seen. Part 32.
• Walk-forward behaviour. Would the region have been findable at each point in the past, using only the data available then? Part 32.
• Sequence and path risk. The same trades in a different order produce a different drawdown, and the equity path you saw is one of many. Part 33.
• Regime dependence. Whether the period contained more than one kind of market at all, and what the rule did in each. Part 30.
• Participation and liquidity. Whether the position sizes the backtest took were a defensible fraction of the actual daily turnover of the symbols it traded. Part 30.
• Execution assumptions. Gaps through the opening price, partial fills, and whether the next open was reachable at all on the days the system most wanted to trade.
• Survivorship. Whether the database contains the companies that failed.
• The selection effect itself. You searched 252 combinations. Even the region you chose was chosen with knowledge of the whole surface.

Verification checklist

Tick every line before you call the lab finished.

• The question and the acceptance thresholds were written before the first run.
• The Range starts after the longest warm-up in the grid, and both dates are recorded.
• The combination count was computed on paper and matches what the run reported.
• The result list was sorted by All trades and the thin cells were identified.
• Two grids exist: the metric and the trade count.
• The chosen region is contiguous, does not touch an edge, and clears the trade-count floor.
• The chosen value is the centre of the region, not the best cell in it.
• The offset (0, 0) row of the neighbourhood run reproduces the same cell from the full run.
• The neighbourhood spread — worst, median, best, candidate — is recorded.
• The cost-sensitivity run was done and its outcome recorded.
• Both period halves were optimized and the region overlap recorded.
• The chosen values are typed into a formula file, and a single backtest of that file reproduces the corresponding optimization cell.
• The decision record exists, contains no performance figure presented as an expectation, and ends with the untested list.

Recap

You ran a real optimization and came away with something more defensible than a pair of numbers: a region, a measured neighbourhood around it, a check that the region survives a harsher cost assumption, a check that it exists in both halves of the period, and a written record that another person could take apart.

The most valuable habits here are the ones that happen before the machine does anything — writing the question so it can be answered “no”, setting the acceptance thresholds in advance, and counting the grid so you know how large a selection process you are running. The most valuable output is the last section of the record. A parameter decision that ships with an honest list of what it has not established is a research artefact. The same decision without that list is a number somebody will eventually mistake for a promise.

Part 32 takes the region you chose and asks the only question that can genuinely test it: what happens on data you had not looked at when you chose it.

Check your understanding

Question 1. Why does the lab insist that the Analysis Range start after the longest warm-up in the grid?

• Because AmiBroker refuses to optimize when any parameter produces Null values
• Because cells using long averages cannot trade until they have enough bars, so otherwise the grid compares different effective periods
• Because the first bars of any database are unreliable
• Because it reduces the number of combinations that need to be tested

Show the answer and why

Answer: Because cells using long averages cannot trade until they have enough bars, so otherwise the grid compares different effective periods

A 260-bar average produces nothing for its first 260 bars, while a 60-bar average is already trading. Without a common start, part of the surface’s shape is just a map of which cells were exposed to the earliest — and often unusual — stretch of history. AmiBroker does not adjust for this; you do it with the Range setting.

Question 2. In the neighbourhood run, the row where both offsets are zero does not match the same parameter pair from the full optimization. What should you do?

• Accept the neighbourhood figures, since it is the more recent run
• Average the two results
• Stop and find the difference between the two runs before interpreting either
• Widen the neighbourhood to ±3 steps and re-run

Show the answer and why

Answer: Stop and find the difference between the two runs before interpreting either

The same rules, the same data and the same parameters must produce the same result. A mismatch means the two runs differed in the range, the watch list, or a setting changed in between — so at least one of them is not measuring what you think. This check exists precisely to catch that, and it is worth more than the sensitivity figures it was meant to produce.

Question 3. After thresholding and masking, the only contiguous block on your surface has four cells and sits against the right-hand edge of the slow-period range. What is the correct response? Select all that apply.

• Widen the slow-period range and re-run, since a block against an edge may be cut off
• Take the best cell in the block, since it is the only region available
• Record that the acceptance rule was not met on this grid
• Lower the threshold until a larger block appears

Show the answer and why

Answer: Widen the slow-period range and re-run, since a block against an edge may be cut off, Record that the acceptance rule was not met on this grid

A block touching an edge tells you the search stopped too early, so widening and re-running is the useful move. If nothing survives afterwards, the honest output is that the rule has no stable region on this universe and period — a real finding. Taking the best cell of a four-cell edge block, or relaxing a threshold you set in advance to manufacture a region, are both ways of deciding the answer after seeing the data.

Question 4. Which sentence belongs in the decision record?

• The optimised system returns 24% a year with a 12% maximum drawdown
• On this watch list from 2010 to 2024, with 0.30% per side and next-open fills, fast 15–30 against slow 100–140 behaved alike; I chose 20/120 as the centre and have not yet tested it out of sample
• The parameters were optimised for maximum profit
• Backtesting shows this to be a reliable configuration

Show the answer and why

Answer: On this watch list from 2010 to 2024, with 0.30% per side and next-open fills, fast 15–30 against slow 100–140 behaved alike; I chose 20/120 as the centre and have not yet tested it out of sample

The second sentence carries the universe, the period, the costs, the fill assumption, the region, the choice, the reason for the choice, and the boundary of what has been shown. The others state results without the assumptions that produced them, or state conclusions the exercise cannot support — and a backtested figure quoted as though it forecast anything is the specific mistake this whole part is designed to prevent.

Check my answersClear