Best Value versus Robust Region

The top row of an optimization result list is the parameter set that fitted your sample best. That sentence is not a criticism; it is a definition. The question this lesson answers is what follows from it — because what follows is that the top row’s score is a systematically optimistic estimate of what those parameters would do again, and the size of the optimism grows with how hard you searched.

By the end you should be able to explain the selection effect in plain arithmetic, define stability operationally rather than as an adjective, choose a parameter set from a region instead of from a ranking, run a neighbourhood sensitivity test in a few minutes, and write up a parameter decision in a form that another person could check.

The arithmetic of picking a maximum

Here is a thought experiment you can carry out in a spreadsheet in two minutes, with no market data at all.

Generate 252 random numbers drawn from a distribution centred on zero — the same number of draws as the two-parameter grid in the previous lesson. Now take the largest of them. It will be a comfortably positive number, typically two to three standard deviations above the centre. Repeat the whole exercise and you get another comfortably positive number. The maximum of a set of noisy measurements is high because it is the maximum, whether or not anything in the set is genuinely different from anything else.

Every cell of an optimization surface is a measurement with noise in it, for all the reasons the last lesson listed: shared-but-not-identical trades, one dominant trade, portfolio slot competition, unequal warm-up. So the score of any given cell is roughly

Pseudocode — not valid AFL

observed score = whatever the parameters are really worth
               + sampling noise from this particular history

and selecting the maximum selects on the sum of those two terms. The winner is the cell where the two happened to be largest together. Its noise component is, on average, positive — and the more cells you search, the more positive it gets.

?Reality check

The claim under examination: a more thorough optimization produces a better system. Held up to the light, the opposite is closer to true. Searching more combinations reliably produces a higher reported number and a worse estimate, because you have given the selection process more noise to choose from. The improvement in the headline figure is partly real and mostly an artefact of the search, and nothing in the result list separates the two. This is the same multiple-comparisons problem Part 30 examines at the level of whole strategies, appearing here inside a single formula.

None of this says optimization is worthless. It says the maximum is the wrong output. The useful output is the shape.

Stability, defined so you can measure it

“Robust” is an adjective people apply after the fact to results they like. Three operational definitions make it checkable.

Local stability. The change in your chosen metric when one parameter moves by one step of the grid. If moving from a fast period of 20 to 25 halves the metric, the setting is not stable, regardless of how good 20 looked.

A robust region. Choose a threshold that means “acceptable” — stated before you look, and justified by something other than the surface itself. Mark every cell at or above it. A robust region is a contiguous block of marked cells, ideally not touching the boundary of the grid you searched.

Neighbourhood spread. For a candidate cell, the distribution of the metric across the cells immediately surrounding it: worst, median, best. A candidate far above its neighbourhood median is a spike. A candidate close to its neighbourhood median is a member of a region.

!Important

A flat surface is not automatically a robust one. If the metric barely moves anywhere in the grid, including in places where theory says it should, the likely explanation is that the parameter is not doing much — the trades are nearly identical across the whole grid. Check the number of trades and the trade list, not just the metric. “Insensitive because the parameter is irrelevant” and “insensitive because the rule is sound over a wide range” look identical in a single column of numbers and mean entirely different things.

Choosing from a plateau

Once you have a region, take the centre of it rather than the best cell inside it. Three reasons, in increasing order of importance.

1. The estimate rests on more evidence. The typical value across a twenty-cell region is informed by twenty backtests. The value in one cell is informed by one.
2. It leaves room on every side. If the useful parameter range drifts slightly — because volatility regimes change, or because your universe changes — a value at the centre of a region is still inside it after the drift; a value at the edge is not.
3. It breaks the selection process by construction. You are explicitly not choosing the maximum, so the bias described above does not attach to your choice in the same way. You are choosing a location, and then reporting what the neighbourhood around that location did.

Centre means the geometric middle of the region in parameter space, not the cell with the median score. If the region runs from fast 15 to 30 and slow 100 to 140, the centre is around fast 22 and slow 120.

A small convention that is worth adopting: within the region, prefer a round number. Choosing 20 rather than 22 is not better in any measurable sense — both are inside the region and that is the entire claim being made — but it makes it evident to a reader, and to yourself in six months, that the value was not arrived at by squeezing the last decimal out of a fit.

A sensitivity analysis you can run in ten minutes

Goal

You have a candidate. You want to know whether it is a member of a region or a lone spike, and you want the answer today rather than after another full grid search. The measurement is simple: re-run the optimization over just the immediate neighbourhood of the candidate and look at the spread.

Complete formula

Complete runnable AFL

neighbourhood-sensitivity.afl

// neighbourhood-sensitivity.afl
// Part 31 - Optimization
//
// The same trend rules as optimizable-trend-system.afl, restricted to the
// immediate neighbourhood of ONE candidate parameter set.
//
// WHY THIS IS A SEPARATE FILE
// After a full optimization you have a candidate cell. The question that
// decides whether you may use it is not "how good is this cell" - you already
// know it is the best one, which is why you are looking at it - but "how good
// are the cells around it". This file answers that in 25 backtests, and it
// reports OFFSETS rather than absolute periods, so the optimization result
// list reads as a map of the neighbourhood with the candidate at (0, 0).
//
// HOW TO USE IT
//   1. Set FastCentre and SlowCentre to your candidate.
//   2. Set FastStepSize and SlowStepSize to the step sizes you optimized with,
//      so one offset unit here means one cell of the original surface.
//   3. Run Optimize. You get 5 x 5 = 25 rows.
//   4. Sort by the column you are judging on and read the SPREAD across all
//      25 rows: the worst, the median and the best. The top row on its own
//      tells you nothing you did not already know.
//
// The offset ranges are written as literal numbers rather than computed from a
// "reach" variable, so that what the optimizer will enumerate is legible at the
// point where it is declared. Widen them by editing the two lines.
//
// ASSUMPTIONS - identical to optimizable-trend-system.afl, and repeated here
// on purpose, because a file that produces numbers must carry its own
// assumptions: the file is what gets copied, not the lesson.
//   - Daily bars, portfolio backtest, cash account, no margin.
//   - One-bar delay on every signal; fills at the next bar's open.
//   - Commission as a percentage of trade value per side, set from the formula.
//   - A turnover floor applied before any entry is allowed.
//   - No survivorship correction.

_SECTION_BEGIN( "Neighbourhood Sensitivity" );

// --- The candidate under examination -------------------------------------
FastCentre   = 20;    // the fast period you are considering trading
SlowCentre   = 120;   // the slow period you are considering trading
FastStepSize = 5;     // the step used in the original optimization
SlowStepSize = 10;    // the step used in the original optimization

// --- The neighbourhood ---------------------------------------------------
// Two steps either way, in both dimensions. Offset 0 is the candidate itself.
FastOffset = Optimize( "Fast offset (steps)", 0, -2, 2, 1 );
SlowOffset = Optimize( "Slow offset (steps)", 0, -2, 2, 1 );

// Max() clamps rather than trusting the centre and the reach to be sensible
// together: a moving-average period below 2 is not a moving average, and the
// formula still runs for combinations that Exclude will later suppress.
FastPeriod = Max( 2, FastCentre + FastOffset * FastStepSize );
SlowPeriod = Max( 3, SlowCentre + SlowOffset * SlowStepSize );

Exclude = FastPeriod >= SlowPeriod;

// --- Fixed assumptions ---------------------------------------------------
StartingEquity = 100000;
MaxPositions   = 10;
CommissionPct  = 0.30;      // percent of trade value, charged per side
MinTurnover    = 2000000;   // in the currency of the database
TurnoverPeriod = 50;

SetOption( "InitialEquity",    StartingEquity );
SetOption( "MaxOpenPositions", MaxPositions );
SetOption( "CommissionMode",   1 );
SetOption( "CommissionAmount", CommissionPct );
SetOption( "AllowPositionShrinking", True );

SetPositionSize( 100 / MaxPositions, spsPercentOfEquity );

SetTradeDelays( 1, 1, 1, 1 );
BuyPrice  = Open;
SellPrice = Open;

// --- Liquidity filter ----------------------------------------------------
Turnover     = Close * Volume;
AvgTurnover  = MA( Turnover, TurnoverPeriod );
LiquidEnough = NOT IsNull( AvgTurnover ) AND AvgTurnover >= MinTurnover;

// --- The rules, unchanged ------------------------------------------------
FastAvg = MA( Close, FastPeriod );
SlowAvg = MA( Close, SlowPeriod );

Buy  = Cross( FastAvg, SlowAvg ) AND LiquidEnough;
Sell = Cross( SlowAvg, FastAvg );

PositionScore = 100 * ( Close - SlowAvg ) / SlowAvg;

_SECTION_END();

Download neighbourhood-sensitivity.afl91 lines

How it works

The trading rules, the costs, the delays and the liquidity filter are identical to optimizable-trend-system.afl — that is the point, since a sensitivity test that differs from the original run in any other way measures the difference rather than the sensitivity.

What changes is the parameter space. Instead of optimizing the periods, the file optimizes two offsets from a centre you set at the top, each running from −2 to +2 in whole steps. The offsets are multiplied by the step sizes of your original grid, so one offset unit here equals one cell of the original surface. Five values each way gives 5 × 5 = 25 runs, and the candidate itself is the row where both offsets are zero.

Max( 2, … ) clamps the periods so that a centre near the edge of your grid cannot produce a moving-average period below 2. Exclude then suppresses statistics for any combination where the fast period is not actually faster.

Key functions

• Optimize( "Fast offset (steps)", 0, -2, 2, 1 ) — the min may be negative; nothing in the signature requires the range to start at zero.
• Max( x, y ) — returns the larger of its two arguments, used here as a clamp.
• Exclude — the reserved variable that suppresses statistics for the current run.

Expected result

Twenty-five rows, with two trailing columns headed Fast offset (steps) and Slow offset (steps) running from −2 to 2. Sort by the metric you are judging on and read three numbers off the list: the worst, the median and the best. Then find the row where both offsets are zero — that is your candidate — and compare it to the median.

Test it

The row with both offsets at zero must reproduce the candidate’s figures from the full optimization exactly. If it does not, the two runs differ in something you did not intend: the Apply to setting, the Range, a setting changed in the Settings dialog between runs, or a centre and step size that do not correspond to the grid you actually searched. Finding that discrepancy is worth more than the sensitivity test itself, because it means one of your two runs was not measuring what you thought.

Common errors

• Step sizes that do not match the original grid. If you optimized slow periods in steps of 10 and set SlowStepSize = 5 here, you are examining a neighbourhood that does not exist on your surface.
• Reading the top row. The top row of a neighbourhood run is by construction the best of 25, which is exactly the quantity you are trying to stop trusting. The spread is the output.
• Changing the date range between the two runs, usually by leaving Range on “N recent bars” while the database has been updated in between.

Extension

Add cost sensitivity as a third dimension by making the commission an optimized variable: CommissionPct = Optimize( "Commission %", 0.30, 0.20, 0.60, 0.10 );. Five cost levels across the 25-cell neighbourhood is 125 runs, still a short job, and it answers a sharper question than the neighbourhood alone: does the region survive a doubling of your cost assumption, and does it move when costs rise? Regions that shift toward longer holding periods as costs rise are behaving exactly as they should, which is quietly reassuring; regions that evaporate were being paid for by an assumption rather than by the rules.

Reporting a region rather than a number

The output of this whole process is not “22 and 117”. It is a short record, and every field in it is a choice you can defend rather than a result you are asserting:

• The question, written before the run.
• Universe, period and data source, including known defects such as missing delisted symbols.
• Cost, delay and fill assumptions, in full.
• The grid searched — ranges, steps, and the resulting number of combinations.
• The metric judged on, spelled the way AmiBroker spells it: CAR/MaxDD, Annual Return %, Max. system % drawdown, and so on.
• The region: bounds in each parameter, and whether it touches the edge of the grid.
• The value chosen, and the statement that it is the centre of the region rather than the best cell.
• The neighbourhood spread: worst, median and best across the 25 cells.
• What has not been tested.

In conversation, the difference sounds like this. Instead of “the optimum is fast 22, slow 117”, you say: “on this universe and period, with these costs, anything from fast 15 to 30 against slow 100 to 150 behaves alike; I chose 20 and 120 as the centre; the best single cell in the grid was better than that, and I did not take it.” The second version is longer, and it is the only one of the two that another person can evaluate.

⚠Warning

A region is a statement about the parameter choice, not about the strategy. A wide, flat, beautifully stable plateau of uniformly poor results is a perfectly common outcome, and it means the rules do not work on this universe over this period at any setting — which is a genuinely useful finding, and one you should be pleased to have obtained cheaply. Do not let the pleasant word “robust” migrate from describing the shape of the surface to describing the merit of the system.

What a region does not buy you

Everything in this lesson happened on the same history you chose the parameters from. A plateau reduces one specific hazard — the risk that your result depends on an arbitrary choice between equally defensible parameter values — and it reduces nothing else.

It does not establish that the rules will behave similarly on data you have not seen. A wide plateau can itself be a feature of one particular sample: an unusually persistent trend in your period can make every trend-following parameter look alike, and the plateau evaporates in a period without one. Part 32 addresses exactly this with holdout and walk-forward testing, and Part 33 asks what resampling can add on top. Until those are done, what you have is a well-documented parameter choice and no evidence about the future — which is precisely what your write-up should say.

Recap

The maximum of a noisy search is high partly because it is the maximum, and the bias grows with the size of the search, so the more thoroughly you optimize the more optimistic your headline figure becomes. The defensible output is a region: a contiguous block of parameter values whose results agree, identified against a threshold you set in advance, checked for whether it touches the boundary of the grid.

Stability is measurable — one-step change, contiguous block, neighbourhood spread — and a 25-run neighbourhood test answers the “spike or region” question in minutes. Choose the centre of the region rather than its best cell, prefer a round number inside it, and report the region, the choice, the spread and the assumptions together.

And keep the two claims separate. A region tells you your parameter choice is not fragile. It tells you nothing about whether the strategy is any good, and nothing at all about data you have not yet looked at.

Check your understanding

Question 1. You search a grid of 1,000 combinations instead of 100. What happens to the reported score of the winning cell, and to its value as an estimate?

• Both improve, because a finer search finds a better system
• The reported score tends to rise; its value as an estimate falls
• Both are unaffected, since the underlying system did not change
• The reported score falls because more poor combinations are included

Show the answer and why

Answer: The reported score tends to rise; its value as an estimate falls

The maximum of a larger set of noisy measurements is larger on average, whether or not anything in the set is genuinely better. The extra height is selection, not merit — so the headline number rises while the estimate becomes more optimistic and less useful. This is why "we optimised more thoroughly" is not a reassuring sentence.

Question 2. A neighbourhood test on a candidate returns 25 rows. The candidate sits at the 90th percentile of its own neighbourhood, and the worst cell in the neighbourhood is roughly a fifth of the candidate’s value. What is the reasonable conclusion?

• The candidate is well chosen, since it beats its neighbours
• The candidate is a spike, and choosing it means betting on one cell out of many that were equally defensible beforehand
• The neighbourhood was too small and should be widened to ±5 steps
• Nothing can be concluded without an out-of-sample test

Show the answer and why

Answer: The candidate is a spike, and choosing it means betting on one cell out of many that were equally defensible beforehand

Beating its neighbours is what a spike does; it is not evidence in the candidate’s favour, because the candidate was selected for being high in the first place. A member of a region sits near its neighbourhood median. Widening the neighbourhood would give more information, and an out-of-sample test is required eventually, but the spread already answers this question.

Question 3. A grid shows almost no variation in the target metric anywhere, including between very short and very long parameter values. Which explanations should you check before calling the system robust? Select all that apply.

• The parameter may barely change which trades are taken
• The trade count may be so low that the metric is dominated by a few trades common to every cell
• A flat surface always indicates a stable, well-behaved system
• The metric chosen may be insensitive to the thing the parameter controls

Show the answer and why

Answer: The parameter may barely change which trades are taken, The trade count may be so low that the metric is dominated by a few trades common to every cell, The metric chosen may be insensitive to the thing the parameter controls

Flatness is ambiguous. It can mean the rule works over a wide range, or that the parameter is doing nothing, or that the metric cannot see the difference, or that every cell is trading the same handful of positions. Comparing trade lists between distant cells separates these in a couple of minutes; the metric column alone cannot.

Question 4. Which of these belongs in a defensible parameter decision record?

• The best value found, and the metric it maximised
• The region, the chosen centre, the neighbourhood spread, the assumptions, and an explicit list of what remains untested
• The equity curve of the winning combination
• The number of combinations tested, since that shows how thorough the search was

Show the answer and why

Answer: The region, the chosen centre, the neighbourhood spread, the assumptions, and an explicit list of what remains untested

The record has to let somebody else re-run and disagree with you. That needs the universe, period, data source, costs, delays, the grid, the metric spelled as AmiBroker spells it, the region, the chosen value with its justification, the spread around it, and — the part most often missing — the list of tests that have not yet been done. The combination count belongs in the record too, but as context for how much selection bias to expect, not as evidence of quality.

Check my answersClear