Reading Parameter Surfaces

An optimization produces one row you were looking for and several hundred you were about to discard. The rows you were about to discard are the evidence. The one you were looking for is a summary statistic of a search you ran, and on its own it cannot tell you whether the search found something or merely found the largest number in a pile of noise.

This lesson is about looking at the whole pile. By the end you should be able to lay a two-parameter optimization out as a surface, open AmiBroker’s 3D chart correctly, tell a plateau from a spike and say why the distinction matters, name four specific mechanisms that put noise into a surface, and read a spiky surface as a statement about your data rather than about your rules.

What a surface is

Two Optimize() calls give you a grid. One parameter runs along one axis, the other along the second, and every intersection is one complete backtest of your whole universe over your whole date range. Choose a metric — any column in the result list — and that metric becomes the height of the grid at each point. A grid of heights is a surface.

Nothing about that requires three dimensions or a chart. A surface is just the result list re-laid-out so that neighbouring parameter values sit next to each other, which is exactly what a list sorted by profit destroys. The list tells you which cell was highest. The grid tells you what the region around it looks like, and that is the information you need.

AmiBroker’s 3D Optimization chart

The built-in viewer draws the surface for you, with documented rules that are easy to trip over.

It needs exactly two Optimize() calls. Not one, not three. Run the optimization first, then use the drop-down arrow on the Optimize button and choose 3D Optimization chart. (Older guide text calls the same command “View 3D optimization graph”.)

It plots Net profit unless you tell it otherwise, and the way you tell it is unintuitive. There is no metric selector. To plot a different column you click that column’s header in the result list to sort by it — a blue arrow appears — and then reopen the 3D chart. The plotted metric follows the current sort column.

✕Common mistake

Opening the 3D chart, looking at a beautiful surface, and reasoning about it as though it showed the metric you care about. By default it shows Net profit, which rewards taking more risk with more capital for longer. If your judgement is CAR/MaxDD, sort by that column first, then reopen the chart. The two surfaces of the same run can look nothing alike.

The viewer’s controls are documented: drag with the left mouse button to rotate, drag with the right to zoom, hold Ctrl and drag to translate, press Space to auto-rotate, use the arrow keys to rotate and the numeric keypad to zoom and pan. Page Up and Page Down move a “water level” that floods the surface below a chosen height — genuinely the most useful control in the viewer, because raising the water until only a few islands remain is a direct way of seeing whether your good results form one continent or an archipelago.

The guide’s own interpretive advice is worth stating in full, because it is unusually direct for vendor documentation. Robust settings are the regions showing gradual rather than abrupt changes in the surface; radical changes or spikes clearly show over-optimization; you should choose a parameter region that produces a broad and wide plateau; and parameter sets producing profit spikes will not work reliably in real trading.

Reading it flat

You do not need the 3D chart, and for careful work a flat grid is better: you can put numbers in it, you can compare two of them side by side, and you can paste it into your research notes. The result list can be copied out — the SetOption( "ExtraColumnsLocation", 1 ) documentation notes that moving the parameter columns changes the visual order only and not the export or copy-paste order — so the usual route is to copy the list into a spreadsheet and pivot it: one parameter down the rows, the other across the columns, your chosen metric in the cells.

Here are two such grids. They are synthetic: I constructed them to show two shapes clearly, and they are not results from any market, any instrument or any real backtest.

∴Research note

Every number in the two tables below is invented for illustration. They exist to show what a plateau and a spike look like as numbers, nothing more. No figure in this part is a performance claim, and you should be suspicious of any course that shows you one without a universe, a period, a cost assumption and a data source attached.

Surface A — a plateau (illustrative, synthetic figures)

Target metric, arbitrary units. Fast period down the side, slow period across the top.

Fast \ Slow	90	110	130	150	170
10	0.55	0.61	0.64	0.60	0.52
15	0.68	0.81	0.86	0.79	0.66
20	0.74	0.89	0.94	0.88	0.71
25	0.70	0.85	0.90	0.83	0.69
30	0.59	0.72	0.75	0.71	0.58

The best cell is 20/130 at 0.94. What makes this surface interesting is not that cell but its neighbours: every cell adjacent to it is between 0.85 and 0.90, and the whole block from fast 15–25 by slow 110–150 sits in a narrow band. If you had been forced to pick blind from anywhere in that block, you would have got something close to the same answer.

Surface B — a spike (illustrative, synthetic figures)

Fast \ Slow	90	110	130	150	170
10	0.31	0.28	0.35	0.22	0.29
15	0.24	0.41	0.19	0.33	0.26
20	0.36	0.22	1.87	0.27	0.31
25	0.29	0.34	0.25	0.38	0.23
30	0.33	0.21	0.30	0.24	0.35

The best cell is 20/130 at 1.87, five times its own neighbours. Sorted as a list, Surface B looks better than Surface A: its top row is much higher. As a surface it is obviously worse, and the reason is that its top row is not connected to anything.

Ask what it would mean to trade 20/130 from Surface B. Twenty and 130 were not chosen because theory says so; they were chosen because they scored highest. A fast period of 19 or 21 would have been an equally defensible choice a priori, and both score around 0.3. The result therefore rests entirely on a decision you had no independent reason to make. That is the definition of a fitted parameter.

Where the noise comes from

Surfaces are not smooth, and understanding why makes you better at reading them. Four mechanisms account for most of it.

Adjacent cells do not share all their trades. Two cells that differ by one step in one parameter share most of their signals, but not all: some crossovers happen a bar earlier, some marginal ones disappear, and a few trades exist in one cell and not the other. Metrics computed from a few dozen trades move noticeably when a handful of them change.

One trade can dominate. Trade outcomes are heavily skewed. If the best trade in a test accounts for a large share of the profit, then every cell where that trade fired scores well and every cell where it did not scores badly — and whether it fired can turn on a single bar. Sort your result list by the number of trades and look at the extremes: the cells with the fewest trades are the cells where this effect is largest, and they are disproportionately represented at both ends of a profit ranking.

Portfolio constraints reshuffle everything. In a portfolio backtest with a position limit, signals compete. Change a parameter slightly, and a symbol that previously missed a slot now gets one, which displaces a different symbol, which changes the cash available three months later, which changes what happened in an entirely unrelated part of the test. Portfolio surfaces are meaningfully noisier than single-symbol surfaces for this reason, and the PositionScore line in your formula is a full participant in the effect.

Cells do not all test the same period. This one is subtle and specific to indicator parameters. MA() returns Null until it has enough bars, so a cell using a 260-bar slow average cannot produce a signal until 260 bars into your data, while a cell using a 60-bar average starts two hundred bars earlier.

Two cells, two different effective test periods

Slow 60

warm-uptradable history

Slow 260

warm-uptradable history

start of dataend of data

Cells with longer averages begin trading later, so the grid is not comparing equal periods unless you make it.

If your data begins in 2010 and the market did something unusual in 2010 and 2011, cells with short slow averages are exposed to it and cells with long ones are not. The fix is to start the Analysis Range after the longest warm-up in your grid, so that every cell is judged on the same history. Nothing in AmiBroker does this for you.

What a spiky surface is telling you

The tempting reading is “this system is fragile”. The more useful reading is that the test was not able to distinguish between parameter settings, and there are only a handful of reasons why.

• Too few trades. With a few dozen trades in total and only a handful per cell, the metric is mostly sampling variation. This is the most common cause and the easiest to check.
• Too short a period, or too narrow a universe. One bull market is one observation. A surface computed over a single market phase describes that phase.
• One instrument dominating. If a single symbol supplies a large share of the trades, the surface is a study of that symbol wearing a portfolio’s clothes. Sorting the trade list by symbol answers this in seconds.
• A discontinuous rule. Rules with thresholds — cross above a level, close above the highest high of N bars — can flip on and off with a one-step parameter change in a way that smooth rules do not. Some spikiness here is structural rather than statistical.

Note what is not on that list: “the market changed”. It might have, but a spiky surface is not evidence for it, and reaching for that explanation is how a diagnostic result gets converted into a story.

!Important

There is one more possibility, and it is worth checking before any of the above: your best cell may be sitting on the edge of the grid you searched. If the optimum is at the minimum or the maximum of a range, you have not found a peak — you have found the wall. Widen the range and re-run. If the optimum moves to the new wall, the parameter is probably not doing what you think it is doing.

Visualising without deceiving yourself

A few habits that make surfaces useful rather than decorative:

1. Plot the metric you will judge on, not Net profit by default. Sort first, then open the chart.
2. Look at a second metric on the same grid. A region that is a plateau in CAR/MaxDD and a spike in Net Profit % is telling you something specific about how it earns.
3. Overlay the trade count. Build the same grid with the number of trades in the cells. Regions of the surface where the count collapses are regions you cannot interpret.
4. Use the water level. Raise it until only ten per cent of the surface is above water and look at the shape of what remains. One continent is a result; scattered islands are a warning.
5. Keep the grid. Paste it into your research notes with the universe, period and costs written above it. A surface you cannot reproduce in six months is a surface you cannot cite.

Recap

A two-parameter optimization is a surface, and the surface carries information that a sorted list throws away. AmiBroker will draw it in three dimensions if you have exactly two Optimize() calls, but it plots Net profit unless you sort by another column first, and a flat grid in a spreadsheet is often the better tool.

A plateau — a broad, contiguous region whose cells behave alike — means your choice within that region barely matters, which is the only kind of parameter choice that is not a bet on your sample. A spike means the opposite: neighbouring values that were equally defensible before you ran the test do far worse, so the result depends on a decision you had no reason to make. Spikes come from thin trade counts, single dominant trades, portfolio slot competition and unequal warm-up periods, and they are most usefully read as statements about the test rather than about the rules.

The next lesson takes this one step further and asks the uncomfortable question directly: if the maximum is usually noise, what exactly should you report instead?

Check your understanding

Question 1. You want AmiBroker’s 3D Optimization chart to show CAR/MaxDD rather than Net profit. What do you do?

• Type CAR/MaxDD into the Optimization target field on the Walk-Forward tab
• Click the CAR/MaxDD column header in the result list to sort by it, then reopen the 3D chart
• Choose the metric from a drop-down inside the 3D chart viewer
• Add a third Optimize() call for the metric

Show the answer and why

Answer: Click the CAR/MaxDD column header in the result list to sort by it, then reopen the 3D chart

The plotted metric follows the current sort column: click the header, wait for the blue arrow, then reopen the chart. There is no metric selector in the viewer. The Optimization target field does something different — it drives smart engines and walk-forward selection, and it does not change what the 3D chart plots.

Question 2. Two surfaces are computed from the same system and data. Surface A’s best cell scores 0.94 with neighbours around 0.88. Surface B’s best cell scores 1.87 with neighbours around 0.30. Which is the better basis for choosing parameters, and why?

• B, because its best result is much higher
• B, because a sharp optimum means the parameters are precisely tuned
• A, because its neighbours agree with it, so the choice within the region barely matters
• Neither can be judged without a third parameter

Show the answer and why

Answer: A, because its neighbours agree with it, so the choice within the region barely matters

In A, any nearby parameter pair would have produced a similar outcome, so the specific choice carries little weight. In B, the result rests entirely on picking one cell out of many that were equally defensible beforehand — which is what a fitted parameter is. The guide says the same thing in its own words: prefer broad plateaus, and parameter sets producing profit spikes will not work reliably in real trading.

Question 3. Which of these genuinely inject noise into an optimization surface? Select all that apply.

• A single very profitable trade that appears in some cells and not others
• A portfolio position limit, so that changing one parameter changes which symbols win slots
• Long moving-average periods needing more warm-up bars than short ones
• Sorting the result list by a different column

Show the answer and why

Answer: A single very profitable trade that appears in some cells and not others, A portfolio position limit, so that changing one parameter changes which symbols win slots, Long moving-average periods needing more warm-up bars than short ones

Skewed trade outcomes, competition for a limited number of slots, and unequal warm-up all change results between adjacent cells for reasons unrelated to the merit of the parameters. Warm-up is the one people miss: a long average cannot trade until it has enough bars, so cells are silently judged on different periods unless you start the Range after the longest warm-up. Re-sorting the list changes only the display.

Question 4. Your best cell sits at the maximum of the range you searched. What is the correct next step?

• Accept it — the optimizer found the highest value available
• Reduce the step size to search the same range more finely
• Widen the range and re-run, since an optimum on the boundary is not a peak
• Switch to a non-exhaustive search engine

Show the answer and why

Answer: Widen the range and re-run, since an optimum on the boundary is not a peak

A boundary optimum tells you only that the surface was still rising when you stopped looking. You cannot see whether there is a peak, a plateau or a cliff beyond the wall. Widen and re-run — and if the optimum simply moves to the new boundary again, treat that as evidence that the parameter is not doing what you assumed.

Check my answersClear