Interpreting Out-of-Sample Degradation

The number to look at is not the out-of-sample result. It is the relationship between the in-sample result and the out-of-sample result, together with the size of gap you should have expected before you started. Without that second half, the first half is uninterpretable: a drop from CAR/MaxDD of 2.1 to 1.3 could be perfectly ordinary or a sign that nothing real is there, and the difference is not visible in the two numbers.

This lesson gives you a way to build the missing half yourself, and then works through the several situations in which a catastrophic out-of-sample result is telling you about your configuration rather than about your idea.

Degradation is built into the arithmetic

Suppose you optimise two parameters over 153 combinations on an in-sample segment. The figure you carry forward is not a typical result. It is the maximum over 153 draws, and a maximum has a systematic upward bias: whatever genuine signal exists is present in all 153 results, but the winner is also the combination whose share of luck was largest.

Three things follow, and they are worth stating as expectations rather than as surprises.

The bias grows with the size of the search. More parameters, finer steps, more combinations — every one of them raises the expected in-sample maximum without changing the underlying edge. Two systems with identical merit will show different degradation purely because one was searched harder.

The bias grows with the noisiness of the metric. A metric computed from few trades bounces around more, so its maximum sits further above its mean. Profit Factor over fourteen trades is mostly noise, and its in-sample maximum is mostly the noisiest combination.

Walk-forward does not remove the bias, it relocates it. Each in-sample segment still selects a maximum, so each step contributes some optimism to its own in-sample figure. What walk-forward gives you is a clean out-of-sample figure to compare it against.

∴Research note

The consequence is that “the out-of-sample result was worse than the in-sample result” is not a finding. It is the expected outcome of the procedure. The finding is how much worse, and that only means something once you know how much worse a system with no edge at all would have looked.

Building your own null

There is no universal figure for acceptable degradation, and any number offered without the size of the search behind it should be treated as decoration. What you can do is construct a comparison inside your own study. Three baselines, in increasing order of effort, and all of them cheap next to the run you have already done.

The fixed-parameter control. Run the identical rules with the parameters frozen at the mid-point of the ranges you searched, over the same span, the same universe and the same costs. No optimisation at all. Compare that against your walk-forward out-of-sample composite. If re-optimising every year produced no improvement over never optimising, the optimisation step is decoration and you have a simpler system than you thought — which is good news, not bad.

The median-parameter walk-forward. Re-run the walk-forward, but instead of taking the best in-sample parameter set at each step, take the median of the in-sample results. This is more work in AmiBroker than the first baseline, because you have to read each step’s result list rather than let the walk-forward pick for you, but it isolates one thing precisely: whether “pick the in-sample winner” beats “pick something typical”.

The mismatched-parameter test. Take the parameters that won on step 3 and apply them to step 8’s out-of-sample segment, and so on for a scrambled mapping. If the out-of-sample results are just as good with the wrong parameters attached, then the adaptation you believe you are performing is not happening, and the out-of-sample performance is a property of the rules and the market rather than of the re-estimation.

The point of all three is the same: give yourself something to subtract. A walk-forward result on its own has no denominator.

Comparing distributions, not points

A walk-forward run does not produce two numbers. It produces two sets of numbers — one in-sample figure and one out-of-sample figure per step — and treating them as sets is where the information is.

Collect, per step: the in-sample value of your optimisation target, the out-of-sample value of the same metric, the number of out-of-sample trades, and the parameter values chosen. Then look at five things.

The median gap, rather than the gap between the two aggregate figures. Aggregates hide a run in which one step carried everything.

The spread of the out-of-sample values. Consistent mediocrity across sixteen steps is a more encouraging result than one magnificent step and fifteen indifferent ones, even when the composite is identical, because the second pattern is what a single lucky market episode looks like.

The worst step. You would have lived through it. Ask whether you could have.

The proportion of steps with a positive out-of-sample result. Ten of sixteen is a different kind of evidence from sixteen of sixteen or four of sixteen, and it is not visible in a composite at all.

The out-of-sample trade counts. Steps containing three trades are not observations, and a run in which most steps are like that has fewer real data points than it appears to.

!Important

Remember what the out-of-sample summary report is doing. Since version 5.60 each out-of-sample step begins with the previous step’s ending equity, so the steps compound. A weak first step leaves less capital for every step after it, and any percent-of-equity sizing shrinks accordingly. That is correct for a summary statistic and misleading if you read the later steps as independent trials of the same system.

Efficiency measures, and their limits

A common summary is walk-forward efficiency: the out-of-sample value of a metric divided by the in-sample value of the same metric, either per step or on the aggregate.

Pseudocode — not valid AFL

for each walk-forward step:
    efficiency[step] = out_of_sample_target / in_sample_target

report median( efficiency ), minimum( efficiency ),
       count( efficiency > 0 ), count( out_of_sample_target > 0 )

Be clear about what this is. AmiBroker does not report a metric called walk-forward efficiency. It is a practitioner construction computed from figures AmiBroker does report, and you will not find a row for it in the backtest report or a column for it in the result list. If you use it, say where the two inputs came from.

It is useful as a ranking and a diagnostic, and poor as a threshold. Three reasons. The ratio is unstable when the denominator is small: an in-sample CAR/MaxDD of 0.2 and an out-of-sample value of 0.1 gives the same efficiency as 4.0 against 2.0, and the two situations are nothing alike. The ratio is meaningless when either value is negative, since two negatives produce a cheerful positive. And a high efficiency achieved by a poor in-sample result is not an achievement — the system merely failed consistently.

Use it alongside the absolute out-of-sample figures, never instead of them, and prefer the median across steps to a single aggregate ratio.

What catastrophic looks like

Some degradation is normal. These patterns are not, and each has a characteristic appearance:

• Out-of-sample results that are materially negative while in-sample results were strongly positive, across most steps rather than one.
• An out-of-sample composite worse than the fixed-parameter control. The optimisation did not merely fail to help; it actively selected badly.
• One step carrying the entire result, with the others flat or negative.
• Chosen parameters oscillating between the extremes of the range from step to step, which usually accompanies the first pattern and explains it.
• An out-of-sample equity curve that is flat for a third of its length, whatever the endpoint says.

When you see these, the temptation is to conclude the idea is dead. Sometimes that is right. Before you conclude it, spend twenty minutes on the alternative.

When degradation means the test was wrong

This is the section that saves ideas, and it is also the section that stops you from “fixing” a broken configuration by abandoning a sound hypothesis. Work through it in order; the early items are the common ones.

The geometry is not what you think. Open the Preview list again. In Advanced mode you can build overlapping in-sample and out-of-sample windows, gaps that skip whole years, or an out-of-sample segment shorter than the step. Check also that you chose the right Easy mode for your data — the EOD and intraday variants differ in whether the boundary day is shared between periods, and the wrong one either double-counts or drops a session on every step.

Use today was ticked. Your segments are not the ones you designed, and they change every time you run.

The optimisation target is not what you think. If you typed a custom metric name, check the spelling character by character against the report column heading. The field is matched by name, so a typo does not announce itself. And remember the default: if you never touched the field, you optimised CAR/MDD.

There are not enough out-of-sample trades. Count them. If your sixteen steps produced forty trades between them, you have not measured degradation, you have measured sampling noise, and the same run repeated on a slightly different universe would produce a materially different answer.

The in-sample window is too short for the formula’s warm-up. A 200-bar trend filter inside a one-year in-sample window leaves a small usable portion of that window, and any indicator with a long lookback quietly eats the start of every segment.

QuickAFL trimmed the calculation. AmiBroker calculates only the part of the array in range, plus a margin computed from the functions you called, and the official Knowledge Base article on QuickAFL lists the cases where that margin can be too small: scripting, for and while loops that reference more than 30 past bars, external indicator DLLs, and recursively-calculated functions such as very long exponential averages or timeframe functions using an interval far above the base interval. A walk-forward run works through restricted date ranges by construction. The official documentation does not spell out the interaction between walk-forward segment ranges and QuickAFL, so treat this as insurance rather than a diagnosis: if your formula contains any of those constructs, put SetBarsRequired( 1000, 0 ); at the top — or SetBarsRequired( sbrAll, sbrAll ); to disable QuickAFL entirely — and see whether the result changes. If it does, the earlier result was wrong.

Compounding across steps is being misread. Because each out-of-sample step starts from the previous step’s ending equity, a bad early step suppresses everything after it. Look at the per-step percentage figures, not just the currency ones.

Something was computed across the whole database. A composite built with AddToComposite over all bars, a static variable holding a full-history ranking, a normalisation using full-sample statistics — any of these carry information across your segment boundaries in both directions, and they can flatter the in-sample side as easily as they can distort the out-of-sample side.

The data changed character in the later years. Out-of-sample segments are, by construction, the recent ones. A ticker that was reused, a currency redenomination, a split that your source adjusted and your database did not, a symbol whose price fell by a factor of ten — all of these appear as an out-of-sample collapse and none of them is about your rules. Chart two or three of the worst out-of-sample trades and look at them.

✕Common mistake

Symptom: the out-of-sample result is catastrophic, but the trade list shows a handful of enormous losing trades in one symbol. Cause: almost never the strategy. Look for a data defect, a position-sizing interaction, or a stop that could not be reached because the instrument gapped. Diagnose the trades before you diagnose the idea.

When to abandon an idea

Decide the rule before the run, not after it. A workable form has three parts: the metric, the threshold, and the minimum number of out-of-sample trades below which you will not draw a conclusion at all. Written down in advance, it protects you from the two failure modes that follow a disappointing result — quietly changing the target, and quietly deciding that this particular test “was not really the point”.

Then act on the outcome:

• The run is uninterpretable — too few trades, or a configuration fault from the list above. Fix and re-run, and record that you did. This is not a free action: each re-run is a look at data you are treating as out-of-sample.
• Out-of-sample results are poor and the configuration is sound. Abandon it. The User’s Guide’s position is that a system with poor out-of-sample performance should not be traded, and that is the right default. Abandoning ideas is the normal output of research; a process that never rejects anything is not testing anything.
• Out-of-sample results are positive but no better than the fixed-parameter control. Drop the optimisation, keep the rules, and re-examine the simpler system. You have learned something genuinely useful.
• Out-of-sample results are positive and beat the control, on an adequate number of trades. You have a candidate. Not a demonstrated edge — a candidate worth forward testing on data that does not exist yet.

The most common honest verdict is the fifth one, which no list ever includes: inconclusive. The evidence was too thin to distinguish the idea from nothing. Say so, record the run, and either gather more data or spend your remaining research budget on an idea with a larger expected effect.

What changed

You should now be unable to read a walk-forward result without asking two questions the report does not answer: how large was the search that produced the in-sample number, and what did the same rules do with no optimisation at all. You should also have a checklist to run before treating a collapse as a verdict, because a broken configuration and a broken hypothesis look identical on the screen and are not remotely the same thing.

The lab that follows runs the whole procedure once, end to end, including the control.

Check your understanding

Question 1. Your in-sample optimisation searched 153 combinations and the winner scored CAR/MDD of 2.4. The out-of-sample figure is 1.4. What can you conclude from those two numbers alone?

• The system is overfitted, because performance fell by more than 40 per cent
• The system is sound, because the out-of-sample figure is still above 1
• Very little — the in-sample maximum is biased upward by the search, and you have no baseline
• The optimisation target was wrong for this system

Show the answer and why

Answer: Very little — the in-sample maximum is biased upward by the search, and you have no baseline

A maximum over 153 draws is systematically higher than a typical draw, so some fall was expected. Without a baseline — a fixed-parameter control, or the same procedure with no edge — the size of the fall carries no interpretation.

Question 2. Walk-forward efficiency of 0.5 is reported for two studies. In study A the in-sample CAR/MaxDD was 4.0 and out-of-sample 2.0. In study B it was 0.2 and 0.1. What does the shared ratio tell you?

• That both systems degraded identically and are equally worth pursuing
• Almost nothing — the ratio is unstable when the denominator is small, and the two cases are not comparable
• That study B is better, because it degraded from a lower base
• That both systems should be abandoned

Show the answer and why

Answer: Almost nothing — the ratio is unstable when the denominator is small, and the two cases are not comparable

Efficiency is a ratio of two numbers whose absolute size matters. It works as a diagnostic alongside the absolute figures and fails as a threshold, particularly near zero and whenever either value is negative.

Question 3. Which of these are configuration faults that can produce a catastrophic out-of-sample result without any problem in the strategy? Select all that apply.

• Use today ticked, so the segments were not the ones designed
• A custom-metric name mistyped in the Optimization target field
• Six out-of-sample trades spread across the whole run
• A rolling rather than anchored in-sample window

Show the answer and why

Answer: Use today ticked, so the segments were not the ones designed, A custom-metric name mistyped in the Optimization target field, Six out-of-sample trades spread across the whole run

The first two silently change what was tested; the third means nothing was really measured. Choosing a rolling window is a methodological decision that changes the question, not a fault.

Question 4. Your walk-forward out-of-sample composite is positive, but a fixed-parameter run using the mid-point of your ranges did slightly better over the same span. What is the reasonable conclusion?

• Widen the optimisation ranges and re-run the walk-forward test
• Trade the walk-forward version, since it adapts to changing markets
• The re-optimisation step is adding nothing; keep the simpler fixed-parameter system and re-examine it
• Change the optimisation target until the walk-forward version wins

Show the answer and why

Answer: The re-optimisation step is adding nothing; keep the simpler fixed-parameter system and re-examine it

The control is what the optimisation has to beat. Failing to beat it means the adaptation is decoration, and a system with fewer moving parts is easier to test, easier to trust and easier to run. Changing the target after seeing the result would be selecting on out-of-sample data.

Check my answersClear