In-Sample and Out-of-Sample

A holdout is the only part of your research budget you cannot top up. Data arrives at one day per day, and once you have looked at a stretch of it you can never un-look. Everything in this lesson follows from that: how to define the boundary so it means something, the half-dozen ordinary ways people cross it without noticing, and what is honestly left to you once it has been spent.

By the end you should be able to say, for any test you have run, precisely which data influenced which decision, and to recognise the point at which the answer became “all of it”.

The definition that matters is about information, not dates

In-sample data is data you have used. Not “data before the split date” — data you have used: looked at, tuned on, iterated against, or that informed any choice you made. Out-of-sample data is data whose results have not influenced any decision you have taken.

That distinction sounds pedantic until you notice how much it rules out. A block of 2021 bars that you charted once, six months ago, while deciding whether an idea was worth pursuing, is in-sample. It does not become out-of-sample again because you later drew a line at 2020-12-31. The line is a bookkeeping device; the contamination is in your head and in the decisions you already made.

The consequence is uncomfortable and worth stating plainly: a holdout is a property of a process, not of a file. Two researchers can run the identical formula over the identical date range, and for one of them the result is evidence while for the other it is a restatement of what they already knew.

A single holdout split

Design

In-sample: build, tune, iterate freelyReserved

Verdict

Untouched during designOut-of-sample: spend once

2006201220182024

The reserved block is worth something only for as long as nothing you decide has been influenced by it.

Designing the holdout

Four decisions define a holdout, and each of them costs you something.

How much. A larger holdout gives a more reliable verdict and leaves less data to develop on. There is no correct fraction. The binding constraint is almost never a percentage of calendar time — it is the number of trades the holdout will contain. A twenty-per-cent holdout that produces eleven trades is a coin toss with extra steps. Work backwards: estimate the trade rate from the in-sample period, decide how many out-of-sample trades would make you willing to change your mind, and size the holdout from that. If the answer is “longer than my data”, the honest conclusion is that this idea cannot be evaluated with the data you have.

Which end. Reserving the most recent block is conventional and defensible: it is the period most like the one you would trade in, and it preserves the arrow of time, so nothing you fit can depend on information from later than the bars it is fitted to. The price is that you develop on the oldest, least representative data. Reserving the earliest block avoids that but tests your rules against a market structure that no longer exists, and invites the subtler problem that you already know what happened afterwards.

Whether the split is by time at all. Time is not the only axis. You can hold out symbols: develop on one half of a universe, test on the other. You can hold out a market: build on one country’s shares, test on another’s. These are useful and under-used, because they attack a different failure — rules fitted to the idiosyncrasies of particular instruments — but they do not test whether an edge survives a change of regime, because both halves live through the same history.

Regime coverage. A holdout that contains only a trending bull market tests one thing. If your in-sample period contains a crash and your holdout does not, a large part of what you learned is simply not on the exam. Look at what each side of your split actually contains before you fix it, and say so in writing.

∴Research note

Write the holdout definition down before you build anything: the exact dates, the universe, the costs, the metric you will judge on, and the number that would count as a pass. A holdout whose pass mark is decided after the result is not a test. It is a description.

Making the boundary mechanical

Discipline that depends on remembering is discipline that fails on a Thursday evening. The cheapest fix is to make the reserved period unreachable while you are developing, so that a careless date range or an absent-minded “let me just see” produces nothing rather than a number you cannot unsee.

Complete runnable AFL

holdout-guard.afl

// holdout-guard.afl
// Part 32 - In-Sample and Out-of-Sample
//
// A development-time guard. While you are still designing, tuning and
// re-running, this makes it mechanically impossible to see a result computed
// on the data you reserved as a holdout: every entry that would be FILLED at
// or after HoldoutFrom is removed, and any position still open on the boundary
// is closed there.
//
// The strategy below is deliberately ordinary. The guard is the part worth
// copying into whatever system you are actually working on.
//
// Assumptions:
//   - Daily bars, long only, portfolio backtest.
//   - Signals evaluated on the close of the signal bar, filled at the NEXT
//     bar's open (SetTradeDelays 1 with BuyPrice/SellPrice set to Open).
//   - Commission 0.15% of trade value each way. No slippage is modelled here,
//     because this file exists to demonstrate the guard, not to produce a
//     result you would act on.
//   - Dates are AmiBroker DateNums: 10000 * (year - 1900) + 100 * month + day,
//     so 2020-01-01 is 1200101 and 2015-12-31 is 1151231.

_SECTION_BEGIN( "Holdout guard" );

// --- The line you have promised not to cross --------------------------------
HoldoutFrom = 1200101;   // 2020-01-01
EntryDelay  = 1;         // must match the buy delay set below

// --- Account and cost model -------------------------------------------------
SetOption( "InitialEquity",    100000 );
SetOption( "MaxOpenPositions", 10 );
SetOption( "CommissionMode",   1 );     // 1 = percent of trade value
SetOption( "CommissionAmount", 0.15 );
SetTradeDelays( EntryDelay, 1, 1, 1 );

BuyPrice  = Open;
SellPrice = Open;
SetPositionSize( 10, spsPercentOfEquity );

// --- The rules under development --------------------------------------------
FastPeriod = 50;
SlowPeriod = 200;

Buy  = Cross( MA( Close, FastPeriod ), MA( Close, SlowPeriod ) );
Sell = Cross( MA( Close, SlowPeriod ), MA( Close, FastPeriod ) );

// --- The guard ---------------------------------------------------------------
Reserved = DateNum() >= HoldoutFrom;

// A signal on the last bar before the boundary is FILLED on the first bar of
// the holdout, because SetTradeDelays moves the signal one bar forward inside
// the backtester. Blocking the signal one bar early keeps the fill out too.
// Ref() with a positive shift is normally look-ahead and normally forbidden;
// here it reads the calendar, which is known in advance, not a price.
NoEntry = Reserved OR Ref( Reserved, EntryDelay );
Buy     = Buy AND NOT NoEntry;

// Removing entries is not enough. A position opened in 2019 would otherwise
// carry its profit or loss through the holdout, so force the exit at the
// boundary and let the delay place the fill on the first holdout bar.
Sell = Sell OR Reserved;

// One entry per crossing, one exit per crossing.
Buy  = ExRem( Buy,  Sell );
Sell = ExRem( Sell, Buy  );

// The guard only means anything if the analysis range actually reaches into
// the reserved period. Printing the range in force costs nothing and catches
// the case where you have quietly been testing 2010-2014 for a fortnight.
// _TRACE writes one line per symbol, so switch this off before an optimization.
_TRACE( "Range in force: " + NumToStr( Status( "rangefromdate" ), 1.0 )
        + " to " + NumToStr( Status( "rangetodate" ), 1.0 )
        + " | holdout begins " + NumToStr( HoldoutFrom, 1.0 ) );

_SECTION_END();

Download holdout-guard.afl75 lines

Two details in that file are worth pulling out, because both are places where a guard that looks right silently is not.

The first is the trade delay. SetTradeDelays() is documented as shifting the signal arrays inside the backtester — Buy = Ref( Buy, -buydelay ) and the same for the other three — after your formula has run. So a Buy on the last bar before your boundary is filled on the first bar of the reserved period. Blocking signals that occur inside the holdout is not enough; you have to block the signals that would be filled inside it, which is why the guard looks one bar forward at the calendar. That positive Ref() shift is the one case where reading ahead is legitimate: it reads a date, and the dates were known in advance.

The second is that removing entries is not sufficient. A position opened before the boundary keeps accruing profit or loss through the reserved period, and its exit price comes from data you promised not to use. The guard forces a flat book at the boundary.

✕Common mistake

The commonest failure is not a bad guard, it is no guard plus a wrong range. You set the Analysis window to a range once, work for a fortnight, and never look at it again. Status( "rangefromdate" ) and Status( "rangetodate" ) return the range actually in force as DateNums; printing them with _TRACE costs nothing and tells you what you are really testing. Turn the trace off before an optimization — it writes one line per symbol per run.

How a holdout gets contaminated

Almost nobody deliberately peeks. Contamination happens through routine work. These are the routes worth checking, roughly in order of how often they occur.

Re-running after a disappointment. You run the holdout test, the result is poor, you change one rule “because that was obviously wrong anyway”, and you run it again. The second run is not an out-of-sample test. You have used the holdout to select between two variants, which is exactly what in-sample data is for. Three runs in and the holdout has become a very small, very noisy training set.

Choosing the universe with hindsight. A watch list assembled today contains the companies that still exist and still meet a screen today. Every symbol in it survived your holdout period. This is survivorship bias wearing a different hat, and it contaminates the holdout specifically, because the selection used information from the reserved block.

Preprocessing across the boundary. Any statistic computed over the whole database and then used inside the in-sample period leaks backwards: a liquidity floor set from full-history average turnover, a volatility normalisation using the full-sample standard deviation, a sector composite built with AddToComposite over every bar and then consulted as a filter. The arithmetic is innocent; the information flow is not.

Choosing the split after seeing results. “Let us call 2018 onwards out-of-sample” is a different sentence when said before running anything than when said after noticing that the rules did well from 2018. If the boundary moved even once, record why, and treat everything after the move as in-sample.

Your own memory. You lived through the period in your holdout. You know which years were kind to trend following and which were not. Priors built from that knowledge are inside every rule you write, and no amount of file discipline removes them. This is not a reason to give up; it is a reason to prefer rules with few decisions in them, and to be suspicious when a rule happens to sidestep exactly the episode you remember.

Community-level reuse. The same handful of liquid markets and the same twenty years of history have been mined by everyone, published on, and turned into the received wisdom you learned from. In a real sense there is no untouched out-of-sample data in liquid equities for anybody. That does not make testing pointless — it makes the size of a claimed effect the thing to be sceptical about.

Spending the holdout once

The protocol is short, which is what makes it hard.

1. Freeze everything: rules, parameters, universe, cost model, date range, position sizing.
2. Write down, before running, the metric you will judge on and the value that constitutes a pass. Use AmiBroker’s own vocabulary so there is no wriggle room later — CAR/MaxDD, Max. system % drawdown, Net Profit %, and the number of trades you require before the figures mean anything.
3. Run it once.
4. Record the result whether or not you like it, together with the run’s settings. A saved Analysis project makes this exact rather than approximate.
5. Do not change anything and run again.

Step 5 is the whole protocol. Steps 1 to 4 are administration.

What to do after you have used it up

You will use it up, usually sooner than planned. The useful question is what remains honest afterwards.

Accept the reduced claim. The strategy is a candidate that survived one test. That is a real, small thing, and it is not the same as demonstrated. Say “not falsified” rather than “validated”, and mean it.

Wait. Time manufactures fresh out-of-sample data at a rate of one day per day, and it is the only genuinely uncontaminated data you will ever get. Paper trading, or trading small, over the next six to twelve months produces a holdout nobody has mined — including you. It is slow, which is exactly why it is uncorrupted.

Move to walk-forward. Rather than one boundary spent once, walk-forward analysis re-estimates parameters on a schedule and evaluates only the segments the procedure had not yet seen. It gives you many small out-of-sample windows instead of one large one, and it tests the procedure rather than a parameter set. That is the subject of the rest of this part — but note now that it is a different question, not a way of getting your holdout back.

Change a different universe. Running the frozen rules on a market you have never touched is a weak but real test. Weak, because market histories are correlated and because your priors travel with you; real, because instrument-specific curve fitting does not travel.

Reduce the number of decisions per unit of data. This is the only structural fix. Every parameter, every filter, every “let us also require” spends holdout in advance. A three-decision system tested on twenty years is on firmer ground than a thirty-decision system tested on the same twenty years, and no amount of clever splitting changes that arithmetic.

What changed

You should now be treating “out-of-sample” as a claim about your own process rather than a property of a date range, and you should be able to list the specific decisions in your current project that were informed by data you had promised to reserve. The mechanical guard is a convenience; the accounting is the discipline. The next lesson takes the same idea and turns it into a repeating procedure, which buys you more out-of-sample observations at the cost of a subtler set of ways to be wrong.

Check your understanding

Question 1. You reserved 2020-2024 as a holdout, ran the frozen system once, disliked the drawdown, tightened the stop, and ran again. What is the status of the 2020-2024 data now?

• Still out-of-sample, because the rules were frozen before the first run
• In-sample, because it was used to choose between two versions of the system
• Out-of-sample for the stop but in-sample for the entries
• Unaffected — one re-run is within normal tolerance

Show the answer and why

Answer: In-sample, because it was used to choose between two versions of the system

A second run after a change makes the reserved data the thing that selected the change. Selection is exactly what in-sample data does, however few candidates were involved.

Question 2. Which of these contaminate a time-based holdout? Select all that apply.

• Building the watch list from symbols that meet a screen today
• Setting the liquidity floor from average turnover computed over the whole database
• Reserving the most recent 25% of bars rather than the earliest 25%
• Moving the split date after noticing the rules did better from 2018 onwards

Show the answer and why

Answer: Building the watch list from symbols that meet a screen today, Setting the liquidity floor from average turnover computed over the whole database, Moving the split date after noticing the rules did better from 2018 onwards

A screen run today, and any statistic computed across the whole history, both carry information from the reserved period into the design. Moving the boundary after seeing results is selection on the holdout. Choosing to reserve the most recent block is a design decision, not a leak.

Question 3. Your formula uses SetTradeDelays( 1, 1, 1, 1 ) and you block every Buy signal that occurs on or after the boundary date. What still leaks?

Buy = Buy AND NOT ( DateNum() >= HoldoutFrom );

• Nothing — signals inside the holdout have been removed
• A signal on the last bar before the boundary is filled on the first holdout bar
• The Sell array is unaffected by trade delays
• DateNum() is not comparable with a numeric literal

Show the answer and why

Answer: A signal on the last bar before the boundary is filled on the first holdout bar

The documented delay mechanism shifts the signal arrays inside the backtester after your formula runs, so the last pre-boundary signal trades at the next bar's open, which is inside the reserved period. Block the signal one bar earlier, and force the book flat at the boundary.

Question 4. Which statement about a holdout that has been spent is defensible?

• The strategy has been validated and can be traded at full size
• Re-optimising on the holdout is acceptable because it is now just more data
• The strategy survived one test and is a candidate, not a demonstrated edge
• A fresh holdout can be created by re-splitting the same history differently

Show the answer and why

Answer: The strategy survived one test and is a candidate, not a demonstrated edge

Surviving one out-of-sample test is weak positive evidence. Re-splitting history you have already worked through produces no new information, and the only source of genuinely fresh data is time.

Check my answersClear