Data Snooping, Curve Fitting and Overfitting

Here is the uncomfortable fact this lesson is built on: if you test enough ideas against one dataset, you will find something that looks excellent, and it will look excellent whether or not anything is there.

That is not a warning about carelessness. It is arithmetic. And the reason it is so dangerous is that the process which produces a spurious result feels exactly like the process which produces a real one — you had an idea, you tested it, you refined it, it improved.

Three names for three different things

They are constantly used interchangeably and they are not the same.

Data snooping (or data dredging) is a property of your whole research process: how many things you tried against this dataset in total. It includes ideas you abandoned, parameters you adjusted, universes you swapped, date ranges you moved. It counts even when each individual test was performed correctly.

Curve fitting is a property of the model: adding complexity until the rules describe the particular wiggles of the sample rather than any general behaviour. A filter that excludes exactly the three worst trades is curve fitting.

Overfitting is the outcome: a model that performs well on the data it was built from and poorly on data it has not seen.

Optimisation is not on that list, and that is deliberate. Choosing a parameter by testing values is a legitimate and necessary activity. It becomes overfitting when the number of things you tried gets large relative to the amount of independent information in your data, and when you report the best one as though it were the only one you tried.

Counting what you actually tried

The number that matters is not how many parameters your final formula has. It is how many distinct specifications were evaluated against this dataset before you settled on one — and almost everybody undercounts it by an order of magnitude.

A realistic count for an ordinary afternoon's work

1. 1Two moving-average periods, 10 values each100 combinations. This is the only part most people count.
2. 2Three exit rules tried×3 = 300.
3. 3Two universes tried, because the first "had data problems"×2 = 600.
4. 4The start date moved once, to avoid "an unrepresentative period"×2 = 1,200.
5. 5A volatility filter added, then removed, then added with a different threshold×3 = 3,600 specifications evaluated against one dataset.

Nobody would describe that afternoon as reckless. Every individual decision had a reason. And at the end of it, the best of 3,600 tries is being reported as though it were one test.

∴Research note

The corrective is unglamorous and it works: keep a research log, and write down every specification you evaluate, including the ones you abandon and why. Not a summary — a list.

Two things happen when you do. You find out that your count is far higher than you believed. And when a result finally looks good, you can put a number next to it: this is the best of N, which is a completely different claim from this is what I found.

The screen journal is the mechanical half of this. The log is the intellectual half.

Degrees of freedom in a strategy

Every choice that could have been made differently is a degree of freedom, whether or not it looks like a parameter:

• Each numeric parameter (period lengths, thresholds, multipliers)
• Each rule that could have been included or excluded
• The universe
• The date range
• The exit design
• The position-sizing rule
• The ranking rule
• The costs assumed

A “two-parameter system” that also involved choosing between three exit designs on two universes has twelve times the search space its author is claiming.

✕Common mistake

The specific version of this that catches careful people: choosing the date range after seeing results on the full history. “The data before 2008 is not representative of the modern market” is a defensible statement made in advance and a form of curve fitting made afterwards. The same words, in a different order, are a different research process.

How much is too much?

There is no threshold, and anyone who gives you one is selling something. But there are two questions that scale correctly with your situation.

How many independent observations does your data contain? Not rows. Not bars. Independent events. A 20-year daily backtest of 200 correlated stocks with an average 30-bar hold does not contain 1,000,000 independent observations; it may contain a few hundred genuinely distinct market episodes. If you evaluated 3,600 specifications against a few hundred effective observations, you should expect the best of them to look good for no reason at all.

Does the result survive small changes to the parameters? A genuine effect should show a broad plateau on the parameter surface — neighbouring values describe nearly the same rule, so they should behave nearly the same way. A single spike surrounded by mediocrity is the signature of a fit to noise.

The null model: what would nothing have produced?

This is the most useful single tool in the lesson, and it is much simpler than a statistical test.

The question “is 14% a year good?” has no answer without knowing what a system carrying no information at all would have returned on the same data, with the same universe, the same costs, the same sizing and the same number of trades.

So build that system and find out.

Complete runnable AFL

random-signal-benchmark.afl

// random-signal-benchmark.afl
// Part 30 - Data Snooping, Curve Fitting and Overfitting
//
// PURPOSE
//   "My system returned X" is not a finding until you know what a system with
//   no information at all would have returned on the same data, with the same
//   universe, the same costs, the same position sizing and the same number of
//   trades. This formula is that null model: entries are drawn from a random
//   number generator, everything else matches your real system.
//
//   Run it twenty or thirty times, changing only the seed, and write down the
//   metric you care about each time. That set of numbers is the distribution
//   your real result has to stand out from. Most of the time it does not.
//
// HOW TO RUN
//   1. Measure your real rule's signal rate: signals divided by bars tested.
//      signal-execution-audit.afl prints the count you need.
//   2. Set SignalRate below to that number, and HoldBars to your real system's
//      Avg. Bars Held.
//   3. Match the cost, sizing and universe settings to your real backtest.
//   4. Backtest once per seed. Record Annual Return %, Max. system % drawdown
//      and CAR/MaxDD each time.
//
// ============================ ASSUMPTIONS =============================
//   Universe        the same universe as the system under test, including the
//                   same survivorship problems. A random benchmark run on a
//                   survivor-only universe will also look good, which is
//                   exactly the diagnostic you want.
//   Periodicity     Daily.
//   Entry           a Bernoulli draw per bar at probability SignalRate, seeded
//                   per symbol so that different symbols get different draws
//                   and the whole run is reproducible from the seed.
//   Fill            next bar's Open, delays 1, plus/minus SlippagePct.
//   Commission      CommissionPct of trade value on each leg.
//   Exit            a fixed N-bar stop, so holding period is controlled rather
//                   than being another free parameter.
//   Liquidity       the same turnover floor as the real system.
//   What it is not  this is a null model for YOUR data and YOUR costs. It is
//                   not a significance test, it does not produce a p-value,
//                   and the draws are independent in a way that real signals
//                   are not. Read it as "could noise have done this?", not as
//                   "this is statistically significant".
// ======================================================================

Seed          = Param( "Random seed", 1, 1, 10000, 1 );
SignalRate    = Param( "Entry probability per bar", 0.01, 0.0005, 0.2, 0.0005 );
HoldBars      = Param( "Bars held per trade", 20, 1, 250, 1 );

AccountSize   = Param( "Assumed account size", 100000, 10000, 10000000, 10000 );
PosQty        = Param( "Max open positions", 10, 1, 50, 1 );
CommissionPct = Param( "Commission per trade (%)", 0.10, 0, 1.00, 0.01 );
SlippagePct   = Param( "Slippage per fill (%)", 0.15, 0, 2.00, 0.01 );
MinTurnover   = Param( "Min 50-bar turnover", 1000000, 0, 50000000, 250000 );

// ---------------------------------------------------------------------
// 1. Portfolio, costs and execution - identical to the system under test
// ---------------------------------------------------------------------
SetOption( "InitialEquity", AccountSize );
SetOption( "MaxOpenPositions", PosQty );
SetOption( "AllowPositionShrinking", True );
SetOption( "AllowSameBarExit", False );
SetOption( "CommissionMode", 1 );
SetOption( "CommissionAmount", CommissionPct );
SetOption( "UsePrevBarEquityForPosSizing", True );

SetTradeDelays( 1, 1, 1, 1 );

BuyPrice   = Open * ( 1 + SlippagePct / 100 );
SellPrice  = Open * ( 1 - SlippagePct / 100 );
ShortPrice = Open * ( 1 - SlippagePct / 100 );
CoverPrice = Open * ( 1 + SlippagePct / 100 );

PositionSize = -100 / PosQty;

// ---------------------------------------------------------------------
// 2. The null signal. mtRandomA returns an array of values in [0,1).
//    The seed is offset by the symbol's ordinal number so that symbols do not
//    all receive the identical draw, while the whole run stays reproducible.
// ---------------------------------------------------------------------
SymbolOffset = Status( "stocknum" );
Draw         = mtRandomA( Seed * 1000 + SymbolOffset );

Turnover = MA( Close * Volume, 50 );
Tradable = Turnover >= MinTurnover AND Volume > 0 AND Close > 0;

Buy  = Draw < SignalRate AND Tradable;
Sell = 0;                       // the N-bar stop below is the only exit

// Ranking must not smuggle information back in. A constant score means the
// engine keeps its own ordering rather than preferring anything.
PositionScore = 1;

ApplyStop( stopTypeNBar, stopModeBars, HoldBars );

_SECTION_BEGIN( "Random benchmark view" );
Plot( Close, "Close", colorDefault, styleCandle );
PlotShapes( Buy * shapeUpArrow, colorGreen, 0, Low );
_SECTION_END();

Download random-signal-benchmark.afl98 lines

Entries are drawn from a random number generator. Everything else — universe, costs, slippage, delays, position sizing, holding period, liquidity floor — matches your real system.

How to use it

1. Measure your real rule’s signal rate: entry signals divided by bars tested. The signal execution audit prints the count you need, or a Scan will.
2. Set SignalRate to that number and HoldBars to your real system’s Avg. Bars Held.
3. Match every cost, sizing and universe setting to the real backtest.
4. Run it twenty or thirty times, changing only the seed. Record Annual Return %, Max. system % drawdown and CAR/MaxDD each time.

That set of numbers is the distribution your real result has to stand out from.

=Expected result

The result that surprises people: on a universe and period where the market rose, random entries with a fixed holding period usually make money. Often quite a lot of it. Because being long a rising market at random times is being long a rising market.

If your real system’s Annual Return sits comfortably inside the spread of thirty random runs, the system has not demonstrated that it knows anything. That is a genuinely useful thing to find out before you commit money to it.

!Important

Two things this is not.

It is not a significance test and it does not produce a p-value. The draws are independent in a way that real trading signals are not, and the null model shares your data’s survivorship problems.

And a random benchmark run on a survivor-only universe will also look good — which is exactly the diagnostic you want. If both your system and pure noise look good on your universe, the universe is doing the work.

Read it as “could noise have done this?”, not as “this is statistically significant”.

One implementation detail worth borrowing

Fragment — not a complete formula

SymbolOffset = Status( "stocknum" );
Draw         = mtRandomA( Seed * 1000 + SymbolOffset );

mtRandomA( seed ) is the array version of mtRandom, returning values in [0,1). Offsetting the seed by the symbol’s ordinal number means different symbols receive different draws — without it every symbol would signal on the same bars, which is a very different null model — while the whole run stays reproducible from one seed. Reproducibility matters: a null model you cannot re-run is a null model you cannot check.

Note also PositionScore = 1. A constant score means the engine cannot prefer anything, which is the point: a rank would smuggle information back into a model that is supposed to have none.

Defences, in order of how much they buy you

1. Write the specification down before testing it. Including the universe, the period and the costs. Everything you change afterwards is a documented change rather than an invisible one.

2. Keep the count. Log every specification evaluated. Report results as “the best of N”.

3. Prefer fewer degrees of freedom. Every parameter you can justify removing is a parameter that cannot be fitted. A rule with two parameters that works is worth more than one with six that works better on this sample.

4. Check the parameter surface. Broad plateau, or single spike? Part 31 builds this properly.

5. Hold out data you have not looked at. Genuinely have not looked at. Part 32 is out-of-sample and walk-forward testing, and it is the only defence on this list that tests the process rather than the model.

6. Compare against the null model. As above.

⚠Warning

The catch with the hold-out defence, and it is a serious one: the moment you look at the out-of-sample result and go back to change the model, that data is no longer out-of-sample. It has joined the training set. You cannot get it back, and doing this twice is how walk-forward testing gets quietly converted into an elaborate form of in-sample fitting.

This is why the log matters more than any single technique. The log is the only record of how many times you have looked.

▷Exercise

Take the last system you built and count honestly. Every parameter value tried, every rule added or removed, every universe, every date range. Multiply.

Then run the random-signal benchmark twenty times on the same universe, period and costs, and write down the twenty results. Put your system’s number next to them.

Most people find that the count is far higher than they expected and the separation far smaller. That is the point of the exercise, and it is much cheaper to learn here than with money.

Recap

Data snooping is about your whole process, curve fitting is about your model, overfitting is the outcome. The count that matters is every specification evaluated against this dataset, and it is routinely ten or a hundred times larger than the author believes. Degrees of freedom include the universe, the range and the rules you tried and discarded, not only the numeric parameters. There is no threshold for “too many”, but there are two good questions: how many independent observations does the data contain, and does the result survive small parameter changes? And the cheapest reality check available is a null model — random entries, everything else identical — which frequently makes money and tells you exactly how much of your result was never yours.

Check your understanding

Question 1. You optimise two parameters over 10 values each, having previously tried three exit rules, two universes and two date ranges. How many specifications have been evaluated against this dataset?

• 2, because the final formula has two parameters
• 100, the size of the optimization grid
• 1,200 — the grid multiplied by every other choice that could have been made differently
• It cannot be counted

Show the answer and why

Answer: 1,200 — the grid multiplied by every other choice that could have been made differently

10 x 10 x 3 x 2 x 2 = 1,200. The abandoned attempts count, because the best result you eventually report was selected from all of them. This is why the research log matters: without it, the count is invisible even to you.

Question 2. Your random-signal benchmark, run thirty times with different seeds, produces annual returns between 6% and 15%. Your real system returned 12%. What follows?

• The system is profitable and should be traded
• The system's result sits comfortably inside what noise produced on the same data, so it has not demonstrated that it knows anything
• The random benchmark is faulty, since random entries should lose money
• The system needs more parameters

Show the answer and why

Answer: The system's result sits comfortably inside what noise produced on the same data, so it has not demonstrated that it knows anything

Random long entries in a rising market make money, because being long a rising market is what produced the return. A result inside the null distribution is a result the null model already explains. Nothing here proves the system is worthless — it means this test did not distinguish it from noise.

Question 3. Which of these are degrees of freedom that should be counted? Select all that apply.

• The choice of universe
• The start date of the test
• A volatility filter that was added, removed, and added again with a different threshold
• The exit design chosen from three candidates

Show the answer and why

Answer: The choice of universe, The start date of the test, A volatility filter that was added, removed, and added again with a different threshold, The exit design chosen from three candidates

All of them. Anything that could have been chosen differently, and was chosen partly because of what the results looked like, is a degree of freedom — whether or not it appears as a number in the final formula.

Question 4. Why does the random benchmark set PositionScore = 1?

PositionScore = 1;

• To make the backtest run faster
• A constant score means the engine cannot prefer any candidate, so no information is smuggled into a model that is supposed to carry none
• Because PositionScore must be positive
• To ensure every signal is taken

Show the answer and why

Answer: A constant score means the engine cannot prefer any candidate, so no information is smuggled into a model that is supposed to carry none

A ranking rule is itself a strategy. If the null model ranked candidates by liquidity or momentum it would no longer be a null model — it would be testing the ranking rule with random entries, which is a different and also interesting experiment.

Question 5. You examine your out-of-sample result, find it disappointing, adjust the model and re-test. What has happened?

• Nothing — that is the normal iterative process
• The out-of-sample data has joined the training set and can no longer serve as a hold-out, and doing this repeatedly converts walk-forward testing into elaborate in-sample fitting
• The result becomes more reliable because it now reflects both periods
• The out-of-sample period must simply be extended

Show the answer and why

Answer: The out-of-sample data has joined the training set and can no longer serve as a hold-out, and doing this repeatedly converts walk-forward testing into elaborate in-sample fitting

A hold-out is only a hold-out while it is unlooked-at. Once a result from it has influenced the model, it is training data. Extending the period does not recover it. The research log is the only record of how many times you have looked, which is why it is the defence that everything else depends on.

Check my answersClear