Insufficient Evidence and Regime Dependence

“How many trades do I need?” is the question everybody asks, and it does not have a number for an answer. What it has is a better question underneath it: how many genuinely independent things did I observe, and did they include more than one kind of market?

Why there is no single number

You will see 30 trades quoted, and 100, and 200, and 1,000. All of them are wrong as universal answers, because the amount of evidence a given number of trades represents depends on three things none of the rules mention.

How variable the outcomes are. A rule whose trades cluster between +1% and +3% tells you something after twenty. A rule whose trades range from −40% to +180% tells you very little after two hundred, because a handful of extreme outcomes dominate every average you compute.

How independent the trades are. Ten positions entered on the same day on the same market-wide signal are close to one observation. A backtest that reports 400 trades on 40 correlated symbols over ten years may contain a few dozen genuinely distinct episodes.

How many things you tried before this one. A first-guess rule with 50 trades is much stronger evidence than the best of 3,600 specifications with 50 trades each. The data-snooping lesson is the other half of this one.

∴Research note

A useful reframing: instead of “do I have enough trades?”, ask “how much would the result change if I removed the five best trades?”

If it collapses, the result is those five trades, and you have five observations however many rows the trade list has. That test takes two minutes — sort the trade list by profit, remove the top five, and recompute — and it is more informative than any rule of thumb about counts.

One bull market is one observation

This is the error that survives every other correction.

A backtest that spans a long, largely rising market has tested your rules against one market. It does not matter that it covers fifteen years and 4,000 bars and 800 trades. There was one broad environment, and every trade is an observation drawn from it.

The consequence is that you have learned “these rules did well in that environment”, and you have learned nothing about what happens in a different one — which is precisely the question that matters, because the environment will change and you will not be told when.

✕Common mistake

The specific version that catches almost everyone: a long-only trend system tested over a period that rose. It will look excellent, because being long a rising market is what produced the result. The rules may have contributed nothing.

The random-signal benchmark is the direct test for this, and the regime split below is the diagnostic.

Splitting the evidence by regime

A backtest report gives you one number per metric for the whole period. That number hides the thing you most need: whether the rule was tested in more than one kind of market, and how many observations landed in each kind.

Complete runnable AFL

regime-split-evidence.afl

// regime-split-evidence.afl
// Part 30 - Insufficient Evidence and Regime Dependence
//
// PURPOSE
//   A backtest report gives you one number per metric for the whole period.
//   That number hides the thing you most need to know: whether the rule was
//   tested in more than one kind of market, and how many observations landed
//   in each kind. This exploration splits every signal into four market
//   regimes defined by a reference index, and reports the count and the mean
//   forward outcome in each.
//
//   The usual finding is not that one regime is better. It is that three of
//   the four cells contain too few observations to support any statement at
//   all - which is itself the result.
//
// HOW TO RUN
//   Analysis -> Apply to: your universe.  Periodicity: Daily.
//   Range: All quotations.  Press EXPLORE. One row per symbol.
//
// ============================ ASSUMPTIONS =============================
//   Regime          defined by a REFERENCE INDEX, not by the traded symbol, so
//                   that every symbol is classified the same way on the same
//                   date. Set the ticker in the parameter below; it must exist
//                   in the database or every regime column will be empty.
//   Trend axis      index close above / below its own TrendPeriod-bar simple
//                   moving average, both known at that bar's close.
//   Volatility axis a 20-bar average true range of the index, as a percentage
//                   of the index close, above / below its own VolPeriod-bar
//                   average. Also known at that bar's close.
//   Outcome         plain percentage change of the traded symbol's Close over
//                   Horizon bars. No costs, no slippage, no position sizing,
//                   no stops. This measures association, not tradability.
//   Independence    forward windows overlap, so consecutive observations are
//                   correlated and the effective sample is smaller than the
//                   count printed. Do not treat the counts as independent
//                   trials.
//   Alignment       Foreign() aligns the index to the traded symbol's bars.
//                   If your data has holes, consider Settings -> General ->
//                   "Pad and align to reference symbol".
// ======================================================================

SetBarsRequired( -2, -2 );   // -2 is sbrAll: the running totals need every bar

RefSymbol   = ParamStr( "Reference index symbol", "^GSPC" );
TrendPeriod = Param( "Index trend MA (bars)", 200, 50, 300, 10 );
VolPeriod   = Param( "Index volatility baseline (bars)", 250, 50, 500, 10 );
SignalMA    = Param( "Signal MA (bars)", 50, 5, 200, 5 );
Horizon     = Param( "Forward window (bars)", 20, 1, 120, 1 );
MinTurnover = Param( "Min 50-bar turnover", 1000000, 0, 50000000, 250000 );

// ---------------------------------------------------------------------
// 1. The reference index, read field by field. Foreign() returns one field per
//    call, so the true range is assembled by hand rather than with ATR(),
//    which would operate on the traded symbol's own price arrays.
// ---------------------------------------------------------------------
IndexClose = Foreign( RefSymbol, "C" );
IndexHigh  = Foreign( RefSymbol, "H" );
IndexLow   = Foreign( RefSymbol, "L" );

IndexPrevClose = Ref( IndexClose, -1 );
IndexTrueRange = Max( IndexHigh, IndexPrevClose ) - Min( IndexLow, IndexPrevClose );
IndexAtrPct    = 100 * SafeDivide( MA( IndexTrueRange, 20 ), IndexClose, Null );

HaveIndex = NOT IsNull( IndexClose ) AND IndexClose > 0;

TrendUp = HaveIndex AND IndexClose > MA( IndexClose, TrendPeriod );
VolHigh = HaveIndex AND IndexAtrPct > MA( IndexAtrPct, VolPeriod );

// ---------------------------------------------------------------------
// 2. The signal being examined, and the outcome that followed it.
//    Ref() with a POSITIVE shift reads bars that had not printed yet. That is
//    correct here, because this is a measurement of what followed, and it must
//    never appear in a Buy expression.
// ---------------------------------------------------------------------
Turnover = MA( Close * Volume, 50 );
Signal   = Cross( Close, MA( Close, SignalMA ) )
       AND Turnover >= MinTurnover
       AND Volume > 0;

FwdReturn = 100 * SafeDivide( Ref( Close, Horizon ) - Close, Close, Null );

LastBarIndex     = LastValue( BarIndex() );
HasForwardWindow = BarIndex() <= LastBarIndex - Horizon;

Measurable = Status( "barinrange" )
         AND HaveIndex
         AND HasForwardWindow
         AND NOT IsNull( FwdReturn );

Observation = Measurable AND Signal;

// ---------------------------------------------------------------------
// 3. Four regimes. Every observation lands in exactly one of them.
// ---------------------------------------------------------------------
RegimeUpQuiet   = Observation AND TrendUp     AND NOT VolHigh;
RegimeUpWild    = Observation AND TrendUp     AND VolHigh;
RegimeDownQuiet = Observation AND NOT TrendUp AND NOT VolHigh;
RegimeDownWild  = Observation AND NOT TrendUp AND VolHigh;

CountUpQuiet   = Cum( RegimeUpQuiet );
CountUpWild    = Cum( RegimeUpWild );
CountDownQuiet = Cum( RegimeDownQuiet );
CountDownWild  = Cum( RegimeDownWild );

// Nz() is deliberate: FwdReturn is Null outside the measurable window, and one
// Null entering a running total destroys every value after it.
SumUpQuiet   = Cum( IIf( RegimeUpQuiet,   Nz( FwdReturn ), 0 ) );
SumUpWild    = Cum( IIf( RegimeUpWild,    Nz( FwdReturn ), 0 ) );
SumDownQuiet = Cum( IIf( RegimeDownQuiet, Nz( FwdReturn ), 0 ) );
SumDownWild  = Cum( IIf( RegimeDownWild,  Nz( FwdReturn ), 0 ) );

MeanUpQuiet   = SafeDivide( SumUpQuiet,   CountUpQuiet,   Null );
MeanUpWild    = SafeDivide( SumUpWild,    CountUpWild,    Null );
MeanDownQuiet = SafeDivide( SumDownQuiet, CountDownQuiet, Null );
MeanDownWild  = SafeDivide( SumDownWild,  CountDownWild,  Null );

TotalObs  = CountUpQuiet + CountUpWild + CountDownQuiet + CountDownWild;
TotalSum  = SumUpQuiet + SumUpWild + SumDownQuiet + SumDownWild;
MeanAll   = SafeDivide( TotalSum, TotalObs, Null );

// How lopsided is the evidence? The share of all observations that fell into
// the single most populated regime. Close to 100 means the rule has effectively
// been tested once, in one kind of market.
Largest        = Max( Max( CountUpQuiet, CountUpWild ),
                      Max( CountDownQuiet, CountDownWild ) );
ConcentrationP = 100 * SafeDivide( Largest, TotalObs, Null );

// How much of the test period each regime occupied, regardless of signals.
BarsMeasurable = Cum( Measurable );
BarsUpQuiet    = Cum( Measurable AND TrendUp     AND NOT VolHigh );
BarsUpWild     = Cum( Measurable AND TrendUp     AND VolHigh );
BarsDownQuiet  = Cum( Measurable AND NOT TrendUp AND NOT VolHigh );
BarsDownWild   = Cum( Measurable AND NOT TrendUp AND VolHigh );

// ---------------------------------------------------------------------
// 4. One row per symbol, written on the last bar of the range.
// ---------------------------------------------------------------------
Filter = Status( "lastbarinrange" ) AND TotalObs > 0;

AddColumn( TotalObs, "Signals", 1.0 );
AddColumn( MeanAll,  "Mean fwd % all", 1.2 );

AddColumn( CountUpQuiet,   "N up/quiet",   1.0 );
AddColumn( MeanUpQuiet,    "Fwd % up/quiet", 1.2 );
AddColumn( CountUpWild,    "N up/wild",    1.0 );
AddColumn( MeanUpWild,     "Fwd % up/wild", 1.2 );
AddColumn( CountDownQuiet, "N down/quiet", 1.0 );
AddColumn( MeanDownQuiet,  "Fwd % down/quiet", 1.2 );
AddColumn( CountDownWild,  "N down/wild",  1.0 );
AddColumn( MeanDownWild,   "Fwd % down/wild", 1.2 );

AddColumn( ConcentrationP, "% of signals in biggest regime", 1.1 );

AddColumn( BarsMeasurable, "Bars measured", 1.0 );
AddColumn( 100 * SafeDivide( BarsUpQuiet,   BarsMeasurable, Null ), "% bars up/quiet", 1.1 );
AddColumn( 100 * SafeDivide( BarsUpWild,    BarsMeasurable, Null ), "% bars up/wild", 1.1 );
AddColumn( 100 * SafeDivide( BarsDownQuiet, BarsMeasurable, Null ), "% bars down/quiet", 1.1 );
AddColumn( 100 * SafeDivide( BarsDownWild,  BarsMeasurable, Null ), "% bars down/wild", 1.1 );

// COUNT and AVERAGE rows. The AVERAGE row weights every symbol equally: a
// symbol with four signals counts as much as one with four hundred. To pool
// properly, export the table and weight each mean by its own signal count.
AddSummaryRows( 2 | 16, 1.2 );

Download regime-split-evidence.afl163 lines

It classifies every signal into four regimes on two axes — index trending up or down, index volatility high or low — and reports the count and mean forward outcome in each.

Why the regime is defined by a reference index

Fragment — not a complete formula

IndexClose = Foreign( RefSymbol, "C" );
TrendUp    = HaveIndex AND IndexClose > MA( IndexClose, TrendPeriod );

Not by the traded symbol. If each symbol were classified by its own trend, “up regime” would mean something different for every row and the cells would not be comparable. A single reference index classifies every symbol the same way on the same date, which is what makes the split meaningful.

Two implementation notes worth borrowing.

Foreign() returns one field per call, so the index’s true range has to be assembled by hand rather than by calling ATR() — ATR() would operate on the traded symbol’s price arrays, which would be a subtle and completely silent bug.

And Foreign() aligns the index to the traded symbol’s bars. If your data has holes, Settings → General → “Pad and align to reference symbol” is the documented fix; without it, missing bars in one series shift the alignment.

The number to read first

Fragment — not a complete formula

ConcentrationP = 100 * SafeDivide( Largest, TotalObs, Null );

The share of all observations that fell into the single most populated regime.

Close to 100 means the rule has effectively been tested once, in one kind of market — regardless of how many trades the report shows. That single figure is more informative about the strength of your evidence than the trade count, the win rate or the Sharpe ratio.

=Expected result

The usual finding is not that one regime is better than another. It is that three of the four cells contain too few observations to support any statement at all.

That is itself the result, and it is worth stating explicitly: “the rule produced 340 signals, of which 290 occurred in a rising, low-volatility index environment; the other three regimes contain 50 signals between them, which is not enough to say anything about them.”

That sentence is a genuine research finding. “The system returned 14% a year” is not.

Note also the two Nz() calls in the running sums. FwdReturn is Null outside the measurable window, and a single Null entering a Cum() destroys every value after it — which would silently empty the entire column rather than producing an obviously wrong number.

Testing across regimes deliberately

The exploration is diagnostic. Three ways to act on what it tells you, in increasing order of effort:

Split the date range and re-run. Take three or four disjoint multi-year spans covering visibly different conditions and run the same backtest on each. Compare results. This is crude, it uses no new machinery, and it is often decisive.

Test on a different universe or a different market. A rule that describes something general should not stop working when you change the instruments. A rule that only works on one universe is telling you about that universe.

Walk-forward. Part 32’s walk-forward methodology tests the process rather than a single parameter set, across many consecutive out-of-sample windows. It is the strongest of the three and the most work.

Confidence intervals, in plain language

You do not need the formulas to use the idea, and the idea is what matters.

Suppose your system’s average trade is +0.8%, and the individual trades vary widely — a standard deviation of, say, 9%. The average of a sample is itself uncertain, and roughly speaking that uncertainty shrinks with the square root of the sample size. With 100 trades, the uncertainty in your estimate of the mean is around 0.9% — larger than the mean itself.

The plain-language conclusion: your best estimate of the average trade is 0.8%, and the data is consistent with the true value being anywhere from clearly negative to clearly positive. You have not established that the system makes money.

Three cautions before you reach for a calculator, and they matter more than the arithmetic:

• The trades are not independent, so the effective sample is smaller than the count and the real uncertainty is larger than any standard formula will tell you.
• The distribution is not symmetric. Trading returns have long tails, and interval formulas built on symmetry understate the extremes.
• The sample was selected by your search process. An interval computed on the best of 3,600 specifications does not mean what it would mean on a pre-registered one.

!Important

Use the idea, not the false precision. The useful takeaway is directional: if the spread of individual outcomes is large relative to the mean, you need far more evidence than your trade count suggests — and there is no arithmetic that can rescue a sample of correlated trades from one market regime.

What to say when the evidence is weak

This is a skill, and it is the one that separates research from marketing.

Say what you measured. “Over this universe, this period, with these costs, the system produced X.” Every qualifier belongs in the sentence.

Say how many independent observations you think you have, and why that number differs from the trade count.

Say which regimes were represented, with counts, and which were not.

Say what would change your mind. If you cannot name a result that would make you abandon the idea, you were not testing it.

Say “I do not know” when that is the answer. Weak evidence is a legitimate finding. A great deal of published technical-analysis material exists because somebody could not bring themselves to report one.

?Reality check

There is a strong professional and personal incentive to convert “the evidence is inconclusive” into “the evidence is positive”. It is not usually dishonesty; it is the natural result of having spent three weeks on something.

The only reliable defence is to write down, in advance, what result would count as an answer — and then to hold yourself to it. That is what the research log is actually for, and it is why the log matters more than any statistical technique in this part.

▷Exercise

Run the regime split on the last system you built, using a reference index that exists in your database.

Write down three numbers: the total signal count, the count in the largest regime, and the concentration percentage. Then write one sentence describing what your backtest actually established, with every qualifier included.

If that sentence is much weaker than what you would have said before running the exploration, the exploration did its job.

Recap

There is no sufficient number of trades, because the amount of evidence depends on how variable the outcomes are, how independent they are, and how many specifications you tried before this one. One bull market is one observation no matter how many bars it contains. Splitting signals by an index-defined regime shows you how lopsided the evidence is, and the concentration figure — the share of observations in the largest regime — tells you more about the strength of your test than any performance metric. Confidence intervals are worth understanding directionally and not worth computing precisely, because the assumptions they rest on are all violated here. And “the evidence is weak” is a result you should be able to write down and publish to yourself without flinching.

Check your understanding

Question 1. A backtest reports 400 trades across 40 symbols over ten years. Why might that be far less evidence than it appears?

• Because 400 is below the accepted minimum of 1,000
• Because trades on correlated symbols entered on the same market-wide signal are close to one observation, so the effective sample may be a few dozen distinct episodes
• Because ten years is too short for any conclusion
• Because AmiBroker counts scale-ins as separate trades

Show the answer and why

Answer: Because trades on correlated symbols entered on the same market-wide signal are close to one observation, so the effective sample may be a few dozen distinct episodes

The count is the number of rows, not the number of independent things observed. Correlation across symbols and clustering in time both reduce the effective sample, and no statistical formula applied to the row count will correct for it.

Question 2. The regime exploration defines the regime using a reference index rather than each traded symbol. Why?

IndexClose = Foreign( RefSymbol, "C" );
TrendUp    = HaveIndex AND IndexClose > MA( IndexClose, TrendPeriod );

• Because Foreign() is faster than computing a moving average per symbol
• Because a single reference classifies every symbol the same way on the same date, so the cells are comparable — per-symbol classification would mean something different in every row
• Because individual symbols do not have trends
• Because the index has cleaner data

Show the answer and why

Answer: Because a single reference classifies every symbol the same way on the same date, so the cells are comparable — per-symbol classification would mean something different in every row

The split is meant to partition time into kinds of market. If each row used its own definition of "up regime", the columns would be aggregating incomparable things and the counts would be meaningless.

Question 3. The exploration reports a concentration figure of 92%. What does that tell you?

• The system wins 92% of the time
• 92% of all signals occurred in a single market regime, so the rule has effectively been tested once, in one kind of market — regardless of the trade count
• 92% of the observations were measurable
• The system was in the market 92% of the time

Show the answer and why

Answer: 92% of all signals occurred in a single market regime, so the rule has effectively been tested once, in one kind of market — regardless of the trade count

It is a measure of how lopsided the evidence is, and it is more informative about the strength of the test than any performance metric. A high concentration means the other regimes contain too few observations to support any statement about them.

Question 4. Why do the running sums wrap FwdReturn in Nz()?

SumUpQuiet = Cum( IIf( RegimeUpQuiet, Nz( FwdReturn ), 0 ) );

• To convert percentages into decimals
• Because FwdReturn is Null outside the measurable window, and one Null entering a Cum() destroys every value after it
• Because Cum() cannot accept an IIf() expression
• To exclude negative returns from the total

Show the answer and why

Answer: Because FwdReturn is Null outside the measurable window, and one Null entering a Cum() destroys every value after it

Null propagates through a running total, so a single empty value would silently empty the whole column from that bar onward. This is one of the most common silent bugs in cross-sectional AFL work.

Question 5. Your average trade is +0.8% with a standard deviation of 9% across 100 trades. Which statements are sound? Select all that apply.

• The uncertainty in the estimated mean is of the same order as the mean itself, so the data is consistent with the true value being negative
• The real uncertainty is larger than any standard formula suggests, because the trades are not independent
• A 100-trade sample settles the question because it exceeds the usual 30-trade minimum
• If the sample was the best of many specifications tried, an interval computed on it does not mean what it would on a pre-registered test

Show the answer and why

Answer: The uncertainty in the estimated mean is of the same order as the mean itself, so the data is consistent with the true value being negative, The real uncertainty is larger than any standard formula suggests, because the trades are not independent, If the sample was the best of many specifications tried, an interval computed on it does not mean what it would on a pre-registered test

The uncertainty in a sample mean shrinks with the square root of the sample size, so 9% over 100 trades leaves roughly 0.9% of uncertainty — larger than the 0.8% mean. Dependence and selection both make the true picture worse, not better, and no fixed trade-count threshold addresses either.

Check my answersClear