Breadth Divergence and Market Regime

This lesson takes the most-repeated claim about breadth, shows exactly why it is hard to test, and then builds the thing that can be tested: a state label that could have been computed at the time, from data available at the time, with no decisions taken after the fact. By the end you will be able to define such a regime, use it as a filter, and — the part that matters most — count how many independent observations of it your history actually contains.

The claim

?Reality check

The claim, as it is usually stated. When an index makes a new high but fewer of its members are participating — fewer advancing, fewer above their moving averages, fewer making new highs than at the previous peak — the advance is “narrowing”, and a decline follows. The divergence between the index and the breadth measure is said to be a warning.

This claim is not obviously silly. There is a mechanism behind it, and the mechanism is arithmetic rather than mystical: a capitalisation-weighted index can be dragged upward by a small number of large constituents, so an index high accompanied by a low breadth reading really does describe a market where most members are not rising. Lesson 1 established that much, and none of it is in dispute.

What is in dispute is the second half — and a decline follows. That is a claim about what happens next, and it needs a different kind of support.

What would have to be true

Turn the claim into something a computer could evaluate and you immediately have to answer five questions the claim itself leaves open:

1. Which breadth measure? Percentage above the 200-bar average, net new highs, the A/D line and the advancing count do not diverge at the same times. Choosing after you have looked at the chart is choosing the answer.
2. What is a “new high” for the index? Highest close in fifty-two weeks? Highest close ever? Any close within one per cent of the prior peak?
3. What is “fewer”? Lower than at the previous index peak — but which previous peak, and by how much? A reading of 61 against 63 is lower; is it a divergence?
4. Over what horizon does the decline have to arrive? Two weeks, three months, a year? Without a horizon, every divergence is eventually vindicated, because markets eventually fall.
5. Compared with what? The relevant number is not “how often did a decline follow a divergence” but “how much more often than after a comparable non-divergent high”. Without the base rate the figure is uninterpretable.

Answering these five questions honestly is most of the work. It is also the reason so much published breadth analysis consists of annotated charts rather than tables of results.

Why divergence is easy to see in hindsight

Three separate mechanisms make a divergence look obvious after the fact and be very hard to call before it.

You choose the peak knowing it was a peak. On a chart, the index high that preceded a large decline is visually unmistakable. At the time it was one of dozens of highs, most of which were followed by more highs. Any study that starts from “look at the top in February” has already used information from March.

The comparison window is chosen after seeing the answer. Divergence is defined relative to some earlier peak. Move the reference peak back three months and a divergence appears or disappears. If the reference is chosen by eye, it will be chosen — without any dishonest intent — to be the one that makes the picture clearest.

There is no failure condition. A rule that says “a decline follows” without saying when cannot be wrong. Compare it with a rule that says “a decline of at least ten per cent begins within sixty trading days”: that one can be, and frequently is, falsified.

The same history, two ways of looking at it

What hindsight sees

ignoredthe divergencethe decline

What you had at the time

candidate divergences, outcome unknownstill unknown

Hindsight selects one episode from a series of similar-looking ones. A rule has to treat them all alike.

∴Research note

The design that answers the question is the one used throughout Part 12’s reality checks: define the condition mechanically, apply it to every bar in the history without exception, measure a fixed forward outcome, and compare that outcome with the unconditional base rate over the same period and universe. Anything less produces a collection of illustrations rather than evidence.

From divergence to regime

There is a version of this idea that survives the criticism above, and it is worth having. Instead of trying to identify a turning point — an event, defined by comparison with a peak you can only recognise afterwards — define a state that you are either in or not in on any given bar, using only that bar and earlier ones.

That is what a market regime is. “More than sixty per cent of members are above their own 200-bar average” is a statement you can evaluate today, with today’s data, and it will not be revised tomorrow. It makes no claim about what happens next. It is a classification, and it can be used as a gate: run this system only while the classification holds.

Defining one you can live with

A single threshold produces an unusable classification. If the rule is “broad when the reading is above fifty”, then a reading oscillating around fifty produces a new regime every few days, and no one could act on it.

The fix is two thresholds and a latch: enter the broad state when the reading rises above a higher level, leave it only when the reading falls below a lower one, and change nothing in between. That gap is hysteresis. AFL expresses it in one call:

Fragment — not a complete formula

EnterBroad  = PctAboveSlow >= 60;
LeaveBroad  = PctAboveSlow <= 40;
BroadRegime = Flip( EnterBroad, LeaveBroad );

Flip() returns 1 from the first true value in its first argument until a true value appears in its second, then zero until the first argument is true again. Two thresholds, one latch, no ambiguity about what state you are in on any bar.

The two levels are parameters, and parameters are where research goes wrong. Choosing 60 and 40 because they are round numbers is defensible. Choosing 63 and 37 because they produced the best-looking backtest is curve fitting, and you should expect the improvement to disappear out of sample.

The regime formula

Goal

Read the participation composite, classify every bar as broad or narrow with hysteresis, show the classification on a chart, and — in exploration mode — list every regime change so that the number of independent episodes is a number you have seen rather than one you have assumed.

Complete formula

Complete runnable AFL

breadth-regime.afl

/*  Breadth regime
    -------------------------------------------------------------------------
    Part 16 - Market Breadth. Turns one breadth composite into a two-state
    classification that a later formula can use as a gate, and - more usefully
    - lets you count how many independent observations that classification is
    actually built on.

    Requires the composites written by breadth-composite-builder.afl.

    THE DEFINITION
      Broad  = at least EnterLevel per cent of members are above their own
               200-bar average.
      Narrow = the reading has since fallen to ExitLevel per cent or below.
    Between the two levels the state does not change. That gap is hysteresis,
    and it exists so that a reading hovering around a single threshold does not
    generate dozens of one-day "regimes" that no one could have traded.

    WHAT THIS IS NOT
    A regime label is a description of the recent past, not a forecast. The
    exploration half exists to make its weakness visible: run it and count how
    few independent episodes a decade of data contains.

    HOW TO RUN IT
      As a chart, on ~BR_MEMBERS or on a broad index whose calendar matches
      the universe.
      As an EXPLORATION with Apply to = Current symbol and Range = All
      quotations, to list every regime change with its date.
*/

// ---- Configuration -------------------------------------------------------
Prefix     = "~BR_";
EnterLevel = Param( "Broad above (per cent)",  60, 50, 90, 1 );
ExitLevel  = Param( "Narrow below (per cent)", 40, 10, 50, 1 );

// ---- The measure ---------------------------------------------------------
Members   = Nz( Foreign( Prefix + "MEMBERS",  "V" ) );
AboveSlow = Nz( Foreign( Prefix + "ABOVE200", "V" ) );

PctAboveSlow = 100 * SafeDivide( AboveSlow, Members, 0 );
HaveData     = Members > 0;

// ---- The classification --------------------------------------------------
// Flip() latches on the first argument and releases on the second, which is
// the whole of the hysteresis rule in one call.
EnterBroad  = IsTrue( HaveData AND PctAboveSlow >= EnterLevel );
LeaveBroad  = IsTrue( HaveData AND PctAboveSlow <= ExitLevel );
BroadRegime = Flip( EnterBroad, LeaveBroad );

RegimeChanged = IsTrue( BroadRegime != Ref( BroadRegime, -1 ) );
BarsInRegime  = Nz( BarsSince( RegimeChanged ) ) + 1;

// How many separate broad episodes exist in the whole history. This number,
// not the number of bars, is the sample size of any claim about regimes.
BroadEpisodes  = Cum( IsTrue( BroadRegime AND NOT Ref( BroadRegime, -1 ) ) );
NarrowEpisodes = Cum( IsTrue( Ref( BroadRegime, -1 ) AND NOT BroadRegime ) );

// ---- Exploration half: one row per regime change -------------------------
if( Status( "action" ) == actionExplore )
{
    Filter = RegimeChanged AND HaveData;

    AddColumn( BroadRegime, "Broad (1) or narrow (0)", 1.0 );
    AddColumn( PctAboveSlow, "Per cent above 200-bar MA", 1.1 );
    AddColumn( Ref( BarsInRegime, -1 ), "Bars in previous state", 1.0 );
    AddColumn( BroadEpisodes, "Broad episodes so far", 1.0 );
    AddColumn( NarrowEpisodes, "Narrow episodes so far", 1.0 );
    AddColumn( Members, "Contributors", 1.0 );

    SetSortColumns( 2 );

    _exit();
}

// ---- Chart half ----------------------------------------------------------
_SECTION_BEGIN( "Breadth regime" );

SetBarsRequired( sbrAll, sbrAll );

// The band behind the line carries the same information as the title text, so
// nothing depends on the reader distinguishing two colours.
Plot( 100, "", IIf( BroadRegime, colorPaleGreen, colorRose ),
      styleArea | styleNoLabel | styleNoTitle, 0, 100, 0, -5 );

Plot( PctAboveSlow, "Per cent above 200-bar MA", colorBlue,
      styleLine | styleThick, 0, 100 );

PlotGrid( EnterLevel, colorDarkGreen );
PlotGrid( ExitLevel,  colorDarkRed );

if( LastValue( Members ) > 0 )
{
    StateText = WriteIf( BroadRegime, "BROAD", "NARROW" );
    Title = StrFormat( "%s   regime %s   reading %.1f per cent   "
                     + "%g bars in this state   %g broad episodes to date",
                       Name(), StateText, SelectedValue( PctAboveSlow ),
                       SelectedValue( BarsInRegime ),
                       SelectedValue( BroadEpisodes ) );
}
else
{
    Title = "NO COMPOSITE DATA - run breadth-composite-builder.afl as a Scan first";
}

_SECTION_END();

Download breadth-regime.afl104 lines

How it works

Members and AboveSlow come from the composites built by the project formula in the next lesson. SafeDivide turns them into a percentage while returning zero rather than an error wherever the member count is zero, which it is on every bar before any member became eligible.

The classification is the three lines above, wrapped in IsTrue() so that a Null cannot leak into the latch. Everything after it is bookkeeping: RegimeChanged marks bars where the state differs from the previous bar, BarsInRegime counts how long the current state has held, and the two Cum() counters accumulate the number of broad and narrow episodes seen so far.

The formula then splits. In exploration mode it filters to regime-change bars only and emits one row each, then _exit()s. Otherwise it draws the chart: a background band behind the line showing the state, the participation line itself, and grid lines at the two thresholds so you can see how close the current reading is to flipping.

Key functions

• Flip( array1, array2 ) is the latch. It is the cleanest expression of hysteresis in AFL, and it appears again in Part 27 when entry and exit states have to be tracked.
• SafeDivide( x, y, valueifzerodiv ) returns the third argument wherever the divisor is zero. All three arguments are required.
• BarsSince( array ) counts bars since the array was last true. Its behaviour before the first true value is not documented, so it is wrapped in Nz().
• Status("action") separates the exploration half from the chart half, exactly as actionScan separated the two halves of the composite builder.
• WriteIf( condition, truetext, falsetext ) returns one of two strings. It is used so the title states the regime in words: the colour band is a convenience, not the only carrier of the information.

Expected result

On the chart, a green band behind the periods when the participation line was above your upper threshold and had not since fallen below the lower one, a rose band otherwise, and a title that names the state, the current reading, how many bars the state has lasted, and how many broad episodes have occurred in the whole history.

=Expected result

That last number is the point of the exercise. On a daily database covering fifteen years, expect something in the range of a handful to a couple of dozen broad episodes, depending on your thresholds. Fifteen years of daily bars is roughly 3,800 bars. It is not 3,800 observations of anything.

Run the same file as an exploration on ~BR_MEMBERS with Range set to All quotations and you get one row per regime change, each showing how long the previous state lasted. Print it out. It is usually a short list.

Test it

Set both thresholds to the same value — say 50 and 50 — and re-run. The hysteresis disappears and the episode count should rise sharply, because the state now flips every time the reading crosses a single level. Restore the gap and watch the count fall. That is the hysteresis doing its job, and it is also a demonstration that the episode count is a function of your parameter choices as much as of the market.

Common errors

• Every bar is narrow. The composites do not exist or are empty, so the percentage is zero everywhere. The title says NO COMPOSITE DATA when the member count is zero on the last bar.
• The exploration returns nothing. Filter selects regime-change bars only, and there are very few. Set Range to All quotations.
• The upper threshold is below the lower one. Flip() will still run and will produce a state that latches on and never releases. The Parameters dialog does not prevent this.
• The bands and the line disagree. You changed a threshold after the chart was drawn but are looking at a cached pane; press Ctrl+R and confirm the parameter values.

Extension

Add a third state. Instead of broad and narrow, classify as broad, narrow, or deteriorating — broad but with the reading below where it stood twenty bars ago. Then look at how many independent episodes of the third state exist. The usual discovery is that adding states divides an already small sample rather than adding information.

Using a regime as a filter

The reason to build a regime is usually to gate something else:

Fragment — not a complete formula

// The composite must already have been built by a scan before this runs.
Members     = Nz( Foreign( "~BR_MEMBERS",  "V" ) );
AboveSlow   = Nz( Foreign( "~BR_ABOVE200", "V" ) );
BroadRegime = Flip( 100 * SafeDivide( AboveSlow, Members, 0 ) >= 60,
                    100 * SafeDivide( AboveSlow, Members, 0 ) <= 40 );

Buy = MySetup AND MyTrigger AND BroadRegime;

Three cautions attach to that one line, and all three are the kind of thing that turns a promising backtest into a wasted month.

The composite must exist before the backtest runs. A composite is stored data, produced by a scan. If you edit the composite definition and then immediately backtest, you are gating on the old composite. Build first, test second — by re-running the scan, by #pragma sequence, or from the Batch window.

A filter removes trades, and removing trades shrinks the sample. A system with 400 trades that keeps 180 after the regime filter has not necessarily improved; it has certainly become harder to evaluate. Report the trade count alongside every filtered result.

A filter that works by excluding one bad period has not been shown to work. If the regime gate’s entire contribution is that it was narrow through 2008, then you have one observation, and you already knew 2008 was bad. Part 30’s lesson on insufficient evidence and regime dependence is the companion to this paragraph.

⚠Warning

There is also a survivorship interaction that is easy to miss. Your regime is computed from composites built over the symbols in your database today. A backtest that runs over that same universe is therefore gated by a breadth series that shares its survivorship bias, so the two errors do not cancel — they compound. Say so in the assumptions block of any system that uses a breadth gate.

The sample-size problem, stated plainly

This is the lesson’s real conclusion, and it applies to every regime study, not only breadth.

A daily database of fifteen years contains roughly 3,800 bars. If a regime definition produces twelve broad episodes and twelve narrow ones, then any statement of the form “the system performs better in broad regimes” is supported by twelve observations of a broad regime, not 3,800. The bars within an episode are not independent: they share the same market, the same participants and, usually, the same trend.

Three consequences follow.

• The confidence you can have is much lower than the bar count suggests. A difference in average return between twelve episodes and twelve episodes is a difference you should expect to see fairly often by chance.
• Adding parameters is expensive. Each threshold you tune is fitted against those twenty-four episodes. Two thresholds against twenty-four episodes is already a generous ratio of freedom to evidence.
• Longer history helps more than finer measurement. Moving from daily to hourly bars multiplies the bar count and adds no episodes at all. Extending the history by ten years — if you can obtain data whose membership you trust — adds real observations.

The honest way to present a regime result is therefore to lead with the episode count, to show the outcome of each episode individually rather than only the aggregate, and to state that the sample is small. Part 33 covers the machinery for turning “the sample is small” into a number.

Recap

Breadth divergence is a claim about what happens next, and the ordinary way of presenting it uses information that was not available at the time: the peak is chosen because it was a peak, the reference window is chosen because it makes the picture clear, and no horizon is specified within which the claim could fail. A regime avoids all three problems by being a classification of the present rather than a prediction, and hysteresis makes it stable enough to act on. What a regime cannot avoid is that a decade of history contains a handful of episodes, so the evidence for any statement about regimes is thinner than the bar count makes it look.

Check your understanding

Question 1. What is the main methodological objection to identifying breadth divergences by looking at a chart?

• Charts do not show breadth accurately
• The peak, the reference window and the outcome horizon are all chosen after the outcome is known
• Breadth data is usually stale
• Divergence requires intraday data

Show the answer and why

Answer: The peak, the reference window and the outcome horizon are all chosen after the outcome is known

Each of those three choices uses information from after the moment being analysed. A testable version fixes all three in advance and applies them to every bar.

Question 2. Why does a regime definition use two thresholds instead of one?

BroadRegime = Flip( Pct >= 60, Pct <= 40 );

• To make the chart easier to read
• Because Flip() requires two arguments
• So that a reading oscillating around a single level does not produce a new regime every few bars
• To allow the regime to be optimised

Show the answer and why

Answer: So that a reading oscillating around a single level does not produce a new regime every few bars

The gap between the levels is hysteresis. It converts a noisy comparison into a state that persists long enough to be acted on and counted.

Question 3. A fifteen-year daily study finds that a system performs better in broad regimes. The regime definition produced 11 broad episodes. How many observations support the claim?

• About 3,800
• About 1,900
• About 11
• It depends on the number of trades

Show the answer and why

Answer: About 11

Bars within an episode are not independent. The unit of evidence is the episode, which is why the episode count belongs at the front of any regime result.

Question 4. Which of these compound rather than cancel when a backtest is gated by a breadth regime built from your own database? Select all that apply.

• Survivorship bias in the traded universe
• Survivorship bias in the breadth composite
• Thresholds tuned on the same history the system is tested on
• Using SafeDivide instead of plain division

Show the answer and why

Answer: Survivorship bias in the traded universe, Survivorship bias in the breadth composite, Thresholds tuned on the same history the system is tested on

The composite is built from the same survivor universe the system trades, so both are biased in the same direction, and tuning the thresholds on that history adds a third source of optimism.

Check my answersClear