Defining Your Universe and Market Regime

Two decisions constrain every number you will ever produce, and both are usually made by accident. The universe decides which instruments were ever eligible to appear in a result. The regime decides when the rules were allowed to fire at all. Change either one and every backtest statistic, every candidate list and every conclusion changes with it — which is precisely why both have to be written down in a form somebody else could argue with.

By the end of this lesson you should be able to state your universe as a specification rather than as a watch list name, name the biases that a universe definition inherits, defend a liquidity floor in terms of participation rather than habit, define a market regime as a classification instead of a forecast, and explain what gating on it costs you as well as what it buys.

Why these two first

Where the constraints bite

1. 1UniverseWhich instruments could ever appear
2. 2RegimeWhen the rules may fire at all
3. 3ScreenWhat actually qualifies tonight
4. 4Everything downstreamCandidates, trades, statistics

Both constraints are applied before a single rule is evaluated, which is why neither can be inspected afterwards from the results alone.

The order is not stylistic. A liquidity floor applied after the backtest is a filter on results; applied before, it is a definition of the study. The first is data snooping and the second is method, and the code can look almost identical.

The universe is a specification, not a watch list

In AmiBroker the mechanics are easy: the Analysis window’s Apply to setting selects all symbols, the current symbol, or a filter built from markets, groups, sectors, industries, watch lists, indices and favourites. From AFL, InWatchListName("Universe") tests membership by name and CategoryGetSymbols() retrieves a comma-separated list for a category. The reserved variable Exclude removes a symbol from a scan, exploration or backtest entirely.

The mechanics are not the decision. The decision is what should be in the list, and it has at least five components that deserve to be written down separately:

1. Instrument type. Ordinary shares only, or also funds, trusts, warrants, depositary receipts? Each has different trading behaviour and some have different bar semantics.
2. Market and currency. One exchange, or several? A turnover floor expressed as a single number across two currencies is not a floor, it is a rounding error with ambition.
3. Liquidity floor. Below, in detail.
4. Price floor. Not snobbery: at a low enough price the minimum tick is a large percentage of the price, and every cost assumption in your backtest is understated.
5. History requirement. A rule using a 200-bar average needs symbols with more than 200 bars, and a symbol with 210 bars contributes almost nothing but does contribute noise.

!Important

Write those five as sentences with numbers in them and store the file beside your results. A universe you cannot state in five sentences is a universe you cannot defend, and it is usually one that quietly changed between the backtest and the live decisions.

The biases you inherit for free

Every universe definition brings biases with it. You cannot avoid them; you can only know which ones you have.

Survivorship. If your database contains the companies that exist today, then every study you run is a study of survivors. Delisted, acquired and bankrupt instruments are simply not there to lose money in your backtest. This is the single largest source of flattering results in retail backtesting, and no formula can fix it — it is a property of the database. Part 30 takes it apart properly.

Membership look-ahead. Using today’s index constituents to define a universe for a ten-year test tells the 2016 version of your strategy which companies would go on to be worth including in 2026. The fix requires point-in-time membership data, which most free sources do not provide. If you do not have it, say so in the write-up rather than hoping.

Liquidity look-ahead. Filtering on today’s average turnover and then backtesting ten years does the same thing in a subtler way. A symbol that is liquid now may have been illiquid for the first six years of the test, and your rules will happily have traded it. Evaluate the floor bar by bar, as the formula below does, so that a bar in 2019 is judged by 2019’s turnover.

Selection by familiarity. A hand-built watch list of instruments you happen to follow is a sample selected by your own attention, which is not a random sample of anything. It is a perfectly reasonable universe as long as the write-up says that is what it is.

?Reality check

A claim you will meet constantly: “I tested it on the whole market.” Almost nobody has tested anything on the whole market. They have tested it on the symbols currently present in a database, which is a different set with a known direction of error. The claim is not usually dishonest; it is usually just unexamined. Ask which instruments were in the database and which were not, and the conversation gets much more interesting.

Liquidity floors

The purpose of a liquidity floor is narrow and specific: to keep your study inside the region where its fill assumptions are defensible. It is not a quality filter and it is not a proxy for anything about the company.

Use turnover, not share volume. Turnover — price multiplied by volume — is comparable across price levels and is denominated in the same unit as your position size. A share at 3.00 trading two million shares and a share at 300.00 trading twenty thousand have wildly different volumes and the same turnover, and it is turnover that answers the question you care about.

Express the constraint as participation. The floor itself is arbitrary; participation is not. Participation is your intended position value divided by average daily turnover, and it answers “what fraction of a normal day’s trading would I have to be?” A position that is one per cent of a day’s turnover is a very different proposition from one that is thirty per cent, and the second one invalidates the backtest’s fill assumption regardless of how impressive the turnover figure looked.

Average it over enough bars, and mind the tail. A single day’s turnover can be dominated by one block. A fifty-bar average is a common choice and is a choice, not a standard. And an average conceals concentration: an instrument that trades its whole daily volume in the first ten minutes has a turnover figure that flatters it.

⌀Data quality warning

A liquidity floor cannot rescue a symbol whose data is defective, and it can hide the problem. Symbols with holes filled by repeated closes often pass a turnover floor comfortably, because the vendor also carried the previous volume forward. Run the data-validation gate from Lesson 1 first; the floor is the second filter, not the first.

Know what the floor costs. Raising it shrinks the universe, which reduces the number of opportunities, increases the concentration of your results in a small number of instruments, and biases the study toward larger companies. That is a trade you may well want to make. It is not a free improvement, and a write-up that reports only what a floor removed and not what it cost is only half a write-up.

Market regime

A market regime in this course is a classification of overall market conditions, computed from data available at the time, used as a gate. That definition contains three constraints and each one is doing work.

A classification — it puts each bar into one of a small number of named states. It does not produce a probability, a target or a forecast.

From data available at the time — the classification for a given bar uses only bars up to that one. A regime label computed from the full history and applied retrospectively is look-ahead bias wearing a respectable hat.

Used as a gate — it decides whether the rules are permitted to fire, not what they should do.

Candidate definitions

None of these is correct. Each is a definition, and the honest position is that you chose one.

Definition	Computed from	What it is sensitive to
Benchmark above its own long moving average	One index symbol	The period chosen; it whipsaws around the crossing
Long moving average of the benchmark rising	One index symbol	The slope window; slower, later, fewer flips
Breadth: proportion of the universe above its own long average	Every symbol, via a composite	Universe definition; needs Part 16’s composites
Benchmark volatility relative to its own history	One index symbol	Whether you use ATR, standard deviation, or a range measure
New highs minus new lows	Every symbol	Data quality in the tails of the universe

The formula in this lesson uses the first two together, because combining two conditions gives three states rather than two, and the middle state — where the two disagree — is real. A two-state gate has to assign those bars somewhere, and pretending they belong cleanly on one side is how a gate acquires confidence it has not earned.

A gate, not a prediction

The distinction is worth being pedantic about, because the two get conflated constantly.

A prediction says something about what the market will do. A gate says something about when your rules were tested and under what conditions their measured behaviour was measured. If your breakout system was profitable in the backtest only during periods when the benchmark was above its long average, then a gate on that condition is not a forecast — it is a refusal to trade a system outside the conditions in which it was evaluated.

∴Research note

This gives you the honest test for a regime gate, and it is not “does it improve the results”. It is: does the strategy’s behaviour differ between the regime states, by more than the difference you would get from an arbitrary partition of the same length? If the strategy performs the same in both states, the gate has told you nothing and you have simply thrown away half your sample. Part 30’s material on regime dependence and insufficient evidence is the right companion to this question.

What a gate costs

Four costs, all of them real:

• Fewer trades. A gate that removes 40 per cent of the calendar removes roughly 40 per cent of your sample, and a smaller sample supports weaker conclusions. This is the cost people consistently forget when they admire a gated equity curve.
• Lag at the turn. Every gate built on a long average is late by construction, at both ends.
• Whipsaw at the boundary. Near the crossing, the state flips. Whether that matters depends on how much your rules do at the moment of the flip.
• A new place to overfit. The gate has parameters. Search over them and you can improve any backtest, which tells you about the search, not the market. If you optimise the gate, it goes into the walk-forward with everything else, not before it.

Documenting both so they can be criticised

The output of this lesson is a specification document, one page, versioned and dated, stored beside the results it constrains. The minimum fields:

Field	Example of a usable entry
Universe name and version	AU-large-liquid v1.2
Instrument types included	Ordinary shares only; no funds, trusts or warrants
Markets and currency	One exchange, one currency, no conversion applied
Liquidity floor	50-bar average turnover at or above 2,000,000, evaluated bar by bar
Participation cap	Intended position at or below 1% of 50-bar average turnover
Price floor	Close at or above 5.00
History floor	250 bars
Known biases	Survivorship: database holds current listings only. Membership: no point-in-time index data
Regime definition	Benchmark close above 200-bar MA, and that MA above its value 20 bars ago
Benchmark symbol	Stated explicitly, with the exact ticker as it appears in the database
Regime states	0, 1 or 2 conditions true; a fourth state for a missing benchmark
What the gate does	Rules fire only in state 2; states 0 and 1 produce no candidates
Date and author	Version date, and who changed it last

The two entries people leave blank are known biases and what the gate does. They are the two that a critical reader goes to first.

The universe and regime formula

Goal

One exploration that answers both questions on the same evening, for every symbol, with the reasons visible: is this symbol in the universe, and if not, which floor did it fail? What regime state is the benchmark in, and what would my intended position be as a share of this symbol’s daily turnover?

Complete formula

Complete runnable AFL

universe-and-regime.afl

// universe-and-regime.afl
// Part 35 - Defining Your Universe and Market Regime
//
// Produces the two documents that constrain everything downstream: which
// instruments are in the universe on this bar, and which regime the benchmark
// is in on this bar. Both are definitions applied consistently. Neither is a
// forecast, and neither says anything about any individual instrument's
// prospects.
//
// Run settings this formula assumes:
//   Apply to: the broadest symbol list you are willing to consider
//   Range:    All quotations, with Filter restricting the output to one row
//             per symbol at the last bar
//
// Assumptions:
//   - Daily bars in a single currency. Turnover is Close * Volume expressed in
//     whatever currency the symbols are quoted in, so a mixed-currency
//     database makes the liquidity floor meaningless until you convert.
//   - The benchmark symbol exists in the same database and covers the same
//     dates. Foreign() aligns it to the current symbol's bars; a benchmark
//     with a different trading calendar will be padded, which is exactly the
//     alignment problem Part 15 describes.
//   - Membership is evaluated bar by bar rather than once at the right-hand
//     edge, so the same expressions can be pasted into a backtest without
//     importing today's liquidity into a decision made in 2015.
//   - The universe this produces still contains only the symbols your database
//     contains. Instruments that were delisted and removed are invisible here,
//     and no formula can recover them. That is a database problem, and it is
//     the survivorship bias Part 30 examines.

_SECTION_BEGIN("Universe and regime");

// ---- The universe specification -----------------------------------------
// Every number below is a decision that a reader of your research is entitled
// to disagree with, which is why they are all named, all at the top, and all
// exposed in the Parameters dialog rather than buried in an expression.

MinTurnover    = Param( "Liquidity floor: average daily turnover", 2000000, 0, 100000000, 100000 );
TurnoverWindow = Param( "Turnover averaging window (bars)", 50, 5, 250, 5 );
MinPrice       = Param( "Price floor (per share)", 5, 0, 500, 0.5 );
MinHistoryBars = Param( "History floor (bars)", 250, 20, 5000, 10 );
IntendedValue  = Param( "Intended position value", 10000, 100, 10000000, 100 );
MaxParticipation = Param( "Participation cap (% of daily turnover)", 1, 0.05, 25, 0.05 );

// ---- The regime specification -------------------------------------------

BenchmarkTicker = ParamStr( "Benchmark symbol", "^GSPC" );
RegimePeriod    = Param( "Benchmark trend period (bars)", 200, 20, 500, 10 );
SlopeBars       = Param( "Slope measured over (bars)", 20, 1, 200, 1 );

// ---- Universe membership, bar by bar ------------------------------------

Turnover    = Close * Volume;
AvgTurnover = MA( Turnover, TurnoverWindow );

BarNumber = Cum( 1 );

LiquidEnough  = Nz( AvgTurnover ) >= MinTurnover;
PricedEnough  = Close >= MinPrice;
HistoryEnough = BarNumber >= MinHistoryBars;

InUniverse = LiquidEnough AND PricedEnough AND HistoryEnough;

// How much of an average day's trading your intended position would be. This
// is the number that decides whether a fill assumption is defensible, and it
// is far more informative than the raw turnover figure it is derived from.
SafeTurnover  = Max( Nz( AvgTurnover ), 1 );
Participation = 100 * IntendedValue / SafeTurnover;
WithinCap     = Participation <= MaxParticipation;

// 0 in universe, 1 too illiquid, 2 too cheap, 3 too little history. Reported
// worst-first so a symbol failing several tests names the hardest failure.
RejectCode = IIf( NOT HistoryEnough, 3,
             IIf( NOT PricedEnough,  2,
             IIf( NOT LiquidEnough,  1, 0 ) ) );
RejectList = "in universe\nbelow turnover floor\nbelow price floor\nbelow history floor";

// ---- Regime, computed once from the benchmark ----------------------------
// Foreign() reads another symbol's array and aligns it to the current symbol's
// bars, so every symbol in the run sees the same regime on the same date.

BenchClose = Foreign( BenchmarkTicker, "C" );
BenchTrend = MA( BenchClose, RegimePeriod );

BenchAbove  = BenchClose > BenchTrend;
BenchRising = BenchTrend > Ref( BenchTrend, -SlopeBars );

// A benchmark that is missing from the database returns Null, and a Null
// regime must not silently become "risk-on". Treat it as unknown instead.
BenchmarkMissing = IsNull( BenchClose ) OR IsNull( BenchTrend );

// Two independent conditions give three states, which is deliberate: the
// middle state is where the two disagree, and pretending it does not exist is
// how a gate acquires false confidence.
RegimeCode = IIf( BenchmarkMissing, 3, BenchAbove + BenchRising );
RegimeList = "both conditions false\none condition true\nboth conditions true\nbenchmark unavailable";

// ---- Output --------------------------------------------------------------

Filter = Status( "lastbarinrange" );

AddTextColumn( FullName(), "Name", 40 );
AddMultiTextColumn( RejectCode, RejectList, "Universe", 24 );
AddMultiTextColumn( RegimeCode, RegimeList, "Regime", 24 );
AddColumn( Close, "Close", 1.2 );
AddColumn( AvgTurnover, "Avg turnover", 1.0 );
AddColumn( Participation, "Participation %", 1.2,
           colorDefault, IIf( WithinCap, colorDefault, colorLightOrange ) );
AddColumn( WithinCap, "Within cap", 1.0 );
AddColumn( BarNumber, "Bars held", 1.0 );
AddColumn( InUniverse, "In universe", 1.0 );

// Column numbers count the two default columns, so "Avg turnover" is column 7.
// Sorting the whole list by liquidity makes the shape of the universe visible:
// where the floor sits relative to the bulk of the candidates.
SetSortColumns( -7 );

_SECTION_END();

Download universe-and-regime.afl118 lines

How it works

Every threshold is a named Param() at the top. That is not decoration: the parameters are the specification, and having them in the Parameters dialog means the specification can be challenged by changing a number and re-running rather than by editing code.

Universe membership is computed bar by bar from three conditions — a turnover floor, a price floor and a history floor — and then collapsed into a rejection code that reports the hardest failure first. Because it is evaluated on every bar rather than once at the right-hand edge, the same expressions can be pasted into a backtest without importing today’s liquidity into a decision made years ago.

Participation is computed from the intended position value you supply, against a floored denominator so that a symbol with no turnover cannot produce an infinity. The Within cap column is a plain 1 or 0 as well as a background colour, because the meaning must never be carried by colour alone.

The regime section reads the benchmark once with Foreign(), which aligns another symbol’s array to the current symbol’s bars, so every symbol in the run sees the same regime on the same date. Two independent conditions are added together to give a state of 0, 1 or 2, and a fourth state exists for the case that matters most: a benchmark that is missing from the database returns Null, and a Null regime must never silently become “conditions met”.

Key functions

• Foreign( ticker, field, fixup = 1 ) — reads another symbol’s array, aligned to the current symbol’s bars. This is what makes a single regime available to every symbol in a run.
• IsNull( x ) — true where a value is Null. Used here as an explicit guard rather than letting Null propagate, which is the AFL habit that costs people the most debugging time.
• Nz( x, valueifnull = 0 ) — converts Null, NaN and infinity to zero or a value you choose. Used on MA( Turnover, ... ) so that warm-up bars fail the floor rather than producing an undefined comparison.
• ParamStr( name, default ) — a string parameter. The benchmark ticker belongs here, not hard-coded, because the correct ticker differs between databases.
• AddMultiTextColumn() — as in Lesson 1, the way to print a per-bar text label.

Expected result

Apply to your broadest symbol list, set Range to All quotations, press Explore. One row per symbol, sorted by average turnover descending, with the whole universe visible — including the symbols that failed.

That last point is deliberate. The exploration does not filter out rejected symbols, because the shape of the rejection is the information: you want to see where your floor sits relative to the bulk of the candidates. If the floor falls in the middle of a dense cluster, small changes to it will move many symbols in and out, and every downstream result is more fragile than it looks.

=Expected result

Look at the Regime column. It must read identically on every row, because it is computed from one benchmark and does not depend on the symbol. If it varies by row, Foreign() is not returning what you think — usually a wrong ticker, or a benchmark whose history does not cover the same dates.

Test it

1. Set the turnover floor to zero. Every symbol should read in universe except those failing the price or history floor. This confirms the codes are ordered as intended.
2. Set the benchmark to a ticker that does not exist. Every row should read benchmark unavailable, not both conditions false. A gate that fails open is worse than no gate, because it fails silently.
3. Raise the intended position value by a factor of ten. Participation should rise by the same factor and the Within cap column should flip for the least liquid symbols first.
4. Change the trend period from 200 to 50 and note how many dates change state, not how the results look. Sensitivity of the classification is what you are measuring.

Common errors

• Every row reads below history floor. The Range is narrow and QuickAFL handed the formula a slice of the array. Either widen the Range or require all bars, as Lesson 1’s formula does.
• The regime column varies between symbols. The benchmark’s history does not align with the symbols’, or the ticker is wrong. Foreign() pads and aligns; it cannot invent history.
• Turnover is enormous and meaningless. The database mixes currencies, or contains symbols quoted in cents alongside symbols quoted in units. The floor is comparing incomparable things.
• The floor was tuned until the results improved. This is not an error the formula can detect. It is the reason the floor belongs in a dated specification written before the test.

Extension

Replace the single-benchmark regime with a breadth measure: the proportion of your universe trading above its own long moving average, built as a composite with AddToComposite() as Part 16 describes. Then run both definitions side by side for a year of history and count the days on which they disagree. The count is the useful output. Two regime definitions that agree on 95 per cent of days are one definition; two that agree on 60 per cent are a genuine choice you have to justify.

Recap

You now have both constraints as documents rather than as habits. The universe is five sentences with numbers in them, plus an explicit list of the biases it inherits — survivorship, membership look-ahead, liquidity look-ahead and selection by familiarity — none of which a formula can remove. The liquidity floor is justified by participation rather than by custom, and evaluated bar by bar so that history is judged by its own conditions rather than today’s. The regime is a classification used as a gate, with a stated cost in sample size, lag, whipsaw and one more surface to overfit, and with an honest test that asks whether behaviour differs between states rather than whether the equity curve looks nicer.

The next lesson picks up where this one stops: a short list of candidates exists, and a human has to decide what to do about them.

Check your understanding

Question 1. You filter your universe on average turnover computed over the last 50 bars as of today, then backtest that universe over ten years. What is wrong?

• Nothing; the filter is applied consistently
• The 50-bar window is too short for a ten-year test
• Symbols are selected using turnover information from the end of the test, so trades in 2018 were placed in instruments chosen for being liquid in 2026
• Turnover should have been average volume instead

Show the answer and why

Answer: Symbols are selected using turnover information from the end of the test, so trades in 2018 were placed in instruments chosen for being liquid in 2026

This is liquidity look-ahead, and it is subtle because the filter itself is reasonable. The fix is to evaluate the floor bar by bar, so that a bar in 2018 is judged by the turnover available in 2018. The same expression written the correct way costs nothing extra to compute.

Question 2. Given this fragment, what happens on a symbol whose benchmark is missing from the database?

BenchClose = Foreign( BenchmarkTicker, "C" );
RegimeOn   = BenchClose > MA( BenchClose, 200 );

• AmiBroker raises an error and stops the exploration
• RegimeOn is Null-propagated rather than true, and the symbol produces no rows - but nothing tells you why
• RegimeOn becomes true, and every symbol passes the gate
• The symbol is automatically excluded from the run

Show the answer and why

Answer: RegimeOn is Null-propagated rather than true, and the symbol produces no rows - but nothing tells you why

A missing Foreign symbol yields Null, and comparisons involving Null do not produce a helpful message. The failure is silent, which is the dangerous kind: an evening with no candidates looks like an evening with no setups. Testing IsNull() explicitly and reporting a distinct "benchmark unavailable" state turns a silent failure into a visible one.

Question 3. Which of these are legitimate reasons to apply a market regime gate? Select all that apply.

• The strategy was evaluated only in periods matching that condition, and you decline to trade it outside them
• The gate raises the backtest’s Annual Return % from 12 to 19
• The gate reduces exposure during the conditions in which the rules have no measured evidence
• The gate identifies when the market is about to fall

Show the answer and why

Answer: The strategy was evaluated only in periods matching that condition, and you decline to trade it outside them, The gate reduces exposure during the conditions in which the rules have no measured evidence

A gate is a statement about the conditions under which your evidence was gathered, not about the future. Improving a backtest statistic is not on its own a reason, because a parameterised gate can improve almost any backtest if you search over it - that is a fact about searching. And no classification of past bars identifies what the market is about to do.

Question 4. A regime gate removes 40 per cent of the trading days from your test. What has definitely happened to your evidence?

• It has improved, because the removed days were the bad ones
• It is unchanged, because the same rules are being tested
• The sample has shrunk by roughly 40 per cent, so every statistic computed on it is less precise, whatever the headline numbers do
• It has become out-of-sample

Show the answer and why

Answer: The sample has shrunk by roughly 40 per cent, so every statistic computed on it is less precise, whatever the headline numbers do

Gating throws away data, and a smaller sample supports weaker conclusions regardless of which direction the summary statistics move. That cost has to be weighed against whatever the gate buys, and it is the cost that admiring write-ups of gated equity curves almost never mention.

Question 5. Why does the specification sheet include a "known biases" field even though nothing can be done about most of them?

• To satisfy a compliance requirement
• Because a stated bias has a known direction, which lets a reader discount the result appropriately, whereas an unstated one is indistinguishable from an edge
• Because AmiBroker records it in the report
• Because it makes the universe smaller

Show the answer and why

Answer: Because a stated bias has a known direction, which lets a reader discount the result appropriately, whereas an unstated one is indistinguishable from an edge

Survivorship bias flatters results in a known direction and by an unknown amount. A reader told about it can reason about the result; a reader not told about it reads the flattery as evidence. The purpose of the field is not to fix the bias - it is to stop the bias from being silently converted into confidence.

Check my answersClear