Survivorship and Selection Bias

Your database is not a record of the market. It is a record of the symbols someone decided to keep. Everything that stopped trading — delisted, acquired, merged, renamed, suspended, or quietly dropped by the data vendor because nobody was downloading it any more — is missing, and it is missing without leaving a hole. The remaining symbols look complete. They are complete. They are also, collectively, the survivors.

By the end of this lesson you should be able to say precisely how survivorship enters a test, run an audit on your own database that shows you whether it is present, explain why a list of index members is worthless without a date attached, recognise the version of the same error that you commit yourself when you pick symbols to test on, reason about the size of the effect instead of guessing at it, and choose an honest mitigation when point-in-time data is out of reach.

The mechanism, stated once

A backtest applies a rule to a set of symbols over a date range. The set is drawn from your database today. Membership of that set was determined by events that happened after the start of the test period — most importantly, by not having ceased to exist.

That is the whole mechanism. There is nothing wrong with the formula. The AFL is irrelevant. The bias enters through the Apply to box in the Analysis window.

What a 2015–2025 test on today's symbol list actually sees

Still listed

in the test throughout

Delisted 2019

should be hereabsent from the database

Acquired 2021

should be hereabsent

Listed 2022

not yet publicin the test

2015201820212025

Three of those four rows are wrong in a way that flatters the result, and the fourth — the late listing — is the only one most people notice, because it is the only one where the missing data is visible as a short history.

The severity depends on how your rule selects. A rule applied uniformly to every symbol is affected in proportion to how much of the universe went missing. A rule that ranks — a rotational system, a relative-strength system, anything using PositionScore to pick the top N — is affected far more, because the names that were removed are disproportionately the ones that would have ranked badly and been avoided, or ranked well shortly before collapsing and been bought.

Where the missing symbols went

It is worth separating the causes, because they need different fixes.

• Bankruptcy and delisting. The company failed or fell below listing standards. Its final months are usually catastrophic, and those returns are exactly what is missing from your test.
• Acquisition and merger. The company was bought. The final return may have been strongly positive, so removing these is not always flattering — but it is always wrong, and it removes the position-closing event your system would have faced.
• Ticker changes and re-listings. The company still exists under a different symbol. Its history may be split across two symbols, or attached entirely to the new one, which silently changes when its history “starts”.
• Vendor pruning. Free and low-cost end-of-day sources frequently carry only currently-active symbols, because the commercial demand for a dead company’s price history is close to zero.

⌀Data quality warning

“Pad and align to reference symbol” (Settings → General) makes this harder to see, not easier. With it on, a delisted symbol’s series is padded forward to the reference symbol’s calendar using the previous close with zero volume, so its last quote date looks current. That is often what you want for composites and ranking, and it is exactly wrong when you are auditing listing dates. Turn it off before you run the audit below, and turn it back on afterwards if your system needs it.

Index membership needs a date attached

“The S&P 500”, “the FTSE 100”, “the NASDAQ-100” are not lists. They are lists as of a date. A membership file downloaded today describes the index as it is today, and using it to define a universe for a ten-year backtest asserts that today’s members were the members throughout — which is false by construction, since indices have documented turnover rules.

The bias this creates is stronger than plain delisting survivorship, because index membership is itself an outcome. Companies enter major indices after performing well and are removed after performing badly. Testing “the current members, over the last decade” is therefore testing a portfolio selected on ten years of subsequent performance.

∴Research note

If your universe is an index, the only defensible source is a historical membership file: a table of symbol, date added, date removed. If you have one, you can build the point-in-time universe in AmiBroker by encoding membership as a date range per symbol and using the Exclude reserved variable to drop a symbol on bars where it was not a member. If you do not have one, say so in your write-up, and treat every result as an upper bound.

Fragment — not a complete formula

// Point-in-time membership, given per-symbol dates you supply yourself.
// Exclude removes the symbol from the scan, exploration or backtest on those
// bars, and also leaves it out of buy-and-hold calculations.
JoinedDatenum = 1080315;    // 15 March 2008, in DateNum() form (YYY = year - 1900)
LeftDatenum   = 1190722;    // 22 July 2019

WasMember = DateNum() >= JoinedDatenum AND DateNum() <= LeftDatenum;
Exclude   = NOT WasMember;

That fragment is only as good as the dates you feed it, and there is no way around that. The honest position is either “I have dated membership data and here it is”, or “I do not, and here is what I did instead”.

The version you commit yourself

Survivorship happens to you. Selection bias is something you do.

The commonest form in retail research is choosing test symbols from memory. Someone develops a trend-following rule and tests it on a handful of large technology names that everyone can name — which is to say, on companies selected for having produced decades of large trends. The rule appears to work. It would be remarkable if it did not.

The same error takes several other shapes:

• Choosing the date range because of what happened in it. A test starting in April 2020 on equities begins immediately after a violent low. That is a choice, and it is a choice made with knowledge of what followed.
• Choosing the universe after a first look. Running a test on 500 symbols, noticing it works on the small caps, and re-running on small caps only, is a legitimate hypothesis-generating move and an illegitimate result. The second run is not evidence; it is a description of the first run.
• Choosing the rule because it is famous. Rules become famous by having worked in some published sample. Testing a rule that survived that filter is survivorship applied to strategies rather than to symbols.
• Dropping the symbol that ruined the average. “That one had a data error” is sometimes true and always suspicious, particularly when the check was triggered by the symbol being the worst performer.

The defence for all four is the same and it is procedural, not technical: decide the universe, the date range and the rule before you look at the result, write those three decisions down, and count every time you change one of them. Part 27 sets up that habit; the overfitting lesson later in this part shows what happens to your statistics when the count gets large.

How big is the effect, and how would you know?

Resist the urge to reach for a number from a blog. The size of survivorship bias is not a universal constant: it depends on the universe, the period, the delisting rate and how differently the removed names behaved. What you can do is reason about the bound.

Suppose a fraction d of the symbols in your universe disappear each year, and that in their final year those symbols underperformed the survivors by an amount u. Then a test that silently omits them overstates the annual return of an equally weighted, always-invested portfolio by roughly d × u.

The following table is illustrative arithmetic, not a measurement of any market. It exists so you can see the shape of the sensitivity.

Annual attrition d	Final-year shortfall u	Approximate annual overstatement
2%	30%	about 0.6 percentage points
2%	60%	about 1.2 percentage points
5%	30%	about 1.5 percentage points
5%	60%	about 3.0 percentage points
10%	60%	about 6.0 percentage points

Two things follow. First, on a broad, stable, large-capitalisation universe the effect is real but modest, and it will not turn a losing rule into a winning one. Second, on a universe with high attrition — small caps, micro caps, growth names, recently listed companies, crypto assets, anything where failure is common — the effect is large enough to be the entire result, and a ranking system in that universe can be affected by considerably more than the table suggests, because ranking concentrates exposure into exactly the names most likely to have been removed.

If you want a real number for your universe, the only way is to obtain a list of what was removed and when, re-run with those symbols restored, and compare. Anything short of that is an estimate, and should be presented as one.

Auditing your own database

Goal

Find out, in one pass, which symbols in your universe do not span the period you intend to test, how much of their coverage is padding rather than trading, and whether the shape of the answer is consistent with a survivor-only database.

Complete formula

Complete runnable AFL

universe-listing-audit.afl

// universe-listing-audit.afl
// Part 30 - Survivorship and Selection Bias
//
// PURPOSE
//   Before you believe a portfolio backtest, you have to know what was in the
//   universe and when. This exploration reports, for every symbol the Analysis
//   window is applied to, when its history starts, when it stops, how much of
//   the requested date range it actually covers, and how much of that coverage
//   is padding rather than trading.
//
//   A database in which every symbol runs from the range start to the range end
//   is not a clean database. It is usually a database from which everything
//   that died has been removed.
//
// HOW TO RUN
//   Analysis -> Apply to: the universe you intend to backtest.
//   Periodicity: Daily.
//   Range: set an EXPLICIT From and To date. This formula compares each
//   symbol's coverage with those dates, so "All quotations" makes the coverage
//   columns meaningless.
//   Press EXPLORE, then sort by "Last quote" ascending and by "Starts late".
//
// ============================ ASSUMPTIONS =============================
//   - A symbol whose last quote is well before the range end either delisted,
//     was renamed, merged, or is simply missing from your data feed. This
//     formula cannot tell you which; it only tells you where to look.
//   - A symbol whose first quote is after the range start either listed during
//     the period, or your history is truncated. Again: it points, it does not
//     conclude.
//   - Zero-volume bars whose Close repeats the previous Close are the
//     signature of padded, non-trading data. With Settings -> General ->
//     "Pad and align to reference symbol" turned on, a delisted symbol is
//     padded forward to the reference symbol's calendar, so its last quote
//     date will look current. Run this audit with padding OFF if you want the
//     genuine listing dates.
//   - Nothing here proves your universe is point-in-time correct. Only a
//     dated, historical membership list can do that.
// ======================================================================

SetBarsRequired( -2, -2 );   // -2 is sbrAll: coverage counts need every bar

MinBarsWanted = Param( "Bars of history you expect", 500, 20, 10000, 10 );

// ---------------------------------------------------------------------
// 1. Coverage of the requested range
// ---------------------------------------------------------------------
RangeFrom = Status( "rangefromdate" );
RangeTo   = Status( "rangetodate" );
InRange   = Status( "barinrange" );

BarsTotal   = Cum( 1 );
BarsInRange = Cum( InRange );

FirstQuote  = ValueWhen( BarsTotal == 1, DateTime() );
FirstDatenum = ValueWhen( BarsTotal == 1, DateNum() );

StartsLate = FirstDatenum > RangeFrom;
EndsEarly  = DateNum() < RangeTo;

// ---------------------------------------------------------------------
// 2. How much of the coverage is real trading
// ---------------------------------------------------------------------
NoVolume     = InRange AND Nz( Volume ) <= 0;
RepeatedBar  = InRange
           AND Close == Ref( Close, -1 )
           AND High == Low;

BarsNoVolume    = Cum( NoVolume );
BarsRepeated    = Cum( RepeatedBar );
BarsPaddedLike  = Cum( NoVolume AND Close == Ref( Close, -1 ) );

// Trailing padding: how many of the most recent bars in range carry no volume
// at all. A long tail here on an otherwise normal symbol is what a delisted
// name looks like once the database has been padded to a reference calendar.
TradedBar      = InRange AND Nz( Volume ) > 0;
BarsSinceTrade = BarsSince( TradedBar );

// ---------------------------------------------------------------------
// 3. Liquidity, because a symbol can be present and still untradable
// ---------------------------------------------------------------------
Turnover       = MA( Close * Volume, 50 );
MedianTurnover = Median( Close * Volume, 250 );

// ---------------------------------------------------------------------
// 4. One row per symbol, on that symbol's last bar inside the range.
// ---------------------------------------------------------------------
Filter = Status( "lastbarinrange" );

AddColumn( FirstQuote,  "First quote", formatDateTime );
AddColumn( DateTime(),  "Last quote",  formatDateTime );
AddColumn( BarsTotal,   "Bars in database", 1.0 );
AddColumn( BarsInRange, "Bars in range", 1.0 );

AddColumn( StartsLate, "Starts late", 1.0 );
AddColumn( EndsEarly,  "Ends early",  1.0 );
AddColumn( BarsInRange >= MinBarsWanted, "Enough history", 1.0 );

AddColumn( BarsNoVolume, "Zero-volume bars", 1.0 );
AddColumn( 100 * SafeDivide( BarsNoVolume, BarsInRange, Null ), "% zero volume", 1.1 );
AddColumn( BarsPaddedLike, "Padded-looking bars", 1.0 );
AddColumn( BarsRepeated, "Flat bars (H=L, C unchanged)", 1.0 );
AddColumn( BarsSinceTrade, "Bars since last real trade", 1.0 );

AddColumn( Close, "Last close", 1.2 );
AddColumn( Turnover, "50-bar avg turnover", 1.0 );
AddColumn( MedianTurnover, "250-bar median turnover", 1.0 );

// COUNT and AVERAGE rows across the universe.
AddSummaryRows( 2 | 16, 1.2 );

Download universe-listing-audit.afl109 lines

How it works

Four blocks. The first compares each symbol’s first quote against the analysis range’s start date and its last bar in range against the range’s end date, using Status( "rangefromdate" ) and Status( "rangetodate" ), both of which return the Analysis window’s range as DateNums. The second counts bars that carry no volume, bars whose close simply repeats the previous close with no range at all, and how many bars have passed since the symbol last actually traded — the signature of padded, non-trading data. The third computes two liquidity measures, because a symbol can be present in the database and still be untradable. The fourth writes one row per symbol on that symbol’s last bar inside the range.

Key functions

Status( "rangefromdate" ) and Status( "rangetodate" ) return the current Analysis range as DateNums, in the same YYYMMDD encoding as DateNum() where YYY is the year minus 1900. ValueWhen( BarsTotal == 1, DateTime() ) captures the symbol’s very first bar, since Cum( 1 ) reaches 1 on bar zero. Status( "lastbarinrange" ) fires on the last bar of the range for each symbol, so Filter produces exactly one row per symbol. BarsSince() counts bars since a condition was last true. AddSummaryRows( 2 | 16, 1.2 ) appends a count row and an average row across the universe.

Expected result

One row per symbol. Sort by Last quote ascending: in a database with genuine history, some symbols end well before the range end, and those are the delisted or renamed names. Sort by Starts late: those are late listings, truncated history, or ticker changes. If every symbol starts on the range start and ends on the range end, that is the finding — you are holding a survivor-only universe, and every portfolio backtest you run on it inherits the bias.

Test it

Pick one symbol you know had a corporate event in the period — an acquisition, a delisting, a ticker change — and check that its row shows what you expect. If it does not appear at all, that absence is the audit result. Then re-run the same exploration with “Pad and align to reference symbol” switched on and compare: the padded run should show later last-quote dates and many more zero-volume bars for the same symbols, which is the effect that hides delisting from you.

Common errors

Running with the range set to “All quotations” makes the Starts late and Ends early columns meaningless, because the range dates are not what you assumed — always set an explicit From and To. Applying the exploration to a watchlist that was itself built from currently-active symbols answers the wrong question: the audit can only tell you about the symbols it was given. Forgetting SetBarsRequired( -2, -2 ) lets QuickAFL evaluate a subset of bars, which corrupts the cumulative counts.

Extension

Add a column that counts the largest run of consecutive zero-volume bars in the middle of the history, rather than at the end. A long internal gap is usually a trading halt or a data outage, and both change what a backtest could have done during that window.

Mitigations when point-in-time data is out of reach

You will often not have a dated universe. The course is designed to be completable without a paid data subscription, so this is the normal case rather than the exception. There are honest things to do.

1. Say so, in the result. “Universe: current constituents of X as at [date], survivorship not corrected” belongs next to every number you quote. This is not a disclaimer; it is part of the measurement.
2. Prefer universes with low attrition. A test on large, long-established instruments — major index futures, the largest liquid shares, broad ETFs with long histories — carries far less survivorship than a test on small caps.
3. Use a liquidity and history filter that is applied at each bar, not once. Requiring a symbol to have 250 bars of history before the decision bar is causal. Requiring a symbol to have 2,500 bars in total is not, because it requires the symbol to still be alive at the end.
4. Test the direction of the bias. Re-run with the shortest-history symbols excluded and then with only them included. If the result depends heavily on short-history names, survivorship is doing significant work.
5. Test on an instrument with no survivorship at all. Index futures, major FX pairs and long-lived ETFs cannot delist out of your sample. If a rule works on a survivor-heavy universe and not on these, that is informative.
6. Treat the result as an upper bound and act accordingly. A rule whose margin over costs is thin on a survivor-only universe has no margin at all.

⚠Warning

Do not “fix” survivorship by adding a plausible penalty to the result. There is no principled way to choose the penalty, the number will be invented, and it will migrate into the write-up as if it were measured. An honest upper bound with the limitation stated is worth more than a corrected figure nobody can reproduce.

Recap

Survivorship enters through the universe, not the formula, which is why no amount of AFL discipline detects it. It is strongest where attrition is high and where the rule ranks rather than filters. Index membership is a dated fact, and using an undated list asserts something false about the past. The self-inflicted version — choosing symbols, ranges, rules or exclusions with knowledge of the outcome — is more common in practice than the database version and is prevented by writing decisions down before looking, not by any tooling.

You now have an audit that tells you which of these you are exposed to, and a set of mitigations that do not require inventing a correction factor.

The next lesson turns to the error you commit while doing everything on this page correctly: searching a single dataset until it produces a result you like.

Check your understanding

Question 1. A rotational system ranks 200 current index members by 12-month momentum and holds the top 10. Why is it more exposed to survivorship than a filter-based system on the same universe?

• Because rotational mode ignores the price arrays
• Because ranking concentrates capital into a small number of names, and the names that were removed from the database are exactly the ones a ranking would have selected or avoided
• Because PositionScore is ranked on absolute value
• Because rotational systems trade more often

Show the answer and why

Answer: Because ranking concentrates capital into a small number of names, and the names that were removed from the database are exactly the ones a ranking would have selected or avoided

A filter applied uniformly is affected roughly in proportion to how much of the universe is missing. A ranking system puts all of its capital into the extremes of the distribution, and the missing names are disproportionately at those extremes — either collapsing after a strong run, or already ranking badly. Removing them changes which ten symbols are held, not just the average.

Question 2. Which of these are point-in-time correct filters for a backtest? Select all that apply.

• Require at least 250 bars of history before the decision bar
• Require the symbol to have at least 2,500 bars in the database
• Require 50-bar average turnover above a threshold, measured up to the decision bar
• Require the symbol to still be listed at the end of the test period

Show the answer and why

Answer: Require at least 250 bars of history before the decision bar, Require 50-bar average turnover above a threshold, measured up to the decision bar

The first and third use only data that existed at the decision bar. The second requires the symbol to be alive long enough to accumulate a total bar count, which is a fact about the future. The fourth is survivorship stated as a rule.

Question 3. With "Pad and align to reference symbol" turned on, what happens to a symbol that delisted in 2019 in a test running to 2025?

• It is excluded from the backtest automatically
• Its series is padded forward using the previous close with zero volume, so its last quote date looks current
• AmiBroker raises a warning
• Its final price is carried forward with the reference symbol’s returns

Show the answer and why

Answer: Its series is padded forward using the previous close with zero volume, so its last quote date looks current

The documented behaviour is that quotes are padded and aligned to the reference symbol, with holes filled from previous bar data. That is useful for composites and cross-symbol ranking, where alignment matters, and actively misleading when you are trying to find out when a symbol stopped trading — which is why the listing audit should be run with padding off.

Question 4. You test a rule on 500 symbols, notice it performs well on the smallest 100, and re-run on those 100 only. What have you produced?

• Evidence that the rule works on small caps
• A description of the first run, and a hypothesis that would need a separate sample to test
• An out-of-sample test
• A robustness check

Show the answer and why

Answer: A description of the first run, and a hypothesis that would need a separate sample to test

The subset was chosen because of its result in the first run, so the second run cannot be independent evidence about it — it is the same data, re-described. The observation is a perfectly good hypothesis. Turning it into evidence requires a sample that was not used to generate it: different symbols, a different period, or both.

Check my answersClear