Ranking versus Filtering

Run a screen for “close above the 200-day average and momentum above 20 per cent” in a strong market and it returns four hundred names. Run the same screen eight months later and it returns nine. Nothing about the screen changed. What changed is that a filter is a question with an absolute answer, and markets move the whole distribution around.

Ranking does not have that property. Ask for the twenty strongest names and you get twenty, in every market, always. That is either exactly what you wanted or a trap, depending on a distinction this lesson is about making explicit.

Two shapes of question

A time-series question is answered inside one symbol’s own history. Is this close above this symbol’s own 200-day average? Everything needed is in one set of arrays. The answer for ABC does not depend in any way on XYZ.

A cross-sectional question is answered across symbols at one moment. Is this symbol’s 126-bar return in the top twenty of its universe today? Nothing in ABC’s arrays can answer that. You have to have looked at every other symbol first.

This is not a philosophical distinction; it has a direct consequence in AmiBroker. Because a filter is self-contained, the Analysis window can hand each symbol to a separate thread and never co-ordinate them — the documented model is one thread per symbol per operation, up to 2 threads per Analysis window on the Standard edition and up to 32 on Professional. Because a ranking is not self-contained, that same independence is in your way: the executions that hold the numbers you need are running beside you, not before you.

What a cross-sectional answer requires

1. 1UniverseDecide who is in the comparison
2. 2Score eachOne number per symbol per bar
3. 3CollectSomewhere all of them can be seen at once
4. 4OrderSort the scores across symbols
5. 5SelectTake the top N, or read your own rank

Only the middle step is new. It is also the one plain AFL has no way to perform.

A filter produces a set; a rank produces an order

	Filter	Rank
Output	Membership: in or out	Position: 1st, 2nd, 3rd
Needs other symbols?	No	Yes
Size of result	Varies with the market	Fixed by you
Fails by	Returning nothing, or everything	Returning something regardless
Threshold lives in	The formula	The number you choose for N

Read the last two rows together, because they are the whole argument. A filter tells you when nothing qualifies. That is information: if a momentum threshold returns nine names out of five hundred, the market has changed, and the screen has said so. A ranking cannot tell you that. Ask it for twenty and it returns twenty even if all five hundred scores are negative — the top of a falling distribution is still the top.

Neither behaviour is better. They are answers to different questions, and the useful move is to use both for what each is good at: filter for eligibility, rank for order.

Eligibility first, order second

Eligibility rules are the ones where an absolute threshold genuinely means something: enough turnover that a position could be built, a price above the level where tick size dominates, enough history for the score to be defined at all. Those belong in Filter. Ranking then answers “of the ones I could act on, which stand where?”.

Goal

An Exploration that keeps the two jobs visibly apart, so you can watch the eligible set grow and shrink while the ordering carries on regardless.

Complete formula

Complete runnable AFL

filter-then-rank.afl

// filter-then-rank.afl
// Part 13 - Ranking versus Filtering
//
// An Exploration that does both jobs and keeps them visibly separate: a filter
// decides who is ELIGIBLE, a score decides who comes FIRST. Toggling between
// "eligible only" and "everything" shows how differently the two behave when
// the market changes.
//
// How to run it: Analysis window, Apply to = Filter (a watch list) or All,
// Range = 1 recent bar, then Explore.
//
// Assumptions declared up front:
//   - Daily bars. The turnover threshold is in the same currency units as
//     Close * Volume, so it has to be re-set for a database in another currency.
//   - Median turnover, not average turnover: one gap-up day on ten times normal
//     volume should not qualify a normally untradeable stock.
//   - The sort below orders the RESULT TABLE. Nothing in this formula can read
//     back the position a symbol ended up in. That limitation is exactly why
//     the next lesson exists.

MinPrice     = Param( "Minimum close price",        5, 0, 200, 0.5 );
MinTurnover  = Param( "Minimum median turnover", 2000000, 0, 50000000, 100000 );
LiquidPeriod = Param( "Turnover median period",    50, 5, 200, 1 );
ScorePeriod  = Param( "Momentum lookback (bars)", 126, 20, 500, 1 );
ShowAll      = ParamToggle( "Rows to show", "Eligible only|Every symbol", 0 );

// ------------------------------------------------------------- eligibility
// Every test below uses only THIS symbol's own data. That is what makes a
// filter cheap: no symbol needs to know anything about any other symbol, so
// AmiBroker can run them all on separate threads without co-ordination.
Turnover      = Close * Volume;
MedTurnover   = Median( Turnover, LiquidPeriod );

LiquidEnough  = MedTurnover >= MinTurnover;
PricedEnough  = Close >= MinPrice;
HistoryEnough = BarIndex() >= ScorePeriod;

Eligible = LiquidEnough AND PricedEnough AND HistoryEnough;

// -------------------------------------------------------------------- score
// One number per bar per symbol. On its own it means nothing at all; it only
// acquires meaning when it is compared with the same number for other symbols.
MomentumScore = ROC( Close, ScorePeriod );

Filter = IIf( ShowAll, HistoryEnough, Eligible );

AddColumn( Close,         "Close",            1.2 );
AddColumn( MedTurnover,   "Median turnover",  1.0 );
AddColumn( MomentumScore, "Momentum %",       1.2 );
AddTextColumn( WriteIf( LiquidEnough,  "", "thin " ) +
               WriteIf( PricedEnough,  "", "low-priced " ) +
               WriteIf( HistoryEnough, "", "short-history " ) +
               WriteIf( Eligible, "eligible", "" ), "Status" );

// Column 1 is the ticker and column 2 is the date, so the first AddColumn
// above is column 3 and the momentum score is column 5. A negative number
// sorts descending.
SetSortColumns( -5 );

Download filter-then-rank.afl58 lines

How it works

The formula splits into three parts. The eligibility block builds three independent Boolean arrays — liquid enough, priced enough, long enough — and combines them. Every one of those tests uses only this symbol’s own arrays, which is what makes them cheap.

The score block computes one momentum number per bar. It is deliberately separate from the eligibility block and deliberately not compared against any threshold, because a score’s job is to be ordered, not to be passed or failed.

The output block sets Filter, adds the columns, and calls SetSortColumns(). The ParamToggle lets you flip between showing the eligible set and showing every symbol with enough history, which is the fastest way to see how much the eligible set moves around as the market changes.

Key functions

• Median( array, period ) — the median of the last period values. Used on turnover rather than an average, so that a single frantic day cannot qualify a stock that is normally untradeable.
• ParamToggle( "name", "values", defaultval = 0 ) — a two-state parameter; the values string separates the labels with a vertical bar.
• SetSortColumns( col1, col2, ... ) — sets the sort. Column numbers are one-based, a negative number sorts descending, and up to ten columns may be given.
• WriteIf( EXPRESSION, "TRUE TEXT", "FALSE TEXT" ) — builds the per-row status text.

!Important

Column numbering trips everyone up once. In Exploration output, column 1 is the ticker and column 2 is the date, so the first AddColumn() you write is column 3. The official AddRankColumn example makes this concrete: after adding Close and Volume it refers to the volume column as the 4th.

Expected result

With “Eligible only” selected you get a table ordered by momentum, strongest first, and the Status column reads “eligible” on every row. Switch to “Every symbol” and the table grows, the excluded rows appear with their reason spelled out, and — this is the part worth pausing on — some of them are near the top of the momentum ordering.

Test it

Run it on a watch list twice: once with the range set to a recent bar, once with the range set to a bar in a quiet or falling period. Record the number of eligible rows each time. Then set the turnover threshold to zero and repeat. The gap between the two eligible counts is what a filter does that a ranking does not.

Common errors

• Every row excluded as “thin”. The turnover threshold is in the currency units of Close * Volume in your database. On an index-level database, or one priced in a different currency, the default is nonsense.
• The sort appears to do nothing. Check the column number. Sorting column 4 when you meant column 5 produces a perfectly valid ordering of the wrong thing.
• An empty table. Almost always the range: an Exploration over “1 recent bar” with a HistoryEnough test that no symbol passes returns nothing at all.

Extension

Add a second score — say, distance above the 200-bar average — as another column, and call SetSortColumns() on it instead. Watch how much the top of the table changes when only the ordering criterion changes and the eligible set does not.

The limit you have just hit

SetSortColumns() orders the result table. It is a display instruction. Nothing in your formula can read back the position a symbol ended up in, and nothing in your formula can act on it, because the sort happens after every symbol has already been processed.

AddRankColumn() adds a rank column according to the current sort. It is worth knowing about, with one caution: the AFL Function Reference page for it carries the version stamp “(AmiBroker 7.70)”, which is ahead of the 7.00.1 release this course is validated against. Treat its availability as build-dependent, and check your own installation’s function list before relying on it. SetSortColumns(), stamped 4.90, is the part you can count on.

Either way, both are output formatting. The moment you want the rank to be an input — to select the top twenty inside the formula, to compute a percentile, to compare a symbol’s rank today against its rank a month ago — you need the rank as an array in AFL, and that is what the next two lessons build.

Top-N and what N is really choosing

N looks like a display preference. It is not. It sets three things at once:

• How far down the distribution you go. Twenty out of five hundred is the top 4 per cent; twenty out of forty is the top half. The same N is a different selectivity in a different universe, which is why a percentile is often the more honest quantity.
• How much the selection turns over. A small N is dominated by the boundary: symbols crossing in and out of the last place account for most of the churn.
• How concentrated any resulting portfolio would be. That is a risk decision wearing a screening decision’s clothes, and it belongs in Part 34, not in a ranking formula.

Ties and stability

Exact ties in a floating-point score are rare. Ties in a rounded score are common, and so are near-ties, which behave like ties for every practical purpose: two symbols whose scores differ in the fourth decimal place will swap places on noise.

AmiBroker’s ranking function takes a tie mode argument with two documented values, 1234 and 1224. In 1224 mode ties are numbered with an equal rank; in 1234 they are numbered consecutively. Which you want depends on what the rank feeds: if you are going to select “rank 20 or better”, 1224 can hand you more than twenty symbols, and 1234 will not.

✦Tip

If a selection built on ranks flickers from day to day, do not reach for a smoother score first. Add hysteresis: a symbol enters the selection when its rank reaches, say, 15 or better, and leaves only when it falls past 30. Two thresholds instead of one removes most boundary churn without changing the measurement at all.

When ranking changes the answer entirely

Consider a universe of five hundred shares in a month where every one of them is down over the past six months. A momentum filter with a fixed threshold returns an empty list, and that emptiness is a true and useful statement about the market. A momentum ranking returns the same twenty rows it always returns, each one a stock that fell less than the others.

Both outputs are correct. Only one of them is capable of telling you that the question you asked has no good answer today. If a ranking is going to drive anything, something else in the system has to carry that information — an eligibility filter with an absolute floor, a market regime test, or simply reading the score column instead of only the rank column.

✕Common mistake

The symptom is a screen that “always finds something”. A user reports that their strongest-stocks list looked perfectly normal through a period when the whole market was falling. Nothing was broken. A rank has no absolute scale, so it cannot report that the whole distribution moved. Add the raw score to the output and the problem becomes visible immediately.

Recap

Filters and ranks answer differently shaped questions. A filter is self-contained, returns a set whose size varies with the market, and can tell you when nothing qualifies. A rank needs the whole cross-section, returns a fixed-size ordering, and cannot. Used together — eligibility as a filter, selection as a rank, with the score reported alongside — they cover each other’s blind spots.

You have also met the wall this part exists to get past: sorting an Exploration is a display instruction, not a value your formula can use. The next lesson explains why AFL cannot see across symbols on its own, and what AmiBroker provides instead.

Check your understanding

Question 1. A momentum filter with a fixed threshold returns 3 symbols out of 500 this month, against 200 last month. What has the screen told you?

• The screen is broken
• The threshold is too high and should be lowered
• The distribution of momentum across the universe has moved - which is information
• Nothing useful; only a ranking is informative

Show the answer and why

Answer: The distribution of momentum across the universe has moved - which is information

A varying result size is what a filter reports that a ranking cannot. Lowering the threshold to restore the old result count would discard exactly the information the screen just gave you.

Question 2. Which statements about SetSortColumns() are correct? Select all that apply.

• Column numbers are one-based
• A negative column number sorts descending
• The formula can read back the position each symbol was sorted into
• In Exploration output the first AddColumn() you write is column 3

Show the answer and why

Answer: Column numbers are one-based, A negative column number sorts descending, In Exploration output the first AddColumn() you write is column 3

Sorting is applied to the result table after every symbol has been processed, so it is a display instruction. Column 1 is the ticker and column 2 the date, which is why the first added column is the third.

Question 3. You select "rank 20 or better" from a ranking generated in tie mode 1224. How many symbols can you get?

• Exactly 20
• At most 20
• 20 or more, because ties share a rank
• It depends on the score, not the tie mode

Show the answer and why

Answer: 20 or more, because ties share a rank

In 1224 mode tied scores are numbered with an equal rank, so several symbols can hold rank 20. Tie mode 1234 numbers ties consecutively and would cap the selection at 20.

Question 4. Why can AmiBroker run a filter over 500 symbols on several threads without any co-ordination between them?

• Because filters are simpler arithmetic than ranks
• Because each symbol’s answer depends only on that symbol’s own arrays
• Because the Analysis window sorts the results afterwards
• Because static variables are thread-safe

Show the answer and why

Answer: Because each symbol’s answer depends only on that symbol’s own arrays

A time-series test is self-contained, so the executions never need to exchange anything. A cross-sectional question breaks that independence, which is the whole difficulty of ranking.

Check my answersClear