Lab: Your First Scan

In the next forty minutes you will run one formula across every symbol you have, get back a list of several hundred signals, and then spend most of the time working out why that list contains what it contains. The running is trivial. The reading is the lab.

▷Exercise

You need a database with at least a few dozen symbols and a year or more of daily bars. Part 3’s starter universe is enough. No live data feed, no Professional edition, no paid subscription — the whole lab runs on end-of-day data.

Goal

Produce a list of moving-average crossover signals across a whole market, restricted to instruments liquid enough to be worth looking at, and be able to answer four questions about the output: how many rows, why that many, why one symbol appears more than once, and what would have to change to make the list shorter.

The scan formula

Complete runnable AFL

first-scan.afl

// first-scan.afl
// Part 12 - Lab: Your First Scan
//
// A Scan formula, and nothing else. Scan mode reports the signals your rules
// produce: for every bar in the range on which Buy, Sell, Short or Cover is
// true, AmiBroker writes one line saying which symbol it was and when it
// happened. The columns of that list are fixed and you do not design them.
// That is exactly what separates a Scan from an Exploration.
//
// Assumptions, stated so they can be checked:
//   - Daily bars. Nothing here is interval-aware, so running it on weekly
//     data changes the meaning of every period without warning you.
//   - Volume is share volume in the symbol's own currency terms. If your feed
//     reports volume in hundreds, or in currency already, the turnover floor
//     below is measuring something other than what it says.
//   - A signal is evaluated on the closing price of the bar it is reported on.
//     Appearing in this list is not a claim that you could have traded there.

_SECTION_BEGIN( "First scan" );

FastPeriod      = 50;
TrendPeriod     = 200;
LiquidityPeriod = 50;
MinTurnover     = 5000000;  // average daily turnover, in the instrument's currency
MinPrice        = 5;

// Turnover rather than share volume: ten million shares of a twenty-cent
// stock is not a liquid instrument, and raw share counts are not comparable
// across symbols at all.
Turnover    = Close * Volume;
AvgTurnover = MA( Turnover, LiquidityPeriod );

Liquid  = AvgTurnover > MinTurnover AND Close > MinPrice;
UpTrend = Close > MA( Close, TrendPeriod );

// Cross() is an event: true only on the bar where the crossing occurred, not
// on every bar where price happens to be above the average.
Buy  = Liquid AND UpTrend AND Cross( Close, MA( Close, FastPeriod ) );
Sell = Cross( MA( Close, FastPeriod ), Close );

// Nothing is cleaned with ExRem() here on purpose. A scan reports every signal
// it finds between the start and the end of the range, so one symbol can and
// will appear several times. That is documented behaviour, not a fault, and it
// is the first thing the lab asks you to observe.

_SECTION_END();

Download first-scan.afl46 lines

How it works

The formula has three sections and no output code at all, which is the point: a Scan’s output is built by AmiBroker, not by you.

The liquidity section computes turnover as Close * Volume and takes a fifty-bar average of it. Turnover in currency, rather than share count, is the only version of this that survives comparison across symbols — ten million shares of a twenty-cent instrument and ten thousand shares of a two-hundred-currency-unit instrument are not remotely the same kind of trading activity, and share volume alone says they are.

The context section is a single state: is the close above its two-hundred-bar average? A state is true over a span of bars. On its own it would generate a signal on every bar of a long uptrend.

The signal section turns that state into events. Cross( Close, MA( Close, 50 ) ) is true only on the bar where the close moved from below the fifty-bar average to above it — one bar, not a span. Combining an event with two states, Buy is true only on the bars where the crossing occurred and the trend and liquidity conditions held.

Sell is deliberately not conditioned on liquidity or trend. A rule for getting out that depends on the same filters as the rule for getting in can leave you in a position that never generates an exit signal, which is a real and expensive class of bug that Part 28 returns to. In a scan it matters less, but the habit is worth forming now.

Key functions

Call	What it gives you
MA( array, periods )	A simple moving average of the array over the given number of bars. Null until enough bars exist.
Cross( array1, array2 )	1 on the bar where array1 crossed from below to above array2, 0 everywhere else. Directional: Cross(a,b) and Cross(b,a) are different questions.
Close * Volume	Not a function — elementwise arithmetic on two arrays, producing one turnover figure per bar.

Running it

1. Save the formula. In the Formula Editor, File → Save As, into your own formulas folder — not into AmiBroker’s supplied Formulas\ subfolders, which a future upgrade may overwrite.
2. Send it to the Analysis window with the Formula Editor’s Send to Analysis button. The formula path is filled in for you.
3. Set Apply to. For this lab choose All symbols the first time, deliberately, so that you can see what a broad universe looks like before you learn to narrow it.
4. Set Range to All quotations.
5. Press Scan.

Rows begin appearing before the run finishes. Wait for it to complete before counting anything.

Expected result

=Expected result

A result list with one line per signal, carrying the ticker, the date and time of the bar, the signal type and a price. On a database of a few hundred symbols with several years of daily history, expect several hundred to several thousand rows. Many symbols appear repeatedly. Some symbols do not appear at all.

If you got zero rows, stop here and check three things in this order: was Scan the button you pressed; is Apply to really set to All symbols rather than Current symbol; and does your data actually have volume in it? A database imported without a volume column gives Volume = 0 for everything, the turnover filter rejects every symbol, and the scan is silent about it. The challenge later in this part is built on exactly this family of failure.

Reading the result list

Now do the part that matters.

Sort by ticker

Click the ticker column header. The list re-sorts. You will see blocks of consecutive rows for the same symbol — a Buy in March, a Sell in May, a Buy in July, and so on.

This is not a defect. A Scan walks to the end of the range and reports every signal it finds there, which is what makes it useful for the question “when did this happen?”. If your database holds ten years of daily bars, a fifty-bar crossover on a moderately trending instrument fires somewhere between ten and forty times over that period. Forty rows for one symbol is a plausible and correct answer.

Sort by date

Click the date column. Now the list reads as a chronology across the whole market, and something else becomes visible: signals cluster. A large number of symbols cross their fifty-bar average within a few days of each other, because they are all responding to the same broad market move.

That clustering has consequences you will meet repeatedly in this course. It means the rows are not independent observations of anything. It means a screen run on the wrong day returns three candidates and on the right day returns ninety. And it means that any study which treats each signal as a separate data point is overstating its sample size, sometimes by an order of magnitude.

Count the rows and ask why

Look at the row count. Then reason about it: number of symbols × number of years × roughly how often a fifty-bar crossover fires per year, halved because the trend filter removes the signals that occurred below the two-hundred-bar average, and reduced again by however many symbols failed the liquidity floor for their whole history.

If your estimate and the actual count differ by more than a factor of two, one of your assumptions about the data is wrong. Find out which before moving on. That habit — predicting the output before reading it — is worth more than any individual technique in this part.

Why the scan returned what it returned

Four separate things decided that output, and only two of them are in the formula:

1. The rules — Cross() is an event, so signals are sparse rather than continuous.
2. The liquidity floor — MinTurnover = 5000000 is a number chosen for a large, liquid market. On a smaller exchange it may reject the entire universe. It is the single most universe-dependent line in the file.
3. Apply to — All symbols included every composite ticker, index and defunct symbol in your database.
4. Range — All quotations meant every bar of history was eligible to produce a signal.

Change any one of the four and the list changes. Only two of them are saved with the formula, which brings us to the last step.

Saving the analysis project

A formula file is not a reproducible piece of research, because it does not record the universe or the range. AmiBroker’s answer is the Analysis Project, an .APX file that is self-contained: it holds the formula, all the options and settings, and the Apply to and Range selections together in one place.

With the Analysis window as the active window, use the File menu to save the analysis project. (If you cannot find the item, the Analysis window is not the active window — the same condition governs the Export entries you will meet later in this part.)

Save this one as first-scan.apx. Two reasons, one immediate and one for later:

• Immediate. Tomorrow you can reopen it and get the same run, rather than reassembling the settings from memory and quietly changing the answer.
• Later. The Batch window loads .APX files. Everything in the “Exporting Results and Building a Daily Workflow” lesson at the end of this part depends on having saved your analysis as a project first.

✦Tip

Name analysis projects after the question they answer, not after the indicator they use. liquid-ma-crossovers.apx will still mean something in six months. ma50.apx will not.

Test it

A scan you have not tested is a list of assertions. Three checks, in increasing strength:

1. Single-symbol check. Set Apply to = Current symbol on a symbol you know produced signals, and re-run. Then open that symbol’s chart, plot Close and MA(Close,50), and confirm by eye that a crossing really happened on the dates the scan reported. If the dates are off by one bar, you have found a genuine and important bug — go back to Part 9 and re-read the sign convention of Ref().
2. Falsification check. Change Buy to Buy = Liquid AND UpTrend AND Cross( MA( Close, FastPeriod ), Close ); — the crossing in the opposite direction — and re-run. The dates should be completely different. If they are not, your Cross() arguments are not doing what you think.
3. Filter-removal check. Comment out AND UpTrend and re-run. The row count must go up, and it must go up by a plausible amount. A filter that removes nothing is not a filter, and a filter that removes 99% of the rows is probably a bug.

Common errors

✕Common mistake

Zero rows, no error. In order of likelihood: the Explore button was pressed instead of Scan; Apply to is on Current symbol; the liquidity threshold is calibrated for a different market; the database has no volume data. The formula is the last place to look, not the first.

Every symbol appears on the same date. You are looking at a database where every symbol has the same last bar and the range is one bar. That is a range setting, not a signal.

The prices in the result list are not the prices you expected. A scan reports a price associated with the signal bar. If your BuyPrice is unassigned, AmiBroker uses its default. Part 28 covers signal prices properly; for now, treat the price column as an identifier of the bar rather than as a fill.

Signals on symbols you thought were filtered out. Check whether those symbols are composites or indexes. Apply to = All symbols includes them, and an index has a plausible price and often a large or zero volume, which sails through a naive liquidity test.

Extension

Add signal cleaning and observe what it does to the count. ExRem() removes redundant signals, keeping only the first of a run:

Fragment — not a complete formula

Buy  = ExRem( Buy, Sell );
Sell = ExRem( Sell, Buy );

Run the scan again and compare the row count. On a crossover system the change is usually small, because Cross() already produces isolated events — which is itself an instructive result. Now change Buy to the state Liquid AND UpTrend AND Close > MA( Close, FastPeriod ) and run it both with and without ExRem(). The difference will be dramatic: thousands of rows become dozens. That contrast is the state-versus-event distinction from Part 9, measured rather than described.

What changed in your understanding

You have now run a formula against a market rather than a chart, and seen that the output is shaped as much by two combo boxes as by the code. You have seen that repeated signals per symbol are correct behaviour, that signals cluster in time across the whole universe, and that a formula without its Apply to and Range settings is not a reproducible piece of work — which is what the .APX file exists to fix.

Check your understanding

Question 1. Your scan returns 4,000 rows across 200 symbols and 10 years. Which change would reduce the row count the most, without changing the rules?

• Adding ExRem() to Buy and Sell
• Setting Range to "1 recent bar(s)"
• Sorting the result list by date
• Switching from All symbols to All symbols with favourites included

Show the answer and why

Answer: Setting Range to "1 recent bar(s)"

Range decides how many bars can produce a signal. Restricting it to the most recent bar asks "who is signalling now?" instead of "when did this ever happen?". ExRem changes little here because Cross() already yields isolated events, and sorting changes nothing at all.

Question 2. What does an .APX analysis project file contain that a .afl formula file does not?

• The result list from the last run
• The Apply to and Range selections and the other analysis settings
• A compiled version of the formula that runs faster
• The data for the symbols in the universe

Show the answer and why

Answer: The Apply to and Range selections and the other analysis settings

An Analysis Project is self-contained: formula plus all options and settings plus the apply-to and range selections. That is exactly what makes a run reproducible, and it is what the Batch window loads.

Question 3. True or false: a symbol appearing eleven times in a scan result list indicates a bug in the formula.

• True
• False

Show the answer and why

Answer: False

False. A Scan proceeds to the end of the range and reports every signal it finds, so multiple lines per symbol are documented behaviour. Whether eleven signals in ten years is too many is a question about your rules, not about the scan.

Check my answersClear