Repeat Scanning: Mechanics and Costs

There is exactly one built-in mechanism in AmiBroker for running a scan over and over during a trading session, and it is a checkbox. Everything that makes repeat scanning hard sits either side of that checkbox: how long your formula takes multiplied by how many symbols you point it at, how many symbols your vendor will stream you, and the uncomfortable fact that a result list is not a picture of the market at a moment — it is a summary of a period, and the period ended before you read it.

By the end of this lesson you should be able to turn repeat scanning on, measure what one pass of your own formula costs, choose an interval from that measurement rather than from a guess, state what your data vendor’s documented symbol limits are and why exceeding them is a bad idea, and put an honest number on how old the oldest row in your result list is.

✱AmiBroker Professional required

The checkbox, and what it does not do

In the New Analysis window, the Settings button has a drop-down arrow. The menu behind it carries four options: Sync chart on select, Wait for backfill, Auto repeat Scan/Explore and Auto repeat interval. Ticking the third and filling in the fourth is the whole of the built-in repeat mechanism.

The interval field has one piece of syntax worth memorising, because it is not obvious and it is documented in a single sentence: a plain number means minutes. Typing 5 gives you a scan every five minutes. To get five seconds you type 5sec or 5s and press Enter.

Three properties of this mechanism decide how you should think about it.

It is a timer, not a subscription. The scan re-runs on schedule whether or not a single new trade has arrived. Nothing in AmiBroker tells your Analysis window “symbol X just changed, look at it”; you re-ask the entire question of the entire universe, every time. That is why the cost arithmetic below matters so much: you pay the full price of the scan on every repetition, including for the several hundred symbols where nothing whatsoever has happened.

It is per Analysis window. Several Analysis windows can run at once, each with its own formula, universe, range and repeat interval. A fast scan over twenty symbols and a slow one over two thousand do not have to share a schedule.

And the result list is non-blocking: you can scroll and sort it while the run is still generating rows. This is convenient and it is a trap. A half-filled list looks exactly like a finished list with fewer hits.

✕Common mistake

Reading a result list before the run has finished and concluding “only three symbols qualified today”. The row count is not final until the run is. Watch the progress indicator, not the list.

What one pass costs

AmiBroker’s threading rule is published and blunt: one operation on one symbol is one thread. A scan over N symbols is N threads’ worth of work. The number that actually run at once is capped twice — by your edition, and by your hardware. The Standard edition allows 2 simultaneous threads per Analysis window; Professional allows 32, and AmiBroker will not exceed the number of logical processors Windows reports either way.

So the wall-clock time of one pass is roughly:

Pseudocode — not valid AFL

run time  ~=  symbols  x  time per symbol  /  min( edition thread limit, logical CPUs )

The interesting term is time per symbol, and it has two independent halves that fail for different reasons. The first is your formula’s array work, which you control. The second is the data plugin’s work fetching and preparing that symbol’s bars, which you do not. AmiBroker’s performance chapter publishes a diagnostic for the second: it reports “Plug-in time per symbol”, and states that if this exceeds 10 ms the plugin is slow or is not using the current data-plugin interface, and you should ask the vendor for an update. That is a rare thing in this documentation — an explicit numeric threshold — and it is worth checking before you spend an afternoon optimising AFL that is not the bottleneck.

Memory enters through the same door. A data bar costs 40 bytes per bar per symbol, so a database set to load 100,000 bars allocates about 4 MB for every symbol it touches. The performance chapter’s advice is to keep the array small enough to stay in CPU cache and calls oversizing “Number of bars to load” the most common mistake people make. A repeated scan makes that mistake expensive every interval instead of once.

Measuring instead of guessing

GetPerformanceCounter() reads the Windows high-resolution timer in milliseconds. Two documented details make it usable: passing True resets the counter for the calling formula — and you should, because otherwise you are subtracting two very large numbers in a language carrying about seven significant digits — and the call itself costs about 0.015 ms, so nothing shorter than that can be measured with it.

The formula below uses it to answer four questions at once: how long a pass takes per symbol, which thread ran each symbol, what the clock said when each row was produced, and how old the newest bar was at that instant.

Goal

Produce a per-symbol cost and staleness report for a universe you are considering scanning repeatedly, so that the repeat interval is chosen from a measurement.

Complete formula

Complete runnable AFL

scan-cost-probe.afl

// scan-cost-probe.afl
// Part 24 - Repeat Scanning: Mechanics and Costs
//
// Measures what a repeated scan costs and how stale its rows are by the time
// you read them. It performs no market analysis whatsoever: it does a fixed,
// deliberately ordinary amount of array work per symbol and then reports how
// long that took, which thread ran it, what the PC clock said at the moment the
// row was produced, and how old the newest bar was at that moment.
//
// How to run it:
//   Formula Editor -> paste -> name it -> Send to Analysis
//   Apply to: Filter, and choose the watch list you would really scan
//   Range:    1 recent bar(s)
//   Press Explore once and read the columns.
//   Then open the Settings drop-down, tick "Auto repeat Scan/Explore", set an
//   interval, and watch which columns change between runs and which do not.
//
// What to look at:
//   - "Row produced at" is NOT the same for every symbol. The spread between the
//     first and last row is how long the run took, and it is the smear that
//     makes a scan result a summary of a period rather than a snapshot.
//   - "Formula time (ms)" x symbol count / threads is roughly your floor on the
//     repeat interval. If that product approaches the interval, runs overlap.
//
// Assumptions declared up front:
//   - "Row produced at" reads the PC clock through Now(5). Bar timestamps come
//     from the database. If the database has a time shift configured, the two
//     are on different clocks and "Bar age" is wrong by exactly that shift.
//     Status("timeshift") reports the shift in seconds; the last column prints
//     it so you can see whether you need to correct for it.
//   - The WorkLoad block below is a stand-in. Replace it with something the same
//     shape as the formula you actually intend to repeat, or the timing tells
//     you about this file rather than about your scan.

SetBarsRequired( sbrAll, sbrAll );   // v5.20 constant: turn QuickAFL off so every
                                     // run does the same amount of work

GetPerformanceCounter( True );       // reset this formula's counter before timing

// A block of unremarkable array arithmetic, sized so that the measurement is
// above the documented ~0.015 ms call overhead of the counter itself.
WorkLoad = 0;
for( n = 1; n <= 8; n++ )
{
    WorkLoad = WorkLoad + MA( Close, 10 * n ) - MA( Close, 5 * n );
}
WorkLoad = WorkLoad + ATR( 14 ) + StDev( Close, 20 );

ElapsedMs = GetPerformanceCounter();   // milliseconds since the reset above

RowClock  = Now( 5 );                          // PC clock, in DateTime format
BarStamp  = DateTime();                        // bar timestamps, from the database
BarAgeSec = DateTimeDiff( RowClock, BarStamp ); // positive when the bar is older

// One row per symbol: only the last bar of the analysis range is reported.
Filter = Status( "lastbarinrange" );

AddColumn( BarCount,               "Bars in array",     1.0 );
AddColumn( BarStamp,               "Newest bar",        formatDateTime );
AddColumn( RowClock,               "Row produced at",   formatDateTime );
AddColumn( BarAgeSec,              "Bar age (s)",       1.0 );
AddColumn( ElapsedMs,              "Formula time (ms)", 1.3 );
AddColumn( Status( "ThreadID" ),   "Thread",            1.0 );
AddColumn( Status( "timeshift" ),  "DB time shift (s)", 1.0 );
AddColumn( LastValue( WorkLoad ),  "Work result",       1.4 );

// Column 5 is "Row produced at". Sorting ascending puts the oldest row at the
// top, so the age of the list is visible without doing arithmetic.
SetSortColumns( 5 );

// MIN (4) + MAX (8) + COUNT (16), restricted to the two numeric columns where a
// minimum and a maximum mean something.
AddSummaryRows( 4 | 8 | 16, 1.3, 6, 7 );

Download scan-cost-probe.afl73 lines

How it works

The formula has four sections. It first turns QuickAFL off with SetBarsRequired( sbrAll, sbrAll ), so that every run evaluates the same number of bars and the timings are comparable between runs and between symbols. It then resets the performance counter, performs a block of ordinary array arithmetic as a stand-in for real analysis, and reads the counter again — the difference is the milliseconds that block consumed for this symbol on this thread.

The third section captures two different clocks. Now(5) returns the PC clock in DateTime format at the moment this symbol’s thread reached that line, which is not the same instant for every symbol in the run. DateTime() returns the bar timestamps out of the database. DateTimeDiff() gives the gap between them in seconds.

The last section reports one row per symbol by accepting only the final bar of the range, sorts ascending by the wall-clock column so the oldest row sits at the top, and asks for minimum, maximum and count summary rows on the two columns where those statistics mean something.

Key functions

GetPerformanceCounter( bReset = False ) returns milliseconds; reset it before timing anything. Status( "ThreadID" ) returns the thread the formula is executing on. Status( "timeshift" ) returns the database time shift in seconds — the amount by which your bar timestamps and your PC clock have been deliberately separated. DateTimeDiff( arg1, arg2 ) returns the difference between two DateTime values in seconds, and is the sanctioned way to compare them, because the ordinary > and < operators on DateTime values are documented as unreliable. AddSummaryRows( flags, format, onlycols... ) adds statistics rows at the top of the list, and its format argument defaults to a maximum-precision setting that prints up to fifteen digits, so always pass one.

Expected result

One row per symbol. The “Row produced at” values will differ, and the spread between the first and the last is the duration of the pass. On a machine with more logical processors than your edition’s thread limit you should see only as many distinct thread IDs as that limit allows. “Formula time (ms)” will vary between symbols mostly with the number of bars loaded.

=Expected result

If “Bar age (s)” is a large negative number or a number close to a whole multiple of 3,600, look at the “DB time shift (s)” column before you conclude your feed is broken. You are comparing a PC clock with deliberately shifted bar timestamps.

Test it

Run it once on ten symbols and note the total run duration reported by the window. Now run it on a hundred symbols from the same database. If threading is behaving, the run time should grow far more slowly than tenfold. Then set the repeat interval to something shorter than the measured run time and watch what happens to the list — this is the overlap condition described below, and it is much more convincing observed than described.

Common errors

Forgetting the True on the first GetPerformanceCounter() call gives you milliseconds since the machine booted, and subtracting two such numbers in single-precision arithmetic produces timings that look plausible and are noise. Leaving QuickAFL on makes short-range runs faster than long-range ones for reasons that have nothing to do with your formula. And comparing Now(5) with DateTime() on a database with a configured time shift gives a “bar age” that is wrong by exactly the shift, every time, on every symbol.

Extension

Replace the WorkLoad block with the actual formula you intend to repeat and re-run it. The cost figure only means something when the work is the work you plan to do.

Choosing the interval

Three constraints, in order of how often they are ignored.

The interval must exceed the run time. If a pass takes 40 seconds and you ask for one every 30, passes overlap. AmiBroker will keep going, but you are now paying for two concurrent runs, competing for the same thread budget, and the list you are reading is being rebuilt underneath you. Measure first, then choose, and leave headroom — run time is not constant, because backfills and vendor latency are not constant.

The interval should relate to the bar interval. If your scan reads only completed bars, then on five-minute data nothing your formula can see changes between 10:35:01 and 10:39:59. Scanning every ten seconds during that window performs thirty passes to discover the same thing thirty times. If your scan reads the forming bar as well — which is the normal intraday case, because the newest bar’s close is the last trade so far — then more frequent scanning does surface changes, but each of those changes may be undone before the bar completes.

The interval buys you nothing your data does not already contain. A one-second repeat against a feed that updates your database every few seconds is measuring your own impatience. Part 21 covers the difference between the feed updating and the display updating; the same distinction applies here.

!Important

The documentation describes the auto-repeat interval as an interval, and says nothing about aligning repetitions to bar boundaries. Do not assume a scan set to five minutes fires just after each five-minute bar completes. If your logic depends on running just after a bar closes, verify the actual firing times on your own machine before you rely on them — the probe formula’s “Row produced at” column is how you check.

What your data vendor allows

This is the constraint that catches people who have already solved the CPU problem. A streaming subscription is sold with a symbol limit, and AmiBroker expects you to tell it what that limit is: the plugin’s configuration dialog, reached through File → Database Settings → Configure, has a symbol-count field, and the official guidance is that it should match your subscription.

The documented ceilings, from the User’s Guide and the Knowledge Base, are worth knowing as orders of magnitude rather than as current commercial fact:

Source	Documented symbol ceiling
Interactive Brokers	100, described as a TWS limit
DTN IQFeed	500 by default, depending on subscription
DDE	500
eSignal, myTrack	As per subscription, entered in the plugin configuration; AmiBroker re-adjusts downward if exceeded

What happens when you exceed the limit is documented and is worse than a simple refusal. AmiBroker rotates symbols: you may add more tickers to the database than your subscription allows, and it keeps the most recently used ones active. Each new symbol that arrives therefore drops the oldest, subscribes the new one, triggers a backfill from its last valid update, and then begins streaming it. Run a screen across several hundred symbols on a fifty-symbol subscription and you are executing that cycle continuously. The Knowledge Base is explicit that this can overload the vendor’s servers and that vendors may act against accounts that abuse streaming limits. The rotation mechanism also does not apply to the Real-Time Quote window, which simply cannot hold more symbols than your subscription allows.

Backfill behaviour is the second vendor constraint, and the Knowledge Base sorts sources into three classes: unlimited backfill (eSignal, IQFeed are the named examples), limited, one symbol at a time (Interactive Brokers), and no backfill at all (DDE, where the documented remedy is ASCII import). Backfill is always on demand and per symbol, requested on first access — which means the first pass of a scan over a universe it has not touched before can silently analyse short or empty arrays.

The Analysis window’s Wait for backfill option exists for exactly that, and it is listed in the official edition table as a Professional-edition feature. It makes the run block until each symbol’s backfill has arrived. That fixes correctness and destroys your timing model: run duration is now set by your vendor’s response times, not by your CPU. It is also documented as a no-op on local databases and on end-of-day plugins, and many third-party plugins never implemented it.

⚠Warning

Do not treat these vendor figures as current. AmiBroker’s data-source directory page carries its own date stamp — it is marked current as of March 2023 — and several provider pages in the User’s Guide describe client software versions and history limits that are visibly older than that. Part 18 records what could and could not be verified. Before you size a scan around a number, confirm it with the vendor.

Why the answer is already old

Assemble the latencies and the result is uncomfortable. Consider a five-minute intraday scan over 400 symbols, and treat every number here as illustrative arithmetic rather than a measurement of any real system.

The age of one row, from bar to eye (illustrative)

Bar forms

up to 5 min of bar

Run

pass takes ~40 s

You look

you glance up

bar opens+2 min+4 minyou read it

Four separate delays stack up. The bar itself is a window, not an instant: with AmiBroker’s default start-of-interval timestamps, a bar stamped 10:35 on five-minute data covers 10:35:00 to 10:39:59, and while it is the newest bar it is unfinished — its close is only the last trade so far and its high and low can still move. The run takes time, and different symbols are examined at different points inside it, so the list is a smear across those forty seconds rather than a snapshot. The interval means the run itself may have started up to five minutes ago. And then there is you, reading a list that has been sitting on screen while you finished something else.

Two consequences follow, and they are the point of the lesson.

First, sort your results by age, not by attractiveness. The probe formula and the project’s scanner both carry a signal-age column for this reason. A list ordered by how good a candidate looks quietly invites you to act on the oldest row in it.

Second, treat a scan result as a filter for attention, not a trigger. What a repeated scan legitimately does is reduce two thousand symbols to six that are worth a human look. What it cannot do is tell you what a price is now, because by construction it tells you what a price was, some seconds or minutes ago, on a bar that had not finished.

?Reality check

The claim: “a real-time scanner catches breakouts as they happen.” Held up against the mechanism, this cannot be right — the scanner sees an unfinished bar, examined at a moment inside a run that started at some earlier point, and reports it into a list you read later still. What a scanner does is notice, with a delay you can measure, that something already happened. The way to test the claim is to record the timestamp on each row and the wall-clock time you acted on it, and to compare the two for a week. Almost nobody who repeats the claim has done that.

One AFL option is worth knowing here. SetOption( "RefreshWhenCompleted", True ) performs a Refresh All once an Analysis operation has finished, and is documented as the thread-safe replacement for driving refreshes through OLE. In a repeated scan that feeds charts you are also watching, it keeps the two consistent without adding a second timer.

Doing all of this without a live feed

None of what you have just read requires a subscription to learn, and two of the three constraints can be measured exactly on a static database.

The cost model needs nothing. Auto repeat is a timer, not a real-time feature: it re-runs your formula on schedule against whatever database is open. Point the probe formula at an end-of-day database with a few hundred symbols, tick Auto repeat, set an interval, and every number in the cost section becomes measurable — run duration, per-symbol formula time, thread count, the effect of edition limits, and the overlap behaviour when the interval is set too short. The only column that will be uninformative is bar age, and it will be uninformative in an instructive way.

The moving data edge comes from Bar Replay. Bar Replay plays historical data back at a chosen speed, and — crucially for this lesson — it is documented as global: it plays back data for all symbols at once, every symbol’s data ends at the playback position, and this affects all formulas “whether used in charts/indicators or in auto-analysis”. So a scan that repeats while replay advances is examining a database whose newest bar keeps changing, which is structurally the same problem a live feed poses.

▷Exercise

Build the offline version now, before the project needs it.

1. Open an intraday database containing at least one full historical session.
2. Send the probe formula to an Analysis window. Set Apply to to a watch list of ten to thirty symbols and Range to 1 recent day(s).
3. Open Tools → Bar Replay. Set Start and End to bracket one session, set Step interval to your database’s base interval, and set Speed to 1 step per second.
4. Press Play. The data now ends at the playback position for the whole program.
5. In the Analysis window, tick Auto repeat Scan/Explore, enter 10s, and press Explore once to start it.
6. Watch the “Newest bar” column advance as replay walks forward, while “Row produced at” advances on the PC clock. Those are the two clocks the whole lesson is about, and now you can see them moving at different rates.

Three documented details make replay behave the way you need. Play and Pause both enter playback mode and both truncate the data — Pause is not “do nothing”, it is “stop here with the future still hidden”, which is what you want while you inspect a list. Stop, or closing the dialog, restores the full data set. And nothing is written to disk; the database is untouched and the Quote Editor still sees everything.

⚠Warning

Bar Replay is global and it is quiet about it. A replay left running truncates every scan, exploration and backtest you run afterwards, in every window. If Analysis results suddenly stop early for no reason, check whether Bar Replay is still active before you go looking for a bug in your formula.

The replay route also gives you something the live route cannot, and the project builds on it: the ability to run one formula twice over the same session — once with the future hidden, once with it present — and require the two answers to match.

What changed

Repeat scanning is a checkbox and an interval field. Everything hard about it is arithmetic you can do in advance: symbols times per-symbol cost, divided by the threads your edition allows, compared against the interval you asked for and the symbol allowance your vendor sold you. The staleness is not a defect to be engineered away — it is inherent in examining a universe one symbol at a time while the universe keeps moving — so the correct response is to measure it, display it, and treat the output as a shortlist for a person rather than as a statement about the present.

Check your understanding

Question 1. A scan takes 40 seconds over your universe and you set the auto-repeat interval to 30 seconds. What happens?

• AmiBroker refuses the setting and reverts to 40 seconds
• Passes overlap: concurrent runs compete for the same thread budget while you read a list that is being rebuilt
• The scan silently drops symbols until it fits in 30 seconds
• Nothing changes, because the timer waits for the previous run to finish

Show the answer and why

Answer: Passes overlap: concurrent runs compete for the same thread budget while you read a list that is being rebuilt

Nothing in the documented behaviour makes the timer wait or makes AmiBroker trim the universe. The interval must be chosen from a measurement of the run, with headroom, because run duration varies with backfill and vendor latency.

Question 2. In the Auto repeat interval field, what does typing 5 give you?

• 5 seconds
• 5 minutes
• 5 bars
• 5 passes, then it stops

Show the answer and why

Answer: 5 minutes

A plain number means minutes. Seconds require the explicit form: 5s or 5sec, followed by Enter.

Question 3. Which of these are documented consequences of running a streaming screen over more symbols than your subscription allows? Select all that apply.

• AmiBroker rotates symbols, dropping the oldest and backfilling each new one
• The Real-Time Quote window rotates in the same way
• Repeated rotation can overload the vendor’s servers
• Vendors may act against accounts that abuse streaming limits

Show the answer and why

Answer: AmiBroker rotates symbols, dropping the oldest and backfilling each new one, Repeated rotation can overload the vendor’s servers, Vendors may act against accounts that abuse streaming limits

Rotation, server load and vendor action are all documented. The rotation mechanism explicitly does not apply to the Real-Time Quote window, which simply cannot hold more symbols than your subscription allows.

Question 4. Why is Bar Replay useful for learning repeat scanning without a data subscription?

• It generates synthetic quotes that resemble live ones
• It truncates the database at a playback position for the whole program, Analysis included, so a repeated scan sees a data edge that keeps moving
• It converts an end-of-day database into an intraday one
• It disables the edition thread limits during playback

Show the answer and why

Answer: It truncates the database at a playback position for the whole program, Analysis included, so a repeated scan sees a data edge that keeps moving

Bar Replay is documented as global: every symbol’s data ends at the playback position and this affects all formulas, in charts and in Analysis. Nothing is synthesised and nothing is written to disk.

Question 5. Your probe reports "Bar age (s)" of about 3,600 on every symbol, all day, with a healthy feed. What is the most likely explanation?

• The feed is delayed by exactly one hour
• The database has a time shift configured, so PC-clock time and bar timestamps are deliberately on different clocks
• GetPerformanceCounter was not reset
• The scan interval is one hour

Show the answer and why

Answer: The database has a time shift configured, so PC-clock time and bar timestamps are deliberately on different clocks

Now(5) reads the PC clock and DateTime() reads database bar stamps. A configured database time shift separates the two by a fixed amount, which is why the probe prints Status("timeshift") next to the age.

Check my answersClear