Static Variables for Cross-Sectional Work

There is a moment in every AmiBroker user’s development where they try to write something like OtherScore = Score of "XYZ"; and discover that AFL has no way to express it. This lesson explains why that gap is deliberate, and what AmiBroker provides in its place.

By the end you should be able to write a formula whose first pass computes something for every symbol in a universe, whose second pass reads those values back per symbol, and which leaves no debris behind for the next run to trip over.

Why one execution cannot see another’s results

An AFL formula runs against one symbol. In the Analysis window, AmiBroker runs it once per symbol, and since version 5.50 it runs many of those executions in parallel: the documented model is one thread per operation per symbol, capped at 2 threads per Analysis window on the Standard edition and 32 on Professional, and never more than the number of logical processors Windows reports.

What one Analysis run actually is

one box per symbol, running side by side

1. Analysis windowOne operation: Scan, Exploration, Backtest
2. Thread pool2 threads (Standard) or up to 32 (Professional)edition
3. One AFL execution per symbolIts own arrays, its own variables
4. Result rowsCollected and displayed afterwards

Each of those executions has its own copy of every variable. They begin at unpredictable times, they finish at unpredictable times, and when one finishes its variables are gone. An ordinary AFL variable therefore cannot carry anything from one symbol’s run to another’s; there is nothing wrong with your code when this fails, and no argument you can pass that changes it.

Foreign() and SetForeign() do not close the gap either. They read another symbol’s stored data — its open, high, low, close, volume — not another execution’s computed results. You can of course recompute a peer’s score inside your own execution, and for a handful of peers that is fine. For a universe it is not: scoring 500 symbols by having each of the 500 executions rebuild all 500 scores is 250,000 foreign reads to produce 500 numbers, and the multi-threading chapter notes that every cross-symbol access takes a global lock, so those reads do not even parallelise well.

What is needed is a place outside any single execution where a value can be left for another execution to pick up. That is exactly what a static variable is.

The store AmiBroker provides

A static variable is a named slot in a key-value store that belongs to the running AmiBroker process. The official pages describe it as having static duration — allocated when the program begins and deallocated when the program ends — and say plainly that static variables allow values to be shared between various formulas.

Fragment — not a complete formula

StaticVarSet( "MyScore" + Name(), ROC( Close, 126 ) );
Back = StaticVarGet( "MyScore" + Name() );

The documented signatures are:

• StaticVarSet( "varname", value, persistent = False, compressionMode = cmDefault ) — returns 1 on success and 0 on failure. Scalars and strings have been supported since 4.60, arrays since 5.30, matrices since 6.10.
• StaticVarGet( "varname", align = True ) — returns the stored value, or Null if the name does not exist. (The official page’s own syntax line has misplaced quote characters; the call is as written here.)

Four properties matter more than the rest.

They outlive your formula. A value written by a chart pane is readable by an Exploration, a Scan, a backtest or another chart, in the same AmiBroker session. Nothing clears them when your formula ends.

A missing name reads back as Null, not zero. Wrap reads in Nz() whenever you intend to accumulate, or a single missing symbol will blank an entire running total. StaticVarGetText() behaves differently and returns an empty string for a missing name, so the existence test for text is a comparison with "", not IsNull().

Arrays cost memory that is never reclaimed on its own. The StaticVarSet page states that a static array consumes 8 bytes per bar and that the memory is not released until AmiBroker closes or you call StaticVarRemove(). Five hundred symbols of five thousand daily bars is roughly 20 MB, per score, per run.

Read once, write once. The documentation is explicit that static arrays are a little slower than ordinary AFL variables and recommends pulling a value into a normal variable, doing all the arithmetic there, and writing it back at the end. Reading a static variable inside a per-bar loop is the classic way to make a fast formula slow.

⚠Warning

Static arrays are not a drop-in replacement for composites. The StaticVarSet page states that a static array stores only as many bars as are currently in use by the given chart, whereas AddToComposite() forces the use and storage of all bars. In an Analysis run, the range you set and QuickAFL therefore decide how much history your ranking covers — and a shorter range silently produces a shorter ranking rather than an error.

There is a second alignment question worth knowing before it bites. StaticVarGet()’s align argument defaults to True, which synchronises the stored values to the reading symbol’s own timestamps. The documentation advises against align = False unless you know exactly what you are doing, notes that it is incompatible with compression, and adds that the speed saving is negligible anyway. Leave it alone. Also read values back in the same interval you wrote them in: the pages state that cross-interval reads perform padding and compression automatically but by a different route from Foreign(), so a daily-written variable read on an intraday chart shows essentially flat lines for each day.

The persistent argument

The third argument of StaticVarSet() is persistent, defaulting to False. When it is True, the variable is written to PersistVars.bin when AmiBroker closes and reloaded automatically on the next startup, preserving values between application runs. The feature arrived in 5.80. The prose on the same page calls the argument persist while the syntax line calls it persistent; it is one argument.

Persistent variables can additionally be auto-saved on a timer with SetOption( "StaticVarAutoSave", interval ), where the interval is in seconds; 0 disables it, and from 6.90 an interval of -1 performs a single one-shot save. The documentation warns that writing many static variables to disk takes time and blocks all static variable access while it happens — it calls saving every second a bad idea and 60 seconds fine.

!Important

For ranking work you almost always want persistent left at its default of False. A persisted score set is a stale score set: it will be sitting in memory the next time you open AmiBroker, before any of today’s data has been read, and it will be used silently if your formula does not overwrite it first. Persistence is for things that are genuinely session-independent, such as a user setting, not for the outputs of a calculation over market data.

The two-pass pattern

The pattern that makes cross-sectional work possible has three steps in a fixed order: write every symbol’s score, do the cross-symbol operation once, then read back per symbol. In the Analysis window the first two steps must happen exactly once, before any other symbol’s execution starts, and AmiBroker provides a documented way to arrange that.

Fragment — not a complete formula

if( Status( "stocknum" ) == 0 )
{
    // runs once, on the first symbol, on one thread
}

The multi-threading chapter states that AmiBroker detects this statement, runs the very first symbol in one thread only, waits for completion, and only then launches all the other threads. That is what makes writes inside the block safe without any locking of your own. The chapter also states a caveat with no exceptions: this statement must not be placed inside an #include file.

Goal

Build the smallest formula that demonstrates the pattern: score every symbol in the Analysis window’s own universe, compute one number that depends on all of them, and let every symbol read that number back.

Complete formula

Complete runnable AFL

two-pass-momentum-scores.afl

// two-pass-momentum-scores.afl
// Part 13 - Static Variables for Cross-Sectional Work
//
// The smallest honest example of the two-pass workflow. Pass 1 runs once, on
// the very first symbol only, and writes one static array per symbol plus two
// universe-wide arrays. Pass 2 runs for every symbol and reads them back.
//
// Nothing is ranked here on purpose. The single point being demonstrated is
// that a formula executing on one symbol can see a number that was computed
// from every symbol - which plain AFL cannot do.
//
// How to run it: Analysis window, Apply to = Filter (a watch list) or All,
// Range = 1 recent bar, then Explore.
//
// Assumptions declared up front:
//   - Daily bars, one trading calendar across the universe. Where calendars
//     differ, StaticVarGet's default align = True re-aligns stored values to
//     the reading symbol's timestamps, which is what we want here.
//   - Static variable names live in one namespace shared by every formula in
//     the running AmiBroker. The prefix below is long and specific on purpose.
//   - Everything this formula writes, it removes first. Nothing is persistent:
//     a score written yesterday is a wrong answer today.

InputPrefix = "P13Score";           // one static array per symbol
MeanName    = "P13UniverseMean";    // one array shared by the whole universe
CountName   = "P13UniverseCount";

ScorePeriod = Param( "Momentum lookback (bars)", 126, 20, 500, 1 );

// Read the Analysis window's own universe setting, so the list we compute over
// and the list we report on can never drift apart. GetOption is the native,
// thread-safe way to do this.
ApplySetting = GetOption( "ApplyTo" );

if( ApplySetting == 2 )
{
    WatchListNum = GetOption( "FilterIncludeWatchlist" );
    SymbolList   = CategoryGetSymbols( categoryWatchlist, WatchListNum );
}
else if( ApplySetting == 0 )
{
    SymbolList = CategoryGetSymbols( categoryAll, 0 );
}
else
{
    Error( "Set Apply to = All symbols, or Filter with an Include watch list." );
}

// ------------------------------------------------------------------- PASS 1
// AmiBroker detects Status("stocknum") == 0 and runs the very first symbol on
// one thread alone, waiting for it to finish before launching the others. That
// is what makes these writes safe without any locking of our own.
if( Status( "stocknum" ) == 0 )
{
    // DO NOT forget the asterisk. Without it nothing is removed, and last
    // run's scores quietly join this run's numbers.
    StaticVarRemove( InputPrefix + "*" );
    StaticVarRemove( MeanName );
    StaticVarRemove( CountName );

    ScoreTotal = 0;
    ScoreCount = 0;

    for( i = 0; ( Sym = StrExtract( SymbolList, i ) ) != ""; i++ )
    {
        // A symbol that is not in the database leaves the price arrays alone
        // and returns 0, so skipping it here avoids scoring the wrong data.
        if( SetForeign( Sym ) )
        {
            SymbolScore = ROC( Close, ScorePeriod );
            RestorePriceArrays();

            StaticVarSet( InputPrefix + Sym, SymbolScore );

            // Nz converts the warm-up Nulls to zero so that one short-history
            // symbol cannot blank the running total for every bar.
            HasScore   = NOT IsNull( SymbolScore );
            ScoreTotal = ScoreTotal + Nz( SymbolScore );
            ScoreCount = ScoreCount + HasScore;
        }
    }

    // Max( ScoreCount, 1 ) keeps the division defined on bars where no symbol
    // had enough history yet.
    StaticVarSet( MeanName,  ScoreTotal / Max( ScoreCount, 1 ) );
    StaticVarSet( CountName, ScoreCount );
}

// ------------------------------------------------------------------- PASS 2
// Read once into ordinary AFL variables, as the documentation recommends, and
// do all the arithmetic on those.
OwnScore      = StaticVarGet( InputPrefix + Name() );
UniverseMean  = StaticVarGet( MeanName );
UniverseCount = StaticVarGet( CountName );

// A missing static variable reads back as Null, not as zero. Anything with no
// score was either skipped in pass 1 or is not in the universe at all.
Filter = NOT IsNull( OwnScore );

AddColumn( OwnScore,                "Score %",           1.2 );
AddColumn( UniverseMean,            "Universe mean %",   1.2 );
AddColumn( OwnScore - UniverseMean, "Score - mean",      1.2 );
AddColumn( UniverseCount,           "Symbols scored",    1.0 );

SetSortColumns( -5 );

Download two-pass-momentum-scores.afl105 lines

How it works

The formula reads the universe from the Analysis window itself rather than from a hard-coded list. GetOption( "ApplyTo" ) returns 0 for all symbols, 1 for the current symbol and 2 for the filter setting; when it is 2, GetOption( "FilterIncludeWatchlist" ) gives the watch list number, and CategoryGetSymbols() turns that into a comma-separated string. Reading the settings natively rather than through OLE is what the multi-threading chapter insists on, and it also means the list you compute over cannot drift away from the list you report on.

Pass one sits inside the Status( "stocknum" ) == 0 guard. It removes everything it is about to write, loops the symbol list, and for each symbol calls SetForeign(), computes the score, restores the price arrays and stores the score under a name built from the prefix and the ticker. It also accumulates a running total and a running count so that, after the loop, it can store the universe mean — a number no single symbol could have computed.

Pass two runs for every symbol, including the first. It reads its own score and the two universe-wide arrays into ordinary variables, and reports them. The Filter line uses the fact that a missing name reads back as Null to exclude anything that was never scored.

Key functions

• Status( "stocknum" ) — the ordinal number of the symbol currently being analysed; == 0 is the documented “do this once per Analysis run” guard.
• GetOption( "ApplyTo" ) and GetOption( "FilterIncludeWatchlist" ) — the native, thread-friendly way to read the Analysis window’s own settings.
• CategoryGetSymbols( category, index, mode = 0 ) — returns a comma-separated symbol list; categoryAll (5.50 and later) means every symbol in the database.
• StrExtract( list, item, separator = ',' ) — pulls item n out of that list, and returns an empty string when it runs off the end, which is what terminates the loop.
• Nz( x, valueifnull = 0 ) — converts Null, NaN and infinities to zero.

Expected result

One row per symbol, with the symbol’s own score, the universe mean of all the scores, the difference between them, and how many symbols contributed. The mean column holds the same value on every row — that is the proof that the cross-symbol step worked. The count column tells you how many symbols actually had enough history on that bar.

Test it

Run the Exploration on a watch list of five or six symbols with the range set to one recent bar. Copy the individual score column into a spreadsheet, average it by hand, and compare with the mean column. They should agree to the displayed precision. Then delete one symbol from the watch list and re-run: the count falls by one and the mean moves. If the mean does not move, pass one did not re-run, and the next section explains why.

Common errors

• Every row shows the same score as well as the same mean. RestorePriceArrays() was missed, so everything after the first SetForeign() computed on foreign data.
• The mean is unchanged after you edit the watch list. Something removed the guard, or the removal call lost its wildcard, so yesterday’s variables are still in memory and still being read.
• An error about Apply To. The formula deliberately refuses to run with Apply To set to “current symbol”, because then the list and the run would describe different things.
• Scores are Null for symbols you know have data. Check the lookback against the Analysis range: a 126-bar rate of change needs 126 prior bars inside the range.

Extension

Add a second universe-wide statistic — the standard deviation of the scores across symbols — and report each symbol’s score as a z-score. You will need a second pass over the stored scores after the mean is known, which is a useful rehearsal for the ranking loop in the next lesson.

Naming schemes and collisions

Static variable names live in one flat namespace shared by every formula, chart pane and Analysis window in the running AmiBroker. There is no per-formula scope, no per-database scope documented in the User’s Guide, and no warning when two formulas pick the same name. Two unrelated formulas that both use "values" will overwrite each other’s data, and the symptom is not an error — it is a plausible-looking wrong answer.

Three rules keep this manageable.

Use a long, specific prefix. "P13Score" is defensible; "score" is not. This matters doubly for ranking, because the ranking function matches input variables by “starts with” and treats whatever follows as a ticker.

Never let one prefix be the prefix of another. If you have "val" and "values", anything scanning for "val" will swallow the "values" family too and read nonsense ticker names out of the remainder.

Prefer names AmiBroker builds for you where it can. The static keyword, enabled by #pragma enable_static_decl (prefix), lets you declare identifiers that are backed by static variables and automatically prefixed to avoid clashes. The prefix may contain the runtime tokens {chartid}, {interval} and {symbol}, which are substituted at run time and give you per-chart, per-interval or per-symbol namespaces for free. The documentation recommends prefixing declared statics with an underscore so they stand out, and notes an important limit: a declared static is read once at the declaration and written once when the formula finishes, so it is not a live channel between threads during a run. For the two-pass pattern, the function-based API is the right tool.

✦Tip

AmiBroker’s own examples in the ranking documentation write a prefix as "rank" and then read the result back as "RankValuesToSort" + sym, so the shipped examples rely on static variable names being matched without regard to case. The guide never states that rule in words. Write your names consistently anyway, and do not build anything on the difference.

Cleaning up between runs

Nothing removes a static variable for you. StaticVarRemove( "variablename" ) is the only documented way to free the memory short of closing AmiBroker, and since 5.30 it accepts wildcards: * matches any number of characters including zero, and ? matches exactly one.

Fragment — not a complete formula

// delete static variables - DO NOT forget the asterisk (wildcard) at the end
StaticVarRemove( "P13Score*" );

That comment is AmiBroker’s own, from its ranking example, and it is shouted for a reason. StaticVarRemove( "P13Score" ) removes exactly one variable named P13Score, which probably does not exist, and leaves every P13ScoreAAPL, P13ScoreMSFT and the rest of last run’s data sitting in memory to be silently included in this run’s answer.

⚠Warning

The opposite mistake is worse. Because the pattern is matched across the whole namespace, an over-broad wildcard destroys state belonging to other formulas — other chart panes, other Analysis windows, any automated code running in the same AmiBroker. Never write StaticVarRemove( "*" ) in a formula you keep. Remove only your own prefix, and make that prefix specific enough that it can only be yours.

Put the removal inside the Status( "stocknum" ) == 0 block so it happens once, before the writes, rather than once per symbol — which would delete the work of the symbols that have already run.

A diagnostic worth keeping

Goal

Have a chart pane that answers, at a glance, what is still resident in the static namespace and how much memory it is using.

Complete formula

Complete runnable AFL

static-var-inspector.afl

// static-var-inspector.afl
// Part 13 - Static Variables for Cross-Sectional Work
//
// A diagnostic to keep in a chart pane while you develop cross-sectional
// formulas. It answers the two questions that explain most "impossible"
// ranking results: what is still resident in memory, and how much of it.
//
// Assumptions declared up front:
//   - StaticVarInfo( pattern, "count" ) needs AmiBroker 6.90 or newer. The
//     other three fields work from 5.60. The running version is printed so a
//     missing count is not mistaken for a count of zero.
//   - This formula only READS. It never removes anything, because a wildcard
//     removal typed into the wrong pane destroys state belonging to other
//     formulas, chart panes and Analysis windows in the same AmiBroker.

_SECTION_BEGIN( "Static variable inspector" );

NamePattern = ParamStr( "Name pattern (wildcards allowed)", "P13*" );

TotalCount   = StaticVarCount();
MatchingList = StaticVarInfo( NamePattern, "list" );
MatchedBytes = StaticVarInfo( NamePattern, "totalmemory" );
AllBytes     = StaticVarInfo( "*", "totalmemory" );

Plot( Close, "Close", colorDefault, styleCandle );

Title =
    "Static variables resident in this AmiBroker process" + "\n" +
    "AmiBroker version: " + NumToStr( Version(), 1.2 ) + "\n" +
    "Total static variables: " + NumToStr( TotalCount, 1.0 ) + "\n" +
    "Memory, all variables: " + NumToStr( AllBytes / 1024, 1.0 ) + " kB" + "\n" +
    "Memory, names matching the pattern: " +
        NumToStr( MatchedBytes / 1024, 1.0 ) + " kB" + "\n" +
    "Names matching the pattern: " + MatchingList;

_SECTION_END();

Download static-var-inspector.afl36 lines

How it works

It reads and reports, and does nothing else. StaticVarCount() gives the global total with no filtering argument. StaticVarInfo() takes a name or a wildcard pattern plus a field: "list" returns the matching names, "memory" the bytes excluding the names themselves, "totalmemory" the bytes including them, and "count" the number of matches. The version is printed alongside because "count" requires AmiBroker 6.90 or newer, and a blank is easy to misread as a zero.

Key functions

• StaticVarCount() — total number of static variables in memory.
• StaticVarInfo( "varname", "field" ) — introspection over the namespace; only the four documented field strings exist.
• Version( minrequired = 0 ) — returns the AmiBroker version as a number.

Expected result

A title block listing the total count, the memory used by everything, the memory used by the names matching your pattern, and those names. Run the two-pass formula, then look here: you should see one variable per symbol plus the two universe-wide ones.

Test it

Note the count. Run the two-pass Exploration on a watch list of ten symbols. The count should rise by twelve. Change the pattern to your prefix and confirm the listed names are exactly the ones you expect, with no leftovers from an earlier prefix.

Common errors

• The count never falls. Nothing removes static variables automatically. If a formula you have stopped using left ten thousand arrays behind, they are still there.
• A blank version or count. "count" needs 6.90 or newer.
• The list is enormous. That is the finding, not a bug. It usually means a formula is writing per-symbol variables without a matching removal.

Extension

Add a parameter for a second pattern and report both counts side by side, so you can watch an input family and its generated output family at the same time — which is exactly what the next lesson produces.

Multi-threading: what you must do, and what you need not

The multi-threading chapter is precise about this, and it is less onerous than people expect.

A single StaticVarSet or StaticVarGet call is atomic. It reads or writes an entire array in one indivisible operation; no other thread will see a half-updated array.

Groups of calls are not atomic. If the correctness of your logic depends on several static variables being updated together, nothing protects that region for you.

The rule that removes the problem is one writer, many readers. Do all writing inside Status( "stocknum" ) == 0, and let every other thread only read. No synchronisation is then needed at all — the guide says so in those terms.

Two functions exist for the cases that rule does not cover. StaticVarAdd( "name", value, keepAll = True, persistent = False ) performs an atomic read-add-write and is the documented multi-threading-safe replacement for AddToComposite(); note that it converts Null values to zero, and that unlike AddToComposite() it does not skip group 253, so composite symbols must be excluded by hand. StaticVarCompareExchange( "varname", exchange, comperand ) is an atomic compare-and-swap on a scalar numeric static variable, and it is the primitive the documentation uses to build a semaphore.

⚠Warning

A lock built on StaticVarCompareExchange() must be released by your own code. Static variables outlive the formula, so a formula that errors out between acquiring and releasing leaves the lock permanently held — and because it survives the run, nothing will ever acquire it again until you remove the variable by hand. Prefer the one-writer/many-readers pattern; reach for a lock only when you can prove you need one.

Recap

AFL executions are isolated by design, which is what makes scanning fast and cross-sectional work impossible without help. Static variables are the help: a process-wide store that outlives every execution, costs 8 bytes per bar per array, and is cleaned up only when you clean it up. The workflow is always the same shape — write once under Status( "stocknum" ) == 0, do the cross-symbol operation once, read back per symbol — and the three things that go wrong are a missing wildcard, a prefix that is too short, and persistence left switched on. The next lesson replaces the hand-rolled universe statistic with AmiBroker’s own ranking function.

Check your understanding

Question 1. A formula reads StaticVarGet("MyScoreABC") for a symbol that was never written. What does it get?

• Zero
• An empty string
• Null
• A run-time error

Show the answer and why

Answer: Null

StaticVarGet returns Null for a name that does not exist, which is why accumulation code wraps it in Nz(). StaticVarGetText is the one that returns an empty string instead.

Question 2. Which of these are true of StaticVarRemove()? Select all that apply.

• Without a wildcard it removes only the exact name given
• The * wildcard matches any number of characters including zero
• It reports whether anything was actually removed
• An over-broad pattern can delete static variables belonging to other formulas

Show the answer and why

Answer: Without a wildcard it removes only the exact name given, The * wildcard matches any number of characters including zero, An over-broad pattern can delete static variables belonging to other formulas

It returns NOTHING, so there is no success feedback - use StaticVarCount() or StaticVarInfo(pattern, "count") if you need proof. The namespace is application-wide, which is what makes a broad pattern dangerous.

Question 3. Why does the two-pass formula put its writes inside if( Status("stocknum") == 0 )?

• To make the formula run faster on the first symbol
• Because AmiBroker runs that first symbol on one thread and waits for it before launching the others
• Because static variables cannot be written from any other symbol
• To avoid using Foreign()

Show the answer and why

Answer: Because AmiBroker runs that first symbol on one thread and waits for it before launching the others

The multi-threading chapter states that AmiBroker detects the statement and serialises the first symbol, which is what makes one-writer/many-readers safe. Static variables can be written from anywhere; the point is that only one execution should.

Question 4. You set the persistent argument of StaticVarSet to True for your ranking scores. What is the most likely consequence?

• The ranking becomes faster on the next run
• The scores are saved to disk on exit and reloaded next session, so a stale ranking can be read before today’s scores are written
• The variables are removed automatically when AmiBroker closes
• Nothing changes unless SetOption is also called

Show the answer and why

Answer: The scores are saved to disk on exit and reloaded next session, so a stale ranking can be read before today’s scores are written

Persistent variables go to PersistVars.bin on close and come back on startup. For a calculation over market data that is a stale result waiting to be read, which is why persistence is left off for ranking work.

Question 5. A static array over 5,000 daily bars is stored for each of 500 symbols. Roughly how much memory does that hold, and when is it released?

• About 20 MB, released when the Analysis run finishes
• About 20 MB, released only by StaticVarRemove() or by closing AmiBroker
• About 2 MB, released automatically after ten minutes
• It depends on the edition

Show the answer and why

Answer: About 20 MB, released only by StaticVarRemove() or by closing AmiBroker

The documented cost is 8 bytes per bar per array: 8 x 5,000 x 500 is about 20 MB. The StaticVarSet page states the memory is not released until the program closes or the variable is removed.

Check my answersClear