Plugins and the Database Relationship

Choosing a data source in AmiBroker takes one click in one dialog, and it changes what your database is. Before the click, AmiBroker owns the quote files and you fill them. After it, a plugin owns them, and your database becomes a cache with settings — settings that decide how much history you have, how it arrives, and what happens to it when the connection drops.

By the end of this lesson you should be able to describe what a plugin-fed database does that a local one does not, explain what backfill actually is and when it happens, read the plugin status light, predict what a disconnection does to your charts and your Analysis runs, and say why the base interval is the one setting worth thinking hardest about before you create anything.

✱AmiBroker Professional required

This lesson describes a database fed by a real-time plugin, which needs a streaming subscription, and several of the behaviours it covers — “wait for backfill”, tick and second base intervals — are documented as Professional-edition features. The plugin connection itself is not: the edition comparison gives both editions automatically updating real-time charts and intraday data from one minute upwards, with Standard capped at ten streaming quote-window symbols and no GetRTData().

Every section below has an observable counterpart on the end-of-day database you already have, and the Level A path near the end of the lesson walks through them.

Selecting a plugin, and confirming what you selected

A data plugin is a DLL in the Plugins subfolder of the AmiBroker program directory. AmiBroker scans that folder at startup, so a plugin that is not there does not exist as far as the program is concerned, and one that is there is locked for writing while loaded — which is why the Plugins window has an Unload button.

The binding happens in File -> Database Settings, in the Data source combo box. The General half of that dialog is enabled only while you are creating a database; the Data source half stays editable for the life of it.

⌀Data quality warning

The list of data sources printed in the User’s Guide’s Database Settings page is out of date. It names eSignal, myTrack, QuoteTracker, Quotes Plus, TC2000/TCNet, FastTrack and Metastock, and omits IQFeed, Interactive Brokers, DDE, ODBC, Norgate and TC2000 Mutual Funds — all of which are real, documented plugins with their own pages elsewhere in the same guide. The authoritative list is whatever your installed copy actually offers in the combo box. Where this course names a plugin, it is naming one with a current documentation page, and Part 18 dates each appendix for exactly this reason.

Two checks are worth making a habit. Tools -> Plugins lists every loaded DLL with its version number, which is the only reliable way to confirm that a manually replaced plugin took effect. And bitness must match: a 32-bit AmiBroker cannot load a 64-bit DLL or the reverse, so a plugin that refuses to appear in the data-source list is a bitness mismatch until you have ruled it out. The Plugins page also carries an official warning that AmiBroker makes no representations about non-certified third-party plug-ins, which can cause instability or crashes, and that using them is at your own risk.

Local data versus streaming: what the database still owns

Selecting a plugin does not make the AmiBroker database go away. The guide says so twice, because learners consistently expect it to. What changes is who supplies the quotes.

• Quotes come from the plugin, read-only. The documentation states that plugins provide read-only access to the vendor’s data and that AmiBroker never writes back.
• Everything else stays yours, in AmiBroker’s own files. Symbols, category, group and watch list assignments, composites, favourites and studies live in the database directory regardless of where the prices came from.
• Local data storage decides whether quotes are cached. It is a separate setting in the same dialog, documented as required for most real-time sources, and it has no effect at all when the data source is (local).
• Number of bars to load caps what is kept. Also plugin-only, also inert on a local database. One bar costs 40 bytes per symbol, so 100,000 bars is roughly 4 MB per symbol actually allocated, and the performance chapter advises against going beyond that figure. The dialog shows you the equivalent number of days beside the field; read it before pressing OK.

✕Common mistake

Increasing “Number of bars to load” on an existing database and expecting the extra history to appear. It does not: symbols that were already backfilled keep what they have. The documented remedy is to Force backfill those symbols afterwards. The symptom is a database where the symbols you added yesterday have twice the history of the ones you added last month, for no visible reason.

Backfill happens on demand, one symbol at a time

This is the behaviour that surprises people most, so state it plainly: nothing is downloaded until something asks for it. AmiBroker requests a symbol’s history from the plugin on first access — you open its chart, or an Analysis run reaches it. A symbol nobody has touched may hold no bars at all, and the symbol tree gives no hint of that.

What happens the first time you open a symbol on a plugin-fed database

1. 1You select the symbolA chart opens, or an Analysis run reaches this symbol in its list
2. 2AmiBroker asks the pluginFor as many bars as "Number of bars to load" permits, at the database base interval
3. 3The plugin asks the vendorThrough the vendor’s local client program, subject to that vendor’s request limits and history depth
4. 4Bars arrive, possibly slowlyMeanwhile the chart draws whatever has arrived so far
5. 5Streaming updates continueNew trades are time-compressed into bars at the base interval

The fourth step is where analysis goes wrong: a scan that does not wait can run on a half-filled array.

That fourth step has a documented remedy and it is edition-gated. “Wait for backfill”, in the Analysis window’s Settings menu, makes scans and explorations wait for each symbol’s backfill to complete before running on it. It is a Professional feature, it is documented for the eSignal, myTrack and IQFeed plugins, and it has no effect at all on local databases, on end-of-day plugins, or with QuoteTracker, which manages its own backfills and exposes no control. Many third-party plugins simply do not implement it.

The documented way to backfill an entire database uses it: write Buy = 1;, send the formula to Analysis, set Apply to All symbols and Range to one recent bar, tick Wait for backfill, and press Scan. The scan is a pretext; touching every symbol is the point.

⚠Warning

Backfill is a request to somebody else’s servers, and vendors count requests. One documented broker feed permits no more than sixty requests in five minutes and runs one backfill at a time; the Knowledge Base warns that repeatedly exceeding a subscription’s symbol limit forces a rotation cycle — drop the oldest symbol, add the new one, trigger a backfill, resume streaming — and that vendors may act against accounts that abuse streaming limits. “Backfill everything overnight” is not a neutral operation.

Two more documented facts about a plugin-fed database that catch people out. Do not ASCII-import into one for symbols it already holds: the plugin will eventually overwrite the imports, and importing end-of-day data over an intraday database is documented as corrupting it. The narrow exceptions are importing intraday data at exactly the same bar interval, or importing symbols the database does not have at all — in which case the importer marks them “use only local database for this symbol”, which is the supported way to bring in an auxiliary series for Foreign(). And the Quote Editor is restricted: on a plugin database you can edit only one-minute-or-higher intervals, only symbols that are fully backfilled, and never the last three bars, because those are held in the plugin’s cache.

The status light

The plugin’s own view of its health lives in a small status area at the lower right of the main window. AmiBroker beeps and shows a bubble tooltip when it changes; the tooltip hides itself after about two seconds, so hover over the area to bring it back.

Light	Meaning	What it implies
OK (green)	Connection is fine and the plugin is operating correctly	Data should be arriving. It does not mean data is arriving for a given symbol
WAIT (yellow)	Connecting, or connected to only some of the vendor’s servers	Usually transient. Persistent WAIT on a DDE link means wrong server or field names, or the DDE server application is not running
ERR (red)	Connection broken — documented causes are invalid credentials or a required third-party program not running	Needs you. The plugin retries once the cause is fixed
SHUT (purple)	A serious failure	The plugin does not reconnect by itself. Reconnect from the status area’s context menu, or restart AmiBroker

Right-clicking that status area gives three commands: Reconnect, Shutdown (Disconnect) and Force backfill. Force backfill re-requests a symbol’s whole intraday history, and the guide documents exactly two situations where it is needed: after enlarging Number of bars to load, and to clear bad ticks in the hope that the vendor has repaired its history — which is documented as working well for one source, because that vendor does fix them.

!Important

There is no AFL function that reads the plugin connection status. Nothing in the function reference exposes OK, WAIT, ERR or SHUT, and nothing forces a reconnect or a backfill from a formula either. The status light is a user-interface display, full stop. This is why a formula cannot ask “am I connected?” and has to infer it from evidence — the age of the last update, the age of the newest bar — which is what the panel in the first lesson does and what the exploration below does across a whole database.

What a disconnection actually does

Nothing dramatic, which is the problem. Bars stop arriving. The chart keeps displaying the bars it has, drawn exactly as before, with the same indicators computed on the same data. Nothing turns red. The last bar simply stops changing, which is also what a symbol that nobody is trading looks like.

Three consequences follow that are worth internalising before Part 21 makes you diagnose one under time pressure.

Analysis silently narrows. A scan running against symbols whose backfill is incomplete analyses short arrays. Without “Wait for backfill” it does not pause, and it does not warn; it reports on what it has. A result set that looks thin after a reconnection is more likely to be a data problem than a market problem.

Recovery is per-plugin and per-state. ERR is retried; SHUT is not. Some plugins backfill the gap on reconnection, some backfill only what their history depth allows, and one documented source supplies no backfill at all, which means a disconnection leaves a permanent hole that only an ASCII import can fill.

Your symbols may not all be equal. If you stream more symbols than your subscription covers, AmiBroker rotates the active set, so a symbol can quietly stop updating while everything else carries on. The rotation does not apply to the Real-Time Quote window, which cannot hold more symbols than the subscription permits.

Reading the state of a whole database

Goal

The panel in the first lesson answers “what is happening to this symbol, right now”. This one answers the other question: across every symbol in the database, which are still receiving data and which are not? That is the question a disconnection, a rotation, a lapsed entitlement and a delisting all produce a “no” to, and none of them announces itself.

The formula

Complete runnable AFL

update-freshness-report.afl

// ===========================================================================
//  Update freshness report
//
//  WHAT IT IS FOR
//    One row per symbol answering a question that has nothing to do with
//    whether the prices are right: is this symbol still receiving data, and
//    is it receiving it at the spacing the interval implies?
//
//    A streaming plugin backfills and updates each symbol separately, on
//    demand. A symbol nobody has looked at may hold nothing at all; a symbol
//    that fell out of a subscription's active rotation may have stopped
//    updating hours ago while every other symbol carried on. Neither shows up
//    on a chart you are not looking at.
//
//  HOW TO RUN IT
//    Analysis window: Apply to = All symbols (or a watch list),
//    Range = All quotations, then Explore.
//    It works on any database. On an end-of-day database every age should be
//    roughly the same and roughly the age of the last completed session; the
//    interesting rows are the ones that are not.
//
//  WHAT EACH COLUMN MEANS
//    Newest bar        the timestamp of the last bar held for the symbol
//    Age (hours)       PC clock minus that timestamp
//    Bars in range     how many bars the symbol holds inside the range tested
//    Interval (s)      the interval this Analysis run is using, in seconds
//    Avg spacing (s)   the mean distance between consecutive bars held
//    Timeshift (h)     the database timeshift, printed once per row
//
//  ASSUMPTIONS, STATED SO THEY CAN BE CHECKED
//    - Age is measured against Now(5), the PC clock, while bar timestamps come
//      from the database. If the database timeshift is not zero, every age in
//      the report carries that offset. The timeshift column is there so the
//      offset is visible rather than silently absorbed. Part 20 deals with
//      time zones properly; this report only refuses to hide the problem.
//    - Average spacing counts closed markets as if they were data. On a daily
//      series it lands near 86400 * 1.4 because of weekends, and on an
//      intraday series it is inflated by every overnight gap. Compare symbols
//      with each other on the same interval, not against the nominal interval.
//    - A large age is not evidence of a fault. Delisted symbols, holidays,
//      instruments that simply did not trade and symbols you have never opened
//      all produce one legitimately. This report finds candidates to look at.
// ===========================================================================

Filter = Status( "lastbarinrange" );
SetOption( "NoDefaultColumns", True );

NowStamp    = Now( 5 );
BarsInRange = Cum( 1 );

// DateTime values are a bitset, not an ordinary number: the documented way to
// order or subtract two of them is DateTimeDiff, which returns seconds.
AgeSeconds = DateTimeDiff( NowStamp, DateTime() );

// Mean spacing between the bars actually held, which is the closest thing to
// "is this symbol keeping up" that a database can answer on its own.
FirstBarStamp  = ValueWhen( BarsInRange == 1, DateTime() );
SpanSeconds    = DateTimeDiff( DateTime(), FirstBarStamp );
AverageSpacing = IIf( BarsInRange > 1, SpanSeconds / ( BarsInRange - 1 ), 0 );

AddTextColumn( Name(), "Symbol", 1.0, colorDefault, colorDefault, 80 );
AddColumn( DateTime(), "Newest bar", formatDateTime );
AddColumn( AgeSeconds / 3600, "Age (hours)", 1.2 );
AddColumn( BarsInRange, "Bars in range", 1.0 );
AddColumn( Interval(), "Interval (s)", 1.0 );
AddColumn( AverageSpacing, "Avg spacing (s)", 1.0 );
AddColumn( Status( "timeshift" ) / 3600, "Timeshift (h)", 1.2 );

// Oldest first: descending on column 3, the age.
SetSortColumns( -3 );

Download update-freshness-report.afl70 lines

How it works

Filter = Status( "lastbarinrange" ) reduces the exploration to exactly one row per symbol, evaluated at the last bar in the range — the standard shape for a per-symbol report.

The age column is the whole point: DateTimeDiff( Now( 5 ), DateTime() ) measures the distance in seconds between your PC clock and each symbol’s newest bar. DateTimeDiff is used rather than subtraction because DateTime values are documented as a bitset, for which only equality comparisons are reliable — ordinary > and < on them can give wrong answers, which is one of the most common silent bugs in intraday AFL.

The spacing column asks a different question. ValueWhen( BarsInRange == 1, DateTime() ) captures the first bar’s timestamp, and the elapsed span divided by the number of intervals gives the mean distance between the bars actually held. A symbol receiving every bar and a symbol receiving one bar in three both look plausible on a chart; they do not look the same in this column.

The timeshift column is there to stop the report lying to you. Bar timestamps come from the database and may be shifted towards exchange time, while Now( 5 ) is your PC clock. If the timeshift is not zero, every age in the report carries that offset — so it is printed rather than absorbed. Part 20 handles time zones properly.

Functions you may not have met

• Status( "lastbarinrange" ) — true on the last bar of the tested range, which is how an exploration produces one row per symbol instead of one per bar.
• Status( "timeshift" ) — the database timeshift in seconds.
• DateTimeDiff( a, b ) — the difference between two DateTime values in seconds.
• ValueWhen( condition, array ) — the value the array had when the condition was last true.
• SetSortColumns( -3 ) — sorts the result descending on the third column, so the stalest symbols arrive at the top where you will see them.

What you should see

=Expected result

On a healthy end-of-day database, a tight cluster: nearly every symbol with the same newest-bar date and an age equal to the time since the last session closed. The interesting rows are the outliers — symbols that stopped weeks ago, which are usually delistings, symbol changes, or instruments your update routine quietly stopped covering.

On a plugin-fed intraday database during the session, most ages should be small and similar. Any symbol with an age much larger than its neighbours is a candidate for rotation, an entitlement gap, or an instrument that simply is not trading — and those three are distinguished by checking a second symbol on the same venue and by looking at the status light, not by staring at this table.

Test it

1. Run it over your whole database with Range = All quotations. Note the modal age.
2. Pick the oldest symbol and open its chart. Confirm the report matched what you see.
3. Start Bar Replay, set the position back a month, and re-run the exploration. Every age should jump by roughly a month, because Bar Replay affects Analysis as well as charts. This is a useful demonstration and a warning: an active Bar Replay silently truncates every backtest and exploration you run afterwards.
4. Press Stop and re-run. The ages should return.

Common errors

• Every age is out by a constant number of hours. Non-zero database timeshift. The timeshift column tells you by how much.
• Average spacing looks far too large on daily data. It is: weekends and holidays are counted as elapsed time. Compare symbols with each other, not against 86,400.
• The report shows symbols with no bars at all. On a plugin database that is expected for symbols nobody has opened, because backfill is on demand. Touch them with a scan and re-run.
• The results stop early and nothing explains why. Bar Replay is still running.

Extension

Add a column that flags staleness relative to the database rather than the clock: compute the median age across symbols in a first pass, store it with StaticVarSet(), and in a second pass flag any symbol more than a chosen multiple above it. Part 23 covers static variables as the mechanism for carrying state between runs, and Part 24 uses the same pattern for scanning.

Why the base interval has to match your intent

Everything above assumes the database can hold what you want. That is decided once, by the base time interval, and it is the closest thing AmiBroker has to an irreversible decision.

The base interval is the smallest grain the database stores. Every coarser interval is produced from it by time compression on the fly, and nothing finer is ever available. The guide states both halves: you cannot use intraday charting or analysis at all until the base interval is set below end-of-day, and once it is set to, say, five minutes, every periodicity from five minutes up is available and nothing below it ever is.

For a real-time source it should be chosen at database creation. The reason is physical rather than a locked control: bars already compressed and stored at the old grain cannot be un-compressed. Changing the setting later does not retroactively give you finer history — you would need a new database at the finer grain, collecting or importing from scratch.

✦Tip

The documented ways to have both long daily history and intraday detail are: enable Allow mixed EOD/Intraday data in Intraday Settings, where the plugin supports it — the guide names IQFeed and eSignal 1.7.0 or higher, in a note dated June 2008 — or run two databases, one end-of-day and one intraday. The second option is boring and always works, and it is what most people end up doing. Part 19 makes this decision properly, with the storage arithmetic attached.

Doing this lesson at Level A

▷Exercise

Level A path. You cannot disconnect a feed you do not have, but every idea above has an observable counterpart on an end-of-day database.

1. Open File -> Database Settings on your existing database and read it without changing anything. Note that Data source says (local), and that Local data storage and Number of bars to load are inert as a result. That is the contrast the lesson is about.
2. Open Tools -> Plugins and write down what is loaded, with versions. This is the check you would make after installing a data plugin, and doing it now means you know what normal looks like.
3. Run update-freshness-report.afl over your whole database. Every symbol whose age is an outlier is a symbol your update routine is not covering — the same fault a rotation or an entitlement gap produces on a live feed, arriving by a different route.
4. Open the oldest outlier’s chart, then run Tools -> Bar Replay and step forward a few bars while the exploration is still open. Re-run it. Watching ages change with the replay position is the closest offline equivalent of watching a feed stop and start.
5. Write down which base interval your database uses and what that permits. If it is end-of-day, note that no chart in it can ever show an intraday bar, whatever you subscribe to later.

Recap

Choosing a data source rebinds your database: the plugin supplies quotes read-only, while symbols, categories, watch lists and studies stay in AmiBroker’s own files. Local data storage and Number of bars to load exist only for plugin-fed databases and decide how much of the vendor’s history you actually keep.

Backfill is on demand and per symbol, which means an untouched symbol may hold nothing, and an Analysis run that does not wait may report on half-filled arrays. “Wait for backfill” fixes that where it is supported, and it is a Professional feature. The status light is the only place the connection state appears — no AFL function exposes it — and SHUT, unlike ERR, does not recover on its own.

A disconnection produces no visible alarm, only a bar that stops changing, so the reliable diagnostics measure age rather than price. And the base interval, chosen once at creation, sets a floor you can compress upwards from but never dig below.

That closes the introduction. Part 18 turns the vendor question into a decision framework with dated, provider-specific appendices; Part 19 builds the intraday database this lesson has been describing.

Check your understanding

Question 1. On a plugin-fed database, you add fifty new symbols to the symbol tree and immediately run an exploration over all of them. What is the most likely outcome?

• All fifty are analysed with their full history
• The exploration refuses to run until backfill completes
• Some or all of the fifty are analysed with little or no data, because backfill starts on first access and the run does not wait for it
• AmiBroker downloads all fifty in advance automatically

Show the answer and why

Answer: Some or all of the fifty are analysed with little or no data, because backfill starts on first access and the run does not wait for it

Backfill is requested on first access and runs asynchronously. Without "Wait for backfill" — a Professional feature, supported only by some plugins — the Analysis run proceeds on whatever has arrived. The results look like real results, which is what makes this failure expensive.

Question 2. The plugin status light turns purple and shows SHUT. What does the documentation say happens next?

• The plugin retries automatically until it reconnects
• The plugin does not reconnect by itself; you reconnect from the status context menu or restart AmiBroker
• AmiBroker switches to local data automatically
• The database is marked read-only until repaired

Show the answer and why

Answer: The plugin does not reconnect by itself; you reconnect from the status context menu or restart AmiBroker

SHUT indicates a serious failure and is documented as the state that does not auto-retry, unlike ERR. This is worth knowing because the chart looks identical in both states — the last bar simply stops changing — so the light is the only thing that distinguishes a condition that will clear itself from one that will not.

Question 3. Which of these settings have no effect at all on a database whose data source is (local)? Select all that apply.

• Local data storage
• Number of bars to load
• Base time interval
• Intraday Settings filtering

Show the answer and why

Answer: Local data storage, Number of bars to load

Local data storage and Number of bars to load are documented as applying to external data sources only, and tuning them on a local database is a classic wasted afternoon. The base interval and Intraday Settings apply to every database, because they govern what is stored and what is displayed rather than what is fetched.

Question 4. A formula needs to know whether the real-time feed is connected. What does the AFL function reference offer?

• Status("connected"), returning the plugin state
• GetRTData("Status")
• Nothing — no documented function exposes the plugin connection state
• PlugInStatus()

Show the answer and why

Answer: Nothing — no documented function exposes the plugin connection state

The connection light is a user-interface display and has no AFL equivalent; nor is there any function to force a reconnect or a backfill. A formula therefore has to infer the state from evidence it can reach — the time of the last plugin update, the timestamp of the newest bar — which is exactly why both example formulas in this part measure ages rather than asking a question that cannot be asked.

Check my answersClear