Choosing a Data Source: A Decision Framework

“Which data feed should I get?” is a question with no answer, in the same way that “which vehicle should I buy?” has no answer. It becomes answerable the moment you say what you are going to carry, how far, how often, and what you are willing to spend on fuel.

This lesson turns the vague question into five specific ones, gives you a tool that measures your current answer to two of them, and then hands you a table that maps answers to options. By the end you should be able to write your data requirement on a single index card — and, more usefully, to notice when a requirement you thought you had was actually a habit you picked up from somebody else’s setup.

Start from the question, not the vendor

Every data requirement descends from a research question. Change the question and the requirement changes with it, sometimes by two orders of magnitude in cost.

The question you are trying to answer	What that actually needs
Does a 200-day trend filter improve a monthly rebalanced portfolio?	Daily bars, twenty years, a wide universe including delisted symbols
Does an opening-range breakout survive costs on liquid US shares?	Minute bars, a few years, a modest universe, accurate session boundaries
How wide is the spread on this instrument at 15:55?	Quote data, not trade data. Bars will not answer it at all
Which of my watch list crossed above its 50-day average today?	Daily bars, updated once per evening
Where is the tape thinning out right now?	A live stream, tick or second resolution, and a screen you are watching

Two things are worth noticing in that table. The first is that most of the research in this course lives in the top and fourth rows, where daily bars are sufficient. The second is that the third row cannot be answered by any amount of historical bar data, at any price — bid and ask are a different kind of record, and no depth of history substitutes for them.

∴Research note

Write your question down before you shop. If you cannot state it in a sentence that could be answered “yes” or “no” by evidence, no data source will rescue it. A requirement derived from a vague question tends to be “everything, at the finest resolution, forever”, which is exactly the most expensive possible answer.

The five requirements

1. Instruments and exchanges

The single most common expensive mistake is buying a feed that carries the asset class but not the venue. “US equities” is not one entitlement; futures, equity options and international futures are separately priced on the feeds that carry them, and AmiBroker’s own IQFeed page is explicit that exchange fees are charged on top of any base subscription.

List the exact instruments, not the categories. If you want to study the FTSE 100 constituents, write down that you need London-listed ordinary shares plus an index series, and check that both are covered. If you want to study index futures, write down the specific contracts and the exchanges they trade on.

Two secondary questions belong here and are easy to forget:

• Do you need delisted symbols? A universe of currently listed shares is a survivorship-biased universe, and every study run over it inherits that bias. Very few free sources carry delisted history. Some paid end-of-day vendors advertise it specifically, and AmiBroker’s data-sources page lists “delisted symbols” as a feature of the paid end-of-day offerings it links.
• Do you need corporate actions handled? A price series that has not been adjusted for splits contains one-day moves of several hundred per cent that are not moves at all. Whether adjustment happens, and whether dividends are included, is a property of the source and sometimes of how you import it.

2. History depth

Depth is a function of the test you want to run, and it is usually longer than people expect. A rule that trades roughly twelve times a year needs a great many years to produce a sample worth arguing about, and the sample has to include more than one kind of market.

Depth is also where the cost curve bends hardest. Twenty years of daily bars is small enough to be free. Ten years of one-minute bars for a few hundred symbols is a different product entirely, and the vendors that offer it price it accordingly.

✕Common mistake

Setting Number of bars to load to a large round number “to be safe” is described in AmiBroker’s own performance guide as the most common mistake people make. The setting applies only to plugin-fed databases, it costs roughly forty bytes per bar per symbol in memory, and oversized arrays fall out of the processor cache, so the penalty is paid in speed as well as in RAM. Enter the number of bars your question needs. The dialog shows you the equivalent number of days as you type.

3. Intraday resolution

Resolution is the base time interval of the database, and Part 19 treats it in detail because it is close to irreversible once a database is populated. For the purpose of choosing a provider, three things matter.

Finer is not better; finer is different. A tick database answers questions about individual prints. A one-minute database answers questions about bar-scale behaviour and is dramatically smaller. AmiBroker’s own eSignal documentation recommends one-minute as the base periodicity and warns that tick and second intervals pull enormous volumes — “several megabytes for just one symbol and very few days of history”.

The edition gates it. Tick and N-second base intervals are Professional-edition features. If you hold a Standard licence, the finest base interval available to you is one minute, and no subscription changes that.

The vendor gates it too, in a way you may not expect. Interactive Brokers’ streaming data is documented as consisting of snapshots taken every 0.2 to 0.3 seconds rather than every print, and AmiBroker’s own advice for that feed is to use five-second, fifteen-second or preferably one-minute bars rather than tick. Buying a tick-capable database does not create ticks that the feed never sent.

4. Budget, and the entitlements hiding inside it

The subscription line on a vendor’s page is rarely the whole cost. Four other items belong in the sum:

• Exchange entitlement fees, charged per exchange, sometimes per asset class. These are set by the exchanges and passed through.
• Your status, professional or non-professional. Exchanges charge markedly different rates, and the definition of “professional” is broader than most people assume — it usually turns on employment, registration and how you use the data, not on how much you trade.
• Add-on data types. Extended intraday history, options data and depth-of-book are commonly separate products.
• The AmiBroker side. The 64-bit build and the unrestricted real-time quote window are Professional-edition features, and AmiQuote is licensed separately from AmiBroker itself.

⚠Warning

Real-time market data is a recurring commitment, usually with an exchange agreement attached, and it is normally billed monthly whether or not you looked at it. Do not subscribe in order to learn. Subscribe when you have a specific study or a specific trading process that free data cannot support, and you can say which one.

5. Symbol count, and the rule you must not break

Streaming feeds are sold with a limit on how many symbols you may watch at once. AmiBroker’s documented ceilings are 100 streaming symbols for Interactive Brokers — which is the Trader Workstation limit — 500 by default for IQFeed, and 500 for the DDE plugin, with eSignal governed by whatever the subscription allows.

AmiBroker will let you put more tickers in the database than the subscription allows, and rotates them so the most recently used stay active. The Knowledge Base is blunt about the consequences of leaning on that: every new symbol forces the oldest one out, which triggers a backfill and a fresh stream, and doing that continuously across a large screen can overload the vendor’s servers. Vendors may act against accounts that do it. The rotation also does not apply to the Real-Time Quote window, which cannot hold more symbols than your subscription permits.

This is why “how many symbols do I need to watch simultaneously” is a requirement in its own right, and why a screen that scans a thousand symbols continuously is a different product from a workspace that watches thirty.

Measure what you already have

Requirements written in the abstract are guesses. The exploration below converts them into measurements, taken from the database in front of you.

Goal

Produce one row per symbol that states what this database actually holds: the grain it stores, how many bars arrived, how many distinct days those bars cover, how many bars a typical day contains, and which part of the session is present. Run it before you shop, so you know what you are missing. Run it again after a trial subscription, so you know whether the vendor delivered.

The formula

Complete runnable AFL

data-requirements-audit.afl

// data-requirements-audit.afl
// Part 18 - Choosing a Data Source: A Decision Framework
//
// Turns "how much data do I need?" into a number you measured rather than a
// number you guessed, and turns "did the provider deliver?" into the same
// measurement run again afterwards.
//
// Run it as an EXPLORATION. One row per symbol, describing what THIS database
// actually holds right now.
//
// Assumptions and limits - read these before believing the output:
//   - Works on any database: local, AmiQuote-fed, or plugin-fed. It reports
//     what is stored, not what the vendor advertises.
//   - Set Range to "All quotations" in the Analysis window. With any other
//     range the figures describe the range, not the database.
//   - "Trading days" counts distinct calendar days present in the loaded data.
//     A session the provider never delivered is simply not counted, so this is
//     a measure of what arrived, never of what the exchange traded.
//   - On a plugin-fed database, symbols are backfilled on first access. Run a
//     backfill pass first, or the audit measures your backfill state and not
//     the provider.

_SECTION_BEGIN( "Data requirements audit" );

// ---------------------------------------------------------------------------
// 1. What grain is this database set to?
// ---------------------------------------------------------------------------
// Interval() returns the bar length in seconds. Zero means tick. The base
// interval is a database-level setting, so every symbol reports the same value
// - a disagreement here means you are looking at a different database than you
// thought you were.
BarSeconds = Interval();

// Interval( 2 ) returns a human-readable name. It is translated in localised
// builds of AmiBroker, so it is safe to display but never safe to compare
// against a literal string. Numeric comparison is used below instead.
IntervalName = Interval( 2 );

if ( BarSeconds == 0 )
{
    Grain = "tick";
}
else if ( BarSeconds < inDaily )
{
    Grain = "intraday";
}
else
{
    Grain = "end-of-day or longer";
}

// ---------------------------------------------------------------------------
// 2. How much history actually arrived?
// ---------------------------------------------------------------------------
// The timestamp of bar zero is the real start of history for this symbol - the
// listing date, the vendor's retention limit, or wherever the backfill stopped,
// whichever came last. The formula cannot tell you which of the three it was.
FirstBarDT = LastValue( ValueWhen( BarIndex() == 0, DateTime() ) );
LastBarDT  = LastValue( DateTime() );
StoredBars = LastValue( Cum( 1 ) );

// A new calendar day begins wherever the day-of-month number changes. Ref()
// returns Null at bar zero because there is no previous bar, so Nz() supplies
// the answer we want there explicitly rather than letting Null propagate into
// the running total.
NewDay      = Nz( Day() != Ref( Day(), -1 ), 1 );
TradingDays = LastValue( Cum( NewDay ) );

// Bars per day is the number that decides whether an intraday study is even
// possible on this data. It is 1 for end-of-day data. For a regular-hours US
// equity session at 1-minute bars it should land close to 390; a number far
// below that means the session is being filtered, or the feed is delivering
// only the bars in which a trade printed.
BarsPerDay = StoredBars / Max( TradingDays, 1 );

// ---------------------------------------------------------------------------
// 3. Which part of the session arrived?
// ---------------------------------------------------------------------------
// TimeNum() returns the bar's time of day as HHMMSS. The earliest and latest
// values present tell you whether this source is delivering the pre-market and
// after-hours sessions or only regular hours - a difference that is usually an
// entitlement you are paying for, or one you are not.
//
// Two caveats. Timestamps are shown in your local time as adjusted by the
// database time shift, so these are not exchange times unless you have set the
// shift. And AmiBroker's Intraday Settings can be filtering extended-hours bars
// out of the display, in which case they are in the database but not in this
// array.
EarliestTime = LastValue( Lowest( TimeNum() ) );
LatestTime   = LastValue( Highest( TimeNum() ) );

// ---------------------------------------------------------------------------
// 4. Report
// ---------------------------------------------------------------------------
// One row per symbol: the exploration evaluates the whole array and shows the
// bars where Filter is true, so restricting Filter to the final bar of the
// range gives exactly one summary row for each symbol.
Filter = Status( "lastbarinrange" );

AddTextColumn( Grain,        "Grain",     1.0 );
AddTextColumn( IntervalName, "Interval",  1.0 );
AddColumn( BarSeconds,       "Bar secs",  1.0 );
AddColumn( StoredBars,       "Bars",      1.0 );
AddColumn( TradingDays,      "Days",      1.0 );
AddColumn( BarsPerDay,       "Bars/day",  1.1 );
AddColumn( FirstBarDT,       "First bar", formatDateTime );
AddColumn( LastBarDT,        "Last bar",  formatDateTime );
AddColumn( EarliestTime,     "Earliest",  1.0 );
AddColumn( LatestTime,       "Latest",    1.0 );

// Click the "Days" column header in the result list to sort shallowest history
// first: those are the symbols that will quietly shorten every study you later
// run over this universe.

_SECTION_END();

Download data-requirements-audit.afl115 lines

How it works

The formula has four parts.

The first reads the database grain from Interval(), which returns the bar length in seconds — zero for tick — and from Interval(2), which returns a readable name. The name is translated in localised builds of AmiBroker, so the code compares the numeric value against inDaily and displays the name only.

The second measures what actually arrived. ValueWhen(BarIndex() == 0, DateTime()) captures the timestamp of the very first bar and carries it forward to every later bar, so LastValue() can read it off the end. Cum(1) counts bars. Counting distinct days needs one more idea: a new day begins wherever the day-of-month number changes, and Ref(Day(), -1) has no previous bar to look at on bar zero, so Nz() supplies the answer explicitly instead of letting a Null propagate into the running total.

The third asks which part of the session is present, by taking the lowest and highest TimeNum() values in the array. On a feed that carries pre-market and after-hours trading these will straddle the regular session; on one that does not, they will not.

The fourth reports. Filter = Status("lastbarinrange") is the standard way to get exactly one summary row per symbol: the exploration evaluates the whole array and displays the bars where Filter is true, so restricting it to the final bar of the range collapses the symbol to a single line.

Key functions

• Interval( format ) — bar length. Format 0 gives seconds, 2 gives a readable name. The documentation warns explicitly against comparing that name to a literal string, because it is translated.
• Status( "lastbarinrange" ) — true on the last bar of the Analysis range.
• ValueWhen( condition, array ) — the value of array at the most recent bar where condition was true.
• Nz( x, valueifnull ) — replaces Null, Nan and infinities with a value you choose, so missing data stops propagating.
• Lowest( array ) and Highest( array ) — the running minimum and maximum from the start of the array, so LastValue() of either is the extreme over the whole range.

Expected result

=Expected result

One row per symbol. On a free end-of-day database of large US shares you should see Grain reading “end-of-day or longer”, Bar secs of 86400, Bars/day of exactly 1.0, and Days equal to Bars. On a one-minute intraday database of the same shares, Bar secs reads 60 and Bars/day should land near 390 for regular-hours data — noticeably higher if extended hours are included, and lower if the source only writes a bar when a trade printed.

Test it

Run it twice: once with the Analysis range set to All quotations, and once with it set to a short recent range. The bar and day counts must change and the grain must not. If the grain changes, you are looking at a different database from the one you thought. If the counts do not change, the range setting did not take effect.

Then check one row by hand. Pick a symbol, open its chart, and compare the first and last bar dates with the columns. They should match exactly.

Common errors

• Every symbol shows one or two bars. On a plugin-fed database, symbols are backfilled on first access, so an Exploration that has never touched them measures your backfill state rather than the provider. Backfill first.
• Bars/day is 1.0 on data you believe is intraday. The database base interval is end-of-day. Nothing downstream can fix that; it is a database-creation setting.
• Days looks wrong on tick data. With Interval() returning zero the day counting still works, but bars-per-day becomes a very large number that is not telling you anything useful about resolution.

Extension

Add a column that counts the bars in the most recent five distinct days, and compare it with BarsPerDay. A symbol whose recent days are much thinner than its average is either losing liquidity or losing data, and the audit cannot tell you which — but it can tell you which symbols to look at.

The decision table

With five answers in hand, the options narrow quickly. This table maps requirements to the class of source, not to a specific vendor, because the class is stable and the vendors are not.

If your requirement is	The class of source you need	Where to read more
Daily bars, long history, wide universe, no cost	Free end-of-day download	Free and end-of-day sources
Daily bars, long history, delisted symbols, sectors, maintained	Paid end-of-day plugin	eSignal and other supported feeds
Intraday bars for study, no live streaming needed	Historical intraday, or accumulated daily downloads	Free and end-of-day sources
Live quotes on instruments you already trade, modest symbol count	Broker-provided feed	Interactive Brokers
Live quotes plus deep, fast intraday backfill, hundreds of symbols	Dedicated market-data vendor	IQFeed
A source with no AmiBroker plugin	A generic bridge, or an import routine	eSignal and other supported feeds

The middle rows are where most people land and where most people over-buy. A broker feed you are already entitled to costs nothing extra to try; a dedicated vendor is a monthly commitment. The appendices exist so that you can tell which of your requirements genuinely forces the second option.

Why most learners should start at Level A

Level A — no subscription, free end-of-day data, historical intraday where you can get it, and Bar Replay for anything that needs to feel live — is not the beginner’s consolation prize. It is the right starting point for four concrete reasons.

Almost everything in this course runs on it. Indicators, patterns, screening, exploration, backtesting, portfolio construction, ranking, optimisation, walk-forward and Monte Carlo are all end-of-day capable, and that is the bulk of the material.

A live feed hides your mistakes. When something looks wrong on a streaming chart there are a dozen candidate causes: the vendor, the plugin, the connection, the session settings, the base interval, your formula. On a static end-of-day database there is one candidate, and it is your formula. Learning to write correct AFL is much faster in the environment with fewer variables.

Bar Replay reproduces the experience you think you are missing. It plays stored data forward for every symbol at once, at a speed you set, and every formula — chart or Analysis — sees only the data up to the playback position. GetPlaybackDateTime() returns that position, or zero when replay is not running. It is the documented tool for exactly this, and it costs nothing.

Subscriptions are easier to start than to justify. A monthly feed bought before you have a process tends to become a monthly feed you keep because you are already paying for it. Buying one after you have a study that free data cannot support is a decision you can defend.

✦Tip

There is a fifth reason, less obvious. Working at Level A forces you to be precise about which of your questions genuinely need finer data. That precision is the whole content of this lesson, and it is much easier to acquire before you have paid for the answer than afterwards.

What changed

You started with a question that has no answer and now have a procedure. State the research question; derive instruments and exchanges, depth, resolution, budget and entitlements, and simultaneous symbol count; measure what your existing database already delivers; map the gap to a class of source; and only then read a vendor’s page — with the appendices beside you, so you know which of its claims AmiBroker’s own documentation supports and which it does not mention at all.

The three appendices that follow are the fourth step. The last lesson of this part is the first step for most readers, and it costs nothing.

Check your understanding

Question 1. A reader wants to test whether the spread widens before an earnings announcement. They have twenty years of daily bars. What do they need?

• More history
• A finer bar interval
• A different kind of record entirely — quote data, not bar data
• A paid end-of-day vendor

Show the answer and why

Answer: A different kind of record entirely — quote data, not bar data

Bars record trades. Bid and ask are quotes, and no depth or resolution of trade-derived bars reconstructs them. Recognising when a question needs a different record type, rather than more of the same one, saves both money and wasted work.

Question 2. Which statements about "Number of bars to load" are correct? Select all that apply.

• It has no effect on a database whose data source is (local)
• Setting it far larger than needed costs memory and speed
• It controls the base time interval of the database
• After increasing it, previously backfilled symbols need a forced backfill to gain the extra bars

Show the answer and why

Answer: It has no effect on a database whose data source is (local), Setting it far larger than needed costs memory and speed, After increasing it, previously backfilled symbols need a forced backfill to gain the extra bars

The setting applies only to plugin-fed databases; it costs roughly forty bytes per bar per symbol and oversized arrays lose the benefit of processor cache; and AmiBroker documents that enlarging it requires a forced backfill for symbols already filled at the smaller count. The base interval is a separate setting entirely.

Question 3. Your subscription allows 100 streaming symbols. You add 900 tickers to the database and run a continuous scan over all of them. What does AmiBroker's documentation say happens?

• AmiBroker refuses to add symbols beyond the limit
• Symbols rotate so recently used ones stay active, and doing this at scale can overload the vendor and provoke action against the account
• The extra symbols are served from local storage instead
• Nothing — the limit applies only to the Real-Time Quote window

Show the answer and why

Answer: Symbols rotate so recently used ones stay active, and doing this at scale can overload the vendor and provoke action against the account

The rotation mechanism is real and documented, and so is the warning about abusing it: each new symbol displaces the oldest, forcing a backfill and a fresh stream. The Real-Time Quote window is the one place rotation does not apply at all.

Question 4. Given `Filter = Status( "lastbarinrange" );` in an Exploration over 500 symbols with the range set to All quotations, how many result rows do you get?

Filter = Status( "lastbarinrange" );

• One per bar
• One per symbol
• One in total
• One per trading day

Show the answer and why

Answer: One per symbol

The Exploration evaluates the whole array for each symbol and shows the bars where Filter is true. Restricting Filter to the final bar of the range yields exactly one summary row per symbol — the standard idiom for a per-symbol report.

Check my answersClear