Intraday Database Settings in Detail

The Database Settings dialog has about eight controls on it. Four of them do nothing whatsoever in a local database, one of them is the most common performance mistake in the program, and one of them silently decides whether TimeNum() in your formulas returns exchange time or the time on your own wall clock.

✱AmiBroker Professional required

Knowing which is which takes twenty minutes and saves an evening.

Two dialogs, and which one wins

Intraday database behaviour is set in two places, and they are not equals.

File → Database Settings holds the per-database settings. Tools → Preferences → Data holds values with the same names, but there they are only defaults used when a new database is created. Database Settings takes precedence — with one documented exception: if the Data source combo is left on Default, or the Local data storage radio is left on Default, the Preferences value is used after all.

The dialog itself has two halves. The General half — the folder path and the Create button — is enabled only at File → New → Database time and is disabled afterwards. The Data source half stays editable for the life of the database and is what you reach through File → Database Settings.

Where each intraday setting actually lives

1. File → Database SettingsData source, Local data storage, Number of bars to load, Base time interval, Flush cache, Configure, Intraday settingsper database, wins
2. File → Database Settings → Intraday settingsSession filtering, trading hours, daily compression basis, Time shift, mixed EOD/Intradayper database
3. Symbol → Categories → Groups → Group uses own intraday settingsA second, per-group copy of the Intraday Settings dialogper group, overrides
4. Tools → Preferences → DataIn-memory cache size, Limit number of saved quotations, and defaults for NEW databases onlyglobal
5. Tools → Preferences → IntradayCustom N-minute and N-tick intervals, bar alignment, compressed-bar timestamps, chart refresh intervalglobal

Troubleshooting an intraday database means checking all five, in this order.

Data source: the control that makes four others inert

The Data source combo binds this database to a plug-in, or to nothing.

Set to (local), AmiBroker maintains the quotes itself in its own files, and you fill it by importing — the ASCII importer, the Import Wizard, or AmiQuote through Tools → Auto-update quotes. Set to a plug-in, AmiBroker stops reading its own quote files for prices and asks the plug-in DLL for them, symbol by symbol, as they are needed.

Two things surprise people here.

First, a plug-in gives read-only access to the vendor’s data, and AmiBroker never writes back to it. But you still need an AmiBroker database folder, because your hand-drawn studies, category assignments, watch lists, composites and favourites are always stored in AmiBroker’s own files regardless of where the prices come from. “External data source” does not mean “no database”.

Second, and this is the one that wastes evenings: in a (local) database, Local data storage and Number of bars to load have no effect at all. Tuning them on a database fed by imported CSV files does nothing, in either direction.

✕Common mistake

A reader builds a local intraday database from files, finds the history is shorter than expected, and raises Number of bars to load from 30,000 to 200,000. Nothing changes, because that setting governs how many bars are pulled from an external source. In a local database the limit on history is the contents of the files you imported. The symptom looks identical to a plug-in backfill limit, which is why the challenge later in this part starts by asking what kind of database you are actually looking at.

Two further notes about plug-in databases. The list of data sources printed in the User’s Guide is dated — it omits IQFeed, Interactive Brokers, DDE, ODBC and Norgate, all of which are real, documented plug-ins — so the authoritative list is whatever your installed copy shows in the combo. And when symbols are added to or removed from the external source, you must reopen File → Database Settings → Configure and press RETRIEVE again, or your symbol list drifts out of date without any warning.

Local data storage

Enabled caches the plug-in’s data in AmiBroker’s own database files. Disabled does not, so every access goes back through the plug-in.

The setting is documented as required (Enabled) for most real-time sources — eSignal, myTrack and QuoteTracker are named. For the end-of-day plug-ins the guide’s setup instructions differ per vendor: Quotes Plus, FastTrack and Metastock are documented with local storage Disabled, TC2000/TCNet with it Enabled, where the guide reports it gives a speed-up of more than ten times because AmiBroker caches the data in its own native format after the first read.

If you are building an intraday database on a real-time plug-in, this is effectively not a choice: enable it.

Number of bars to load: a history limit wearing a performance-setting costume

This is the control that causes most incomplete intraday history, and the reason is that its unit is wrong for the way people think.

You think in days of history. The setting is in bars per symbol, counted backwards from the newest bar. The conversion factor between them is how many bars your feed stores per day, which depends on the base interval, on whether extended-hours bars are collected, and on how actively the symbol trades.

The documented examples give you the scale: about 2,600 bars for ten years of end-of-day; about 30,000 for sixty days of one-minute intraday; 100,000 in the IQFeed walkthrough, described there as the maximum history that feed offered, roughly eight months.

Three mechanical facts govern how you set it:

• The memory is really allocated. One data bar is 40 bytes, so 100,000 bars is about 4 MB per symbol. With a 500-symbol in-memory cache the performance chapter notes you can reach around 2 GB.
• Bigger is slower as well as larger. Arrays that fit in the CPU’s on-chip cache are processed roughly ten times faster than arrays that do not. With a 4 MB CPU cache the guide advises strongly against exceeding 100,000 bars.
• The dialog tells you the answer. Next to the field, AmiBroker displays the equivalent number of days for the bar count and base interval you have chosen. The guide calls oversizing “the most common mistake people make”, citing someone who entered 50,000 into an end-of-day database — which is 192 years.

!Important

After increasing Number of bars to load, symbols that were already backfilled at the smaller count do not gain the extra history by themselves. You have to force a backfill for them. This is the single most frequently missed step in intraday database work, and the next lesson covers how to do it.

The default ceiling on this field was 500,000 bars historically and is 1,000,000 from AmiBroker 6.17 onwards, raisable further through a MaximumNumberOfBars registry value. That last detail comes from the official support forum, in a thread answered by AmiBroker’s author, rather than from the User’s Guide — so treat it as accurate but less authoritative than the guide, and note that if you are contemplating it you have almost certainly designed the database wrongly.

In-memory cache, and the setting that can lose data

Tools → Preferences → Data carries two caches that are easy to confuse.

In-memory cache (max. symbols) and (max. megabytes) control how many symbols’ data AmiBroker keeps in RAM after reading them. By default it holds the ten most recently accessed symbols. Larger is faster, because data already in RAM does not need to be re-read or re-requested from a plug-in. The minimum value is 11 symbols, and the guide is explicit that you should not shrink it for small databases — reduce it only when a very large database, or a 32-bit build whose process is limited to 2 GB of virtual memory, is running you out of memory.

Limit number of saved quotations with its companion Max. number of saved quotations is a completely different thing, and it is dangerous on an intraday database. It caps how many quotations are saved, and the guide’s recommended minimum for an intraday database is 3,000 (against 300 for end-of-day). A value carried over from an end-of-day setup will quietly discard intraday history that you successfully downloaded.

⌀Data quality warning

If your intraday database appears to lose old bars every time you close AmiBroker, check Limit number of saved quotations before you blame the feed. It is one of very few settings in the program that can destroy data you already have.

Intraday Settings: sessions, filtering and time shift

File → Database Settings → Intraday settings opens a second dialog, and it governs what is shown rather than what is stored.

• Filtering. Show 24 hours trading (no filtering), Show day session only, Show night session only, or Show day and night session only.
• Filter weekends. Weekend data is still collected; it is simply not displayed.
• Trading hours. Start and end times for the day session and the night session, defined separately — and defined in your local time zone, not the exchange’s.
• Daily time-compression uses. Exchange time, Local time, or the day/night session times defined above. This decides which bars are grouped into each daily bar when you view an intraday database at daily periodicity.
• Time shift. The number of hours between your local time zone and the exchange’s.
• Allow mixed EOD/Intraday data. Lets the database hold both true daily bars and intraday bars.

!Important

Filtering hides bars; it does not delete them. The Quote Editor is documented as the one window that always shows every stored bar regardless of filtering. If bars vanish from a chart but are present in the Quote Editor, you are looking at a filter, not at missing data — and you have just saved yourself from re-downloading history you already own.

There is also a per-group override: Symbol → Categories → Groups has a checkbox “Group uses own intraday settings” with its own copy of this dialog. It exists so that instruments on different exchanges can have different session hours in one database. It is also why an instrument can ignore the database-wide session times you just set, which is baffling until you know the override exists.

Two settings on Tools → Preferences → Intraday interact with all of this and belong in your notes:

• Time compressed bars show — first tick, last tick, the start time of the interval, or the end time of the interval. This changes the timestamp written on every compressed bar, and therefore what TimeNum() compares against in your formulas. A session-open test that works on one machine and fails on another is often this setting.
• Align custom minute bars to regular market hours — without it, 45-minute bars start at 9:00, 9:45, 10:30; with it, and a 9:30 open, they align to 9:30, 10:15, 11:00.

Reading what you actually configured

Settings dialogs tell you what you asked for. They do not tell you what the database is doing. This short indicator does.

Goal

Put a permanent readout on screen that states the interval of the current pane, the database time shift, how many bars this formula was actually handed, the ends and span of that data, how many sessions it covers, and whether Bar Replay is truncating everything right now. Every one of those has been the answer to a “why is my intraday database wrong?” question.

Complete formula

Complete runnable AFL

database-settings-readout.afl

// ===========================================================================
//  Database settings readout
//  Prints, in the chart title, what this database and this chart pane are
//  actually configured to do - as opposed to what you believe you configured
//  when you created them. Nearly every intraday problem in Part 19 is a
//  mismatch between those two things.
//
//  HOW TO RUN
//    Formula Editor -> Apply Indicator, in its own small pane.
//    To read the database's BASE interval rather than some other interval,
//    switch the chart to the intraday toolbar button marked "i", which means
//    "the base intraday interval set in File -> Database Settings".
//
//  WHAT IT REPORTS
//    Chart interval      the interval of THIS pane, in seconds and by name
//    Time shift          the database time shift, from Status("timeshift")
//    Bars loaded         how many bars AFL actually received for this symbol
//    First / Last bar    the ends of what was loaded, not of what is stored
//    Calendar span       days between the first and the last bar
//    Sessions            how many distinct calendar days those bars cover
//    Bars per session    loaded bars divided by sessions
//    Bar Replay          whether playback is truncating the data right now
//
//  ASSUMPTIONS, STATED SO THEY CAN BE CHECKED
//    - "Bars loaded" is not "bars stored". AmiBroker may hand a formula fewer
//      bars than the database holds, because of the Analysis range, because of
//      QuickAFL, or because Bar Replay is active. The Quote Editor is the only
//      window documented to always show every stored bar.
//    - A "session" here means a distinct calendar date, which is not the same
//      as an exchange session for instruments that trade overnight. On a
//      24-hour instrument one exchange session can straddle two dates.
//    - Interval() returns the interval of the chart, never the base interval
//      of the database. They are the same number only when the chart is set to
//      the base interval.
// ===========================================================================

_SECTION_BEGIN( "Database settings readout" );

Plot( Close, "Close", colorDefault, styleCandle );

IntervalSeconds = Interval();
IntervalName    = Interval( 2 );

// Status("timeshift") returns the database time shift in seconds (AmiBroker
// 5.60 and later). It is the setting from Database Settings -> Intraday
// settings -> Time shift, expressed as seconds rather than hours.
TimeShiftSeconds = Status( "timeshift" );
TimeShiftHours   = TimeShiftSeconds / 3600;

BarNumber  = Cum( 1 );
BarsLoaded = LastValue( BarNumber );

FirstBarDateTime = LastValue( ValueWhen( BarNumber == 1, DateTime() ) );
LastBarDateTime  = LastValue( DateTime() );

// DateTimeDiff returns seconds, positive when the first argument is later.
SpanDays = DateTimeDiff( LastBarDateTime, FirstBarDateTime ) / 86400;

// A new calendar date starts wherever the day number changes. Cum(1) == 1
// forces the very first bar to count, because Ref() has no previous bar there.
NewSession   = BarNumber == 1 OR Day() != Ref( Day(), -1 );
SessionCount = LastValue( Cum( NewSession ) );

BarsPerSession = IIf( SessionCount > 0, BarsLoaded / SessionCount, Null );

// GetPlaybackDateTime() returns the Bar Replay position, or zero when replay
// is not active. A forgotten Bar Replay truncates every chart and every
// Analysis run in the program, so it is worth showing permanently.
PlaybackPosition = GetPlaybackDateTime();

ReplayLine = "Bar Replay: not active";

if ( PlaybackPosition != 0 )
{
    ReplayLine = "BAR REPLAY ACTIVE - data truncated at "
               + DateTimeToStr( PlaybackPosition );
}

IntervalLine = "Chart interval: " + IntervalName
             + "  (" + NumToStr( IntervalSeconds, 1.0 ) + " s)";

if ( IntervalSeconds >= 86400 )
{
    IntervalLine = IntervalLine
                 + "  - this pane is NOT intraday";
}

Title = "DATABASE READOUT - " + Name() + "\n"
      + IntervalLine + "\n"
      + "Time shift: " + NumToStr( TimeShiftHours, 1.2 ) + " h"
      + "  (" + NumToStr( TimeShiftSeconds, 1.0 ) + " s)\n"
      + "Bars loaded: " + NumToStr( BarsLoaded, 1.0 )
      + "   Sessions: " + NumToStr( SessionCount, 1.0 )
      + "   Bars per session: " + NumToStr( LastValue( BarsPerSession ), 1.1 ) + "\n"
      + "First bar: " + DateTimeToStr( FirstBarDateTime )
      + "   Last bar: " + DateTimeToStr( LastBarDateTime )
      + "   Span: " + NumToStr( SpanDays, 1.1 ) + " days\n"
      + ReplayLine;

_SECTION_END();

Download database-settings-readout.afl100 lines

How it works

It reads four independent sources of truth and prints them next to each other.

Interval() reports the interval of the chart pane the formula is running in — never the base interval of the database. They coincide only when the chart is set to the base interval, which is what the intraday toolbar button marked i does. Interval(2) gives the same thing as a readable name.

Status("timeshift") returns the database time shift in seconds, so the formula divides by 3,600 to show hours. If that number is not what you set in Intraday Settings, something is overriding it — most likely a per-group override.

The bar accounting is deliberately blunt. Cum(1) numbers the bars from one, so LastValue(Cum(1)) is how many bars this formula received. That is not the same as how many bars the database holds: an Analysis range, AmiBroker’s QuickAFL optimisation or an active Bar Replay all reduce it. Showing it makes the difference visible instead of mysterious.

Sessions are counted by detecting where the day number changes, with the first bar forced to open a session because Ref() has no previous bar to compare against there. Dividing bars by sessions gives bars per session, which is the conversion factor you need for every piece of sizing arithmetic in this part.

Finally, GetPlaybackDateTime() returns the Bar Replay position, or zero when replay is not running. A Bar Replay left paused truncates every chart and every Analysis run in the program, so it earns a permanent line on the readout.

Key functions

• Interval( format ) — bar interval of the current chart, in seconds by default; format = 2 returns the interval’s name as a string.
• Status( "timeshift" ) — database time shift in seconds, available from AmiBroker 5.60.
• DateTimeDiff( a, b ) — difference between two DateTime values in seconds, positive when a is later.
• DateTimeToStr( number ) — renders a DateTime value as readable text.
• GetPlaybackDateTime() — Bar Replay position, or zero when replay is inactive.

Expected result

A four-line title block above the price chart. On a 1-minute United States equity database with the chart on the i interval, “Chart interval” reads 1-minute (60 s), “Bars per session” lands somewhere near 390 if only regular hours are stored and appreciably higher if extended hours are, and “Bar Replay: not active” sits on the last line.

Test it

Switch the chart from the base interval to 15-minute. The interval line should change and bars per session should fall by roughly a factor of fifteen while the session count stays the same. Then open Tools → Bar Replay, set a start date, press Pause, and watch the last line change to the replay warning and the “Last bar” date jump backwards. Press Stop and both revert. If they do not, the formula is not being re-evaluated — click on the chart to force a refresh.

Common errors

Reading the “Chart interval” line as the base interval is the big one; it is the interval of that pane and nothing more. Expecting “Bars loaded” to equal the stored bar count is the second; use the Quote Editor when you need the true stored count. And a time shift of zero on a database whose bar timestamps are clearly not local is not a contradiction — it means the shift is being applied somewhere else, such as a group override or the vendor’s own timestamps.

Extension

Add a line that flags when bars per session differs by more than ten per cent from the value seen on the previous session, which turns the readout into an early warning that the feed’s session coverage has changed.

Sizing the database for your machine

A procedure you can defend, rather than a number you copied:

1. Decide history in sessions, not days. “Six months” is roughly 126 trading sessions; “one year” is roughly 252.
2. Measure bars per session on your own data, using the readout above. Do not assume 390 — extended-hours collection and illiquid symbols both move it.
3. Multiply, then add about 20 per cent for half-days, unusual sessions and the fact that the cap is consumed from the newest bar backwards.
4. Enter it and read the days-equivalent the dialog prints beside the field. If it disagrees with step 1, one of your assumptions is wrong.
5. Check the memory: bars × 40 bytes × the number of symbols your in-memory cache holds. Compare that against the RAM you are willing to spend.
6. Compare against 100,000. If you are above it, either your universe is too large for one intraday database or your history requirement belongs in a separate end-of-day database.

If you have no live feed

Everything on the Intraday Settings dialog, the whole of Tools → Preferences → Intraday, the base interval, the in-memory cache and the saved-quotations limit apply exactly as described to a (local) database populated by importing files. Set them, change one at a time, and watch the readout formula respond.

What you cannot exercise is the plug-in half: Data source, Local data storage, Number of bars to load, Flush cache and Configure → RETRIEVE are all inert without an external source. Read those sections for understanding, and use the lab’s Route B to build the database itself. When you later add a feed, the only genuinely new decision is the bar count — and by then you will have measured your own bars per session and can size it properly.

Recap

Database Settings beats Preferences, except where a control is left on Default. In a local database, Data source, Local data storage and Number of bars to load are inert, which makes “raise the bar count” a non-fix for a history problem you have. Number of bars is a per-symbol allocation at 40 bytes each, sized in bars rather than days, and after raising it you must force a backfill. The in-memory cache should usually be left large; the saved-quotations limit is the one that can actually lose data. Intraday Settings controls display and compression rather than storage, its trading hours are in your local time zone, and a per-group override can quietly countermand it.

The next lesson is about filling the database and keeping what is in it correct.

Check your understanding

Question 1. A reader with a local intraday database built from CSV files raises "Number of bars to load" from 30,000 to 200,000 and forces a refresh. What happens to their history?

• It extends back further, up to whatever the files contain
• Nothing: the setting has no effect in a database whose data source is (local)
• It extends back only after AmiBroker is restarted
• It is truncated to the first 200,000 bars

Show the answer and why

Answer: Nothing: the setting has no effect in a database whose data source is (local)

Number of bars to load governs how many bars are pulled from an external source. In a local database there is no external source, so the control does nothing. The history is whatever was imported.

Question 2. Bars are visible in the Quote Editor for a session but absent from the chart. What is the most likely explanation?

• The bars are corrupted and the chart is refusing to draw them
• Session filtering in Intraday Settings is hiding them
• The backfill has not completed
• The base interval is set too coarsely

Show the answer and why

Answer: Session filtering in Intraday Settings is hiding them

Filtering hides data rather than deleting it, and the Quote Editor is documented as always showing every stored bar. This pair of observations is the fastest way to separate a display problem from a data problem — and it is the difference between changing one setting and re-downloading a month of history.

Question 3. Which of these can actually cause you to lose intraday data you already downloaded? Select all that apply.

• Setting "Limit number of saved quotations" to an end-of-day-sized value
• Reducing the in-memory cache to its minimum of 11 symbols
• Switching session filtering to "Show day session only"
• Deleting broker.workspace from the database folder

Show the answer and why

Answer: Setting "Limit number of saved quotations" to an end-of-day-sized value, Deleting broker.workspace from the database folder

The saved-quotations limit caps what is written, and 300 is far too low for intraday, where the guide recommends 3,000 or more. Deleting broker.workspace loses the database settings and resets category names. The in-memory cache only affects speed, and filtering only affects display.

Question 4. What does Interval() return when called from a chart pane?

IntervalSeconds = Interval();

• The database base time interval, in seconds
• The interval of the chart pane the formula is running in, in seconds
• The interval of the most recently opened Analysis window
• The vendor’s native bar interval

Show the answer and why

Answer: The interval of the chart pane the formula is running in, in seconds

Interval() is a property of the execution context, not of the database. It equals the base interval only when the chart happens to be set to it — which is what the "i" toolbar button does. Confusing the two produces sizing arithmetic that is wrong by whatever compression factor the chart is applying.

Question 5. Your intraday session tests behave differently on a colleague’s machine with the same database. Which setting is the most likely cause?

• In-memory cache (max. megabytes)
• Preferences → Intraday → "Time compressed bars show"
• Local data storage
• Maximum number of chart sheets

Show the answer and why

Answer: Preferences → Intraday → "Time compressed bars show"

That setting decides whether a compressed bar is stamped with its first tick, last tick, the start of its interval or the end of it. Change it and every TimeNum() comparison shifts by up to one bar. It is a global preference, so two machines can disagree while sharing an identical database.

Check my answersClear