Daylight Saving and Alignment Failures

Twice a year a large number of countries move their clocks, and they do not do it on the same weekend. For a few weeks either side of each change, the offset between any two of those countries is not what it was — and any part of your setup that encoded that offset as a number is now wrong by an hour, without having changed.

This lesson is about the specific damage that hour does, how to find it in a history you already have, and how to write intraday code whose correctness does not depend on the offset staying still.

What actually changes, and what does not

Three things move independently.

The exchange shifts its session because the country it sits in shifts its clocks. In local terms nothing happened: the market still opens at the same local time it always did. In your terms, if you are somewhere else, the session just moved by an hour.

Your operating system shifts on its own country’s date, which affects Now(), the Windows scheduler, and therefore when a repeated scan actually runs and when an alert actually reaches you.

The provider’s stamps shift or do not shift depending on the zone the provider expresses them in. A feed stamped in coordinated universal time never shifts. A feed stamped in an exchange’s local time shifts with that exchange. A feed stamped in the provider’s own local time shifts with the provider’s country, which may be neither yours nor the exchange’s.

Offset between two markets across one year, schematically

Offset

Baseline−1 hBaseline+1 hBaseline

JanAprAugDec

Schematic, not a calendar. Two countries that both observe daylight saving but change on different dates spend two short windows a year at an offset that differs from the rest of the year. The widths and the sign depend entirely on which two countries, and the dates are set by legislation and revised from time to time — look them up for the year in question rather than trusting a remembered rule.

Three things do not change and this is where the trouble comes from.

The database time shift is a constant number of hours. The Intraday Settings dialog describes it as the difference between local and exchange time, Status("timeshift") returns it as a fixed number of seconds, and the ASCII importer’s $TIMESHIFT command takes a single number applied to the whole import. A constant is exactly the right shape for a difference that is constant. It is exactly the wrong shape for a difference that takes two values during the year.

The literal numbers in your formulas do not change. TimeNum() >= 93000 means the same thing in March as in November. Whether it means what you intended is a different question.

And the schedules you configured — a repeated scan every five minutes between two wall-clock times, an alert window, a batch job — keep firing on the machine’s clock, which is now an hour away from the session it was aimed at.

!Important

Not every country observes daylight saving, and among those that do the dates differ, the direction differs between hemispheres, and the rules are changed by legislation more often than most people expect. Do not encode “the clocks change on the last Sunday in October” anywhere. Encode nothing about the calendar at all, and derive what you need from the data in front of you. The rest of this lesson is about how.

What an hour does to an opening range

Consider a formula that computes an opening range from the bars stamped 09:30:00 to 09:55:00 and takes a signal when price closes above that range’s high. It is correct on Friday. On Monday the stamps in the database have moved an hour earlier, because the exchange’s country changed its clocks and your provider stamps in exchange time while your database applies a fixed shift.

The same session, before and after a one-hour displacement

Illustrative stamps for five-minute bars, start-of-interval convention. On Monday the six bars that make up the opening range are outside the window the formula tests, and the only bar inside it is the seventh — an hour into the session.

Bar	Bar 1	Bar 2	Bar 3	Bar 4	Bar 5	Bar 6	Bar 7
Friday stamps	093000	093500	094000	094500	095000	095500	100000
Monday stamps	083000	083500	084000	084500	085000	085500	093000
Friday: in range?	1	1	1	1	1	1	0
Monday: in range?	0	0	0	0	0	0	1

Illustrative stamps for five-minute bars, start-of-interval convention. On Monday the six bars that make up the opening range are outside the window the formula tests, and the only bar inside it is the seventh — an hour into the session.

Follow what the formula does with that. HighestSince( RangeStartBar, High ) needs a bar stamped 09:30:00 to start counting from; on Monday the bar carrying that stamp is the seventh bar of the session. ValueWhen( RangeEndBar, ... ) needs a bar stamped 09:55:00; there is none at all on Monday, because the session’s stamps now run 08:30 to 15:00 and the market closed before reaching 09:55 in the new alignment — or, in the opposite-direction case, that stamp lands in the middle of the afternoon.

The result is one of two symptoms, and which one you get depends on the direction of the shift:

• Silence. The end-of-range stamp does not exist, ValueWhen returns Null, the opening range levels are empty, Cross( Close, Null ) is never true, and the formula produces no signals at all. Nothing on screen says why.
• Nonsense. Both stamps exist but in the wrong places, so the “opening range” is computed from an hour of mid-morning trading and the “breakout” is measured against it. Signals appear. They are signals about a range that has no relationship to the open.

The second is much worse than the first, because the first at least stops. A scan that returns nothing gets investigated. A scan that returns a plausible list gets traded.

The rest of the damage

The opening range is the demonstration case because the failure is visible. The same displacement quietly changes several other things at the same time.

Daily bar construction. The Intraday Settings dialog’s Daily time-compression uses option chooses between exchange time, local time and your defined day/night session times as the boundary for turning intraday bars into daily bars. If that boundary and the actual session stop lining up, some of a session’s bars are assigned to the previous or the next calendar day. The daily open, high, low and close all change, and so does everything computed from them.

Anything anchored to the session open. A volume-weighted average price that resets at the start of each session is anchored to a moment. Anchor it to the wrong bar and it is a different statistic with the same name — the weights are computed over a window that starts an hour off, and the number it produces looks entirely ordinary.

Alerts and scheduled work. A repeated scan configured to run between two wall-clock times on your machine is now aimed at a session that starts an hour earlier or later. You will still get alerts. They will be about the wrong part of the day, and for the first hour of the session there will be no scanning happening at all.

Intraday backtests. Every one of the above is present in a backtest over a period that contains a transition, and none of it is flagged. The test runs, the report is produced, and the results for those weeks were computed from session definitions that did not match the sessions. Part 30 catalogues the ways a backtest can flatter itself; this is one that does not even require a mistake in the trading logic.

⚠Warning

The worst property of this fault is its schedule. It appears twice a year, persists for a few weeks in the mismatched window, and then repairs itself when the second country catches up. Anyone diagnosing it in the middle looks at a formula that is now working again and concludes the problem was transient. It was not transient; it was seasonal, and it will be back.

The 23-hour day, the 25-hour day and the repeated hour

Two structural oddities follow from the clock change itself rather than from a mismatch between countries.

On the day clocks go forward, one hour of local time does not exist. An instrument trading through that moment has a day that is one hour shorter, with a corresponding hole in the stamps. On the day clocks go back, one hour occurs twice. An instrument trading through that moment produces two distinct hours of trading whose stamps are identical, if the stamps are expressed in a zone that observes the change.

For an instrument that is closed overnight in the stamping zone, neither of these is visible: the clocks move while the market is shut, and the next session simply starts at a different point relative to your own clock. For an instrument that trades continuously — overnight futures, currencies — the transition happens mid-session and the effects are in the data.

⌀Data quality warning

How a specific plugin resolves duplicate stamps in a repeated hour is a property of that plugin and that vendor, not of AmiBroker, and it is not something the official documentation settles in general. Do not assume; check. The diagnostic in the challenge at the end of this part counts how many bars carry a given stamp on each day, and a count above one on a transition date tells you what your own data actually did. Treat any bar that shares a stamp with another bar as suspect until you have established what your provider intends by it.

Detecting a shift in your own data

You do not need to know the calendar. A clock change has a signature that is visible in the bar stamps: the first and the last bar of the day both move, by the same amount, in the same direction, by exactly sixty minutes.

Goal

Scan a history of at least a year and report every trading day whose first bar is stamped at a different clock time from the previous available trading day, along with how far the end of the day moved and how many bars each day held. Then separate the clock changes from the short sessions and the data gaps by their fingerprints rather than by the calendar.

The formula

Complete runnable AFL

session-shift-detector.afl

// ===========================================================================
//  Session shift detector
//  Reports every trading day whose first bar is stamped at a different clock
//  time from the previous available trading day. Twice a year, around a
//  daylight-saving transition, that difference is exactly sixty minutes and
//  the whole session moves with it. The rest of the time a difference usually
//  means a half day, a missing early session, or a gap in the data.
//
//  HOW TO RUN
//    Analysis window -> Apply to: Current symbol or a watch list,
//    Range: at least a year, so both transitions fall inside it,
//    Periodicity: the database's base interval, then Explore.
//
//  READING THE RESULT
//    "Start shift" and "End shift" both equal to +60 or both to -60 is the
//    signature of a clock change: the entire session moved by an hour.
//    A start shift near zero with a much earlier end is a short session.
//    A shift with a small bar count on either side is more likely to be
//    missing data than a change of rules.
//    None of these is a diagnosis. The table tells you which day to look at.
//
//  ASSUMPTIONS, STATED SO THEY CAN BE CHECKED
//    - With the session filter off, "first bar of the day" means the first bar
//      of ANY session the database holds for that date. On a database that
//      mixes pre-market with regular hours, that is the pre-market open, and
//      the comparison is still valid but is no longer about regular hours.
//    - Missing days are skipped, not treated as shifts: each day is compared
//      with the previous day that is actually present, so a Monday after a
//      holiday Friday is compared with the Thursday.
//    - The detector finds a change of stamp. It cannot tell you whether the
//      exchange moved, your computer moved, or the provider changed convention.
// ===========================================================================

MinShiftMinutes  = Param( "Report shifts of at least (minutes)", 1, 1, 240, 1 );
UseSessionFilter = ParamToggle( "Restrict to a session", "No|Yes", 0 );
SessionStart     = Param( "Session start (HHMMSS)",  93000, 0, 235959, 100 );
SessionEnd       = Param( "Session end (HHMMSS)",   155959, 0, 235959, 100 );

// Converts minutes-since-midnight into a readable HHMM number, so 570 prints
// as 930 rather than as a count nobody can read at a glance.
function MinutesToHHMM( Minutes )
{
    WholeHours = Int( Minutes / 60 );
    return WholeHours * 100 + ( Minutes - WholeHours * 60 );
}

// Always an ARRAY, never a scalar: Ref() and Cum() below need one bar per bar.
InScope = TimeNum() >= 0;
if( UseSessionFilter )
{
    InScope = TimeNum() >= SessionStart AND TimeNum() <= SessionEnd;
}

PreviousDate = Nz( Ref( DateNum(), -1 ), 0 );
NewDay       = DateNum() != PreviousDate;
DayEnds      = Nz( Ref( NewDay, 1 ), 1 );

// Position within the day, counted over in-scope bars only.
CumScope        = Cum( InScope );
ScopeBarInDay   = CumScope - Nz( ValueWhen( NewDay, Ref( CumScope, -1 ) ), 0 );
FirstScopeBar   = InScope AND ScopeBarInDay == 1;
LastScopeBar    = InScope AND ( DayEnds OR NOT Nz( Ref( InScope, 1 ), 0 ) );

MinutesOfDay = Hour() * 60 + Minute();

// ValueWhen with n = 2 reaches the second most recent occurrence, which is the
// previous trading day that actually has data. That is what makes holidays and
// suspended days harmless here.
FirstMinuteToday = ValueWhen( FirstScopeBar, MinutesOfDay, 1 );
FirstMinutePrev  = ValueWhen( FirstScopeBar, MinutesOfDay, 2 );
LastMinuteToday  = ValueWhen( LastScopeBar,  MinutesOfDay, 1 );
LastMinutePrev   = ValueWhen( LastScopeBar,  MinutesOfDay, 2 );

StartShift = FirstMinuteToday - FirstMinutePrev;
EndShift   = LastMinuteToday  - LastMinutePrev;

BarsToday = ValueWhen( LastScopeBar, ScopeBarInDay, 1 );
BarsPrev  = ValueWhen( LastScopeBar, ScopeBarInDay, 2 );

// The clock-change signature: the whole session moved by the same whole hour.
WholeSessionMoved = abs( StartShift ) == 60 AND StartShift == EndShift;

Filter = LastScopeBar AND abs( StartShift ) >= MinShiftMinutes;
SetOption( "NoDefaultColumns", True );

AddTextColumn( Name(), "Symbol", 1.0, colorDefault, colorDefault, 70 );
AddColumn( DateTime(), "Day", formatDateTime );
AddColumn( MinutesToHHMM( FirstMinuteToday ), "Start today (HHMM)", 1.0 );
AddColumn( MinutesToHHMM( FirstMinutePrev  ), "Start prev (HHMM)",  1.0 );
AddColumn( MinutesToHHMM( LastMinuteToday  ), "End today (HHMM)",   1.0 );
AddColumn( MinutesToHHMM( LastMinutePrev   ), "End prev (HHMM)",    1.0 );
AddColumn( StartShift, "Start shift (min)", 1.0, colorDefault,
           IIf( abs( StartShift ) == 60, colorRose, colorDefault ) );
AddColumn( EndShift,   "End shift (min)",   1.0 );
AddColumn( BarsToday,  "Bars today", 1.0 );
AddColumn( BarsPrev,   "Bars prev",  1.0 );

// 1 means "whole session moved by exactly one hour". Stated as a number so the
// finding survives being read aloud, printed in monochrome or exported to CSV.
AddColumn( WholeSessionMoved, "Whole session moved 1h (1=yes)", 1.0 );

SetSortColumns( 2 );

Download session-shift-detector.afl102 lines

How it works

The detector rests on one function used in an unusual way. ValueWhen( EXPRESSION, ARRAY, n ) returns the value the array held on the n-th most recent occurrence of the expression, and the documentation is explicit that n may be greater than one. Setting n = 1 gives today’s first-bar time; setting n = 2 gives the first-bar time of the previous day that actually has bars.

That second point is what makes the detector robust to the calendar. It never asks “what was yesterday”. It asks “what was the previous day that exists”, so a Monday after a holiday Friday is compared with the Thursday, and a suspended week is stepped over without producing a false alarm.

Everything else is the machinery from the previous two lessons. The day boundary is a change in DateNum(). The position within the day is Cum() of the in-scope flag minus whatever that cumulative total was at the end of the previous day. The last in-scope bar of a day is one where either the next bar starts a new day or the next bar is out of scope, which handles both the filtered and unfiltered cases in one expression.

The arithmetic is in minutes since midnight, for the reason the first lesson gave: the TimeNum() encoding packs the clock into decimal digits and cannot be subtracted. The small helper function turns minutes back into a readable four-digit HHMM number for display, so a session starting at 09:30 prints as 930 rather than as 570.

The last column states the finding as a number rather than as a colour: 1 where the whole session moved by exactly sixty minutes, 0 otherwise. A finding that survives being printed in monochrome, read aloud or exported to a spreadsheet is worth more than one that lives in a background tint.

Key functions

• ValueWhen( expr, array, n ) — with n = 2, the value at the second most recent occurrence. The mechanism behind “the previous day that exists”.
• Cum( array ) — a running total from the start of the array. Subtracting its value at the previous day boundary converts it into a within-day counter.
• ParamToggle( name, "No|Yes", default ) — a two-state parameter, used here to switch the session restriction on and off without editing the formula.
• Int() — truncation towards zero, used to split minutes into hours and minutes.

What you should see

=Expected result

On an intraday history covering a full year of an instrument whose market observes daylight saving, and whose stamps are expressed in a zone that observes it too: a small number of rows, of which two show a start shift and an end shift both equal to +60 or both to −60, with the last column reading 1. Everything else is short sessions, late starts and gaps. On a history stamped in a zone that does not observe the change, or on a market that does not, you should see no rows with the whole-session signature at all — which is information about your data, not a failure of the detector.

Test it

Run it with the minimum shift set to 1, and count the rows. Then run it with the minimum set to 59 and count again. The second run should keep the clock changes and drop most of the noise. If it drops everything, your history does not contain a transition, and you should widen the range before drawing any conclusion.

For a positive control, import a small synthetic file in which you deliberately shift one day’s bars by an hour, and confirm that the detector finds exactly that day with the signature column reading 1. A detector that has never been shown to fire is not yet a detector.

Common errors

• Running it on a range shorter than a year. Both transitions have to be inside the loaded range or there is nothing to find.
• Leaving the session filter off on a database that includes pre-market bars. The detector then compares pre-market opens, which is still valid, but the numbers are about the extended session rather than the one you were thinking of.
• Reading every reported day as a clock change. Short sessions produce an end shift with no start shift. Late starts produce a start shift with no end shift. Only both, equal, sixty, is the clock-change signature.
• Concluding the exchange moved. The detector reports that the stamps moved. Whether the exchange, the provider or your own configuration is responsible needs the timestamp audit from the first lesson and a look at the database time shift.

Extension

Store the detected transition dates as static variables with StaticVarSet() and have your other intraday formulas print a warning when the current bar falls within a few days of one. Part 23 covers static variables in the real-time context; the point here is that a finding is worth more when it is available to the code that needs it than when it is in a table you ran once.

Defensive time handling

Six habits. None is difficult, and together they remove most of this class of bug.

Anchor to the data, not to the clock. “The first bar inside the session” is a description of the data. “The bar stamped 09:30:00” is a description of a stamp. Under any displacement the first description still finds the right bar and the second one does not. The SessionBarNumber pattern from the previous lesson exists for this.

Count bars, not minutes, for anything positional. An opening range of six bars is six bars on a normal day and six bars on a short day. An opening range of thirty minutes is thirty minutes only while the stamps mean what you assumed.

Compare DateTime values with DateTimeDiff(). The documentation states that a DateTime is a bitset and that only equality and inequality are reliable with the normal operators. Every ordering comparison should go through DateTimeDiff(), which returns seconds and is positive when the first argument is later.

Never subtract TimeNum() values. Convert to minutes since midnight with Hour() * 60 + Minute(), or use DateTimeDiff() on DateTime values. TimeNum() is a packed decimal encoding and arithmetic on it is wrong in a way that produces plausible numbers.

Print the assumption. Every formula in this part reports the session start it actually found, or the number of bars that matched the stamp it needed. That single line converts a silent failure into a visible one, and it costs nothing.

Keep the session definition in one place. A Param() at the top of the formula, or better, an include file with one function per instrument class. When an exchange changes its hours — which happens, and not only for daylight saving — you want one edit rather than a search across every formula you have written. Part 11 covers building that library.

✦Tip

Add the timestamp audit and the shift detector to your database maintenance routine alongside the completeness checks from Part 19, and run them on a schedule rather than when something looks wrong. A fault you find in a monthly check is a curiosity. The same fault found because a scan stopped producing signals costs you the time you spent believing the scan.

Level A: seeing a transition without a live feed

▷Exercise

You do not need a subscription and you do not need to wait for October.

Manufacture one. Take a small intraday CSV of a few days. Duplicate it, and in the copy subtract one hour from every timestamp after a date you choose. Concatenate the two, import the result, and run session-shift-detector.afl against it. You have built a transition with a known date and a known direction, and you can confirm the detector reports exactly that day with the whole-session signature and nothing else.

Then break something with it. Run opening-range-fragile.afl from the challenge on the same database and watch the levels stop appearing at the date you chose. That is the whole failure mode, reproduced on your own machine, in a file you wrote, with no ambiguity about the cause.

Or use the history you have. Any stored intraday history spanning a year contains the real transitions if the market and the stamping zone observe them. The detector reads bars; it never touches a feed.

And for the alert side, note that this half of the problem is about your operating system’s clock rather than the data, so it is visible without any market data at all: schedule anything on your machine for a wall-clock time, and it will keep that wall-clock time across a transition while the market it was aimed at does not.

Recap

Clocks change twice a year, in different countries on different dates, in different directions between hemispheres, under rules that legislation revises. AmiBroker’s time shift is a single constant, which is the correct shape for a fixed offset and the wrong shape for one that takes two values a year.

A one-hour displacement produces either silence or nonsense from clock-anchored intraday code, and the nonsense is the dangerous case. The same displacement moves daily bar boundaries, session-anchored averages, alert timing and every intraday backtest that covers the affected weeks.

You can find it without a calendar: both ends of the session move, by the same amount, by exactly an hour. And you can largely avoid it by anchoring calculations to the data rather than to the clock, counting bars rather than minutes, and printing the assumption your formula is relying on so that the failure is loud.

The challenge that follows puts all of that to work on a formula that has already broken.

Check your understanding

Question 1. Why can the database Time shift setting not fix a daylight saving mismatch between two countries?

• Because it only accepts whole hours
• Because it is applied at import time and cannot be changed
• Because it is a single constant, and the offset takes two different values during the year
• Because it only affects charts and not Analysis windows

Show the answer and why

Answer: Because it is a single constant, and the offset takes two different values during the year

The shift is one number applied to the whole database — Status("timeshift") returns a single value in seconds. When two countries change on different dates, the true offset is one value for most of the year and another for the mismatched weeks, and no single constant is right for both.

Question 2. A formula computes an opening range between bars stamped 09:30:00 and 09:55:00. After a transition the stamps move one hour later. Which symptom is more dangerous?

• No signals at all, because the stamps no longer exist
• Signals computed from an hour of mid-morning trading called the opening range
• Both are equally dangerous
• Neither — AmiBroker reports an error

Show the answer and why

Answer: Signals computed from an hour of mid-morning trading called the opening range

Silence gets investigated. Plausible output does not. When both stamps still exist but land in the wrong part of the session, the formula produces a normal-looking list of signals derived from a range that has nothing to do with the open, and nothing on screen indicates a problem.

Question 3. Which pattern in the shift detector's output is the signature of a clock change rather than a short session?

• Start shift of 0 and end shift of −180
• Start shift of +60 and end shift of +60
• Start shift of +5 and end shift of −5
• A day with far fewer bars than usual

Show the answer and why

Answer: Start shift of +60 and end shift of +60

A clock change moves the whole session: both ends shift by the same amount in the same direction, and that amount is an hour. A short session leaves the start where it was and pulls the end forward. Small shifts in opposite directions are noise or a partial download.

Question 4. Which of these are safe against a whole-hour displacement of the bar stamps? Select all that apply.

• Defining the opening range as the first six bars of the session
• Defining the opening range as TimeNum() >= 93000 AND TimeNum() < 100000
• Comparing two DateTime values with DateTimeDiff()
• Computing elapsed minutes as TimeNum() - Ref(TimeNum(), -1)

Show the answer and why

Answer: Defining the opening range as the first six bars of the session, Comparing two DateTime values with DateTimeDiff()

Counting bars from the first in-session bar is a description of the data and moves with it. DateTimeDiff() is the documented way to order DateTime values and returns real seconds. The clock-time window breaks under displacement, and subtracting TimeNum() values is wrong regardless of any displacement because the encoding packs the clock into decimal digits.

Question 5. True or false: an instrument that is closed overnight in the stamping zone will show a repeated hour in its data when the clocks go back.

• True
• False

Show the answer and why

Answer: False

False. The repeated hour occurs while that market is shut, so no bars carry the duplicated stamps. The effect shows up in continuously traded instruments — overnight futures and currencies — where the transition happens mid-session. What the closed market does show is a session that has moved relative to your own clock.

Check my answersClear