Referencing Past Bars with Ref()

Almost every interesting question in technical analysis compares this bar with some earlier bar. Is today’s volume higher than yesterday’s? Has price closed above where it closed a fortnight ago? Did the average turn up? None of those can be asked with the tools of Part 8 alone, because every expression you have written so far compares arrays at the same bar.

Ref() is the function that breaks that restriction. It shifts an array along the time axis, so that a bar can read a value belonging to a different bar. It is the most-used function in this part, and it is also the one that quietly ruins more backtests than any other single construct in AFL — because of a sign convention that is the opposite of what most people assume.

By the end of this lesson you will be able to shift an array in either direction deliberately, and you will be able to look at somebody else’s formula and spot, from one character, whether it is reading data that had not happened yet.

The shift, stated exactly

The documented signature is:

Fragment — not a complete formula

Ref( ARRAY, period )

It returns an array of the same length as the input. The official function reference states the rule in one sentence: a positive period references “n” periods in the future; a negative period references “n” periods ago.

That sentence is worth reading twice, because it is the reverse of the way most people read it the first time. Ref( Close, -1 ) is the previous bar’s close. Ref( Close, 1 ) is the next bar’s close. The documentation’s own examples are unambiguous: ref( CLOSE, -14 ) is described as the closing price fourteen periods ago, and ref( C, 12 ) as the closing price twelve periods ahead — a case the page annotates, in its own words, as looking up the future.

Here is one bar-by-bar view, using the volume series that appears in the User’s Guide’s worked example of how AFL evaluates a formula:

Ref() shifts the array; the sign chooses the direction

A negative period pulls values forward from the past; a positive period pulls them back from the future. The final cell of the forward shift is discussed below.

Bar	0	1	2	3	4	5
Volume	8310	3021	5325	2834	1432	5666
Ref( Volume, -1 )each bar reads the bar to its left	Null	8310	3021	5325	2834	1432
Ref( Volume, 1 )each bar reads the bar to its right	3021	5325	2834	1432	5666	?

A negative period pulls values forward from the past; a positive period pulls them back from the future. The final cell of the forward shift is discussed below. Prices in this diagram are invented for the illustration. They are not market data and nothing should be inferred from them.

Read across the Ref( Volume, -1 ) row and compare each cell with the cell diagonally above and to the left. They match. That is all the function does: it copies the array, offset by a number of positions.

Why the sign feels wrong

The trap is that people read the argument as a distance backwards — “Ref of one means go back one” — when it is actually an offset applied to the bar index.

The mental model that makes the sign obvious is index arithmetic. If a bar’s position in the array is i, then:

Fragment — not a complete formula

// Conceptually, for every bar i:
//   Ref( array, period )[ i ]  is  array[ i + period ]

Adding a positive number to an index moves you towards the newer end of the array, which is the future. Subtracting moves you towards the older end, which is the past. Once you read Ref( C, -1 ) as C[i - 1] rather than as “one bar back”, the convention stops being something to memorise.

✕Common mistake

The symptom of getting the sign backwards is usually not an error message. It is a backtest whose results look implausibly good, or an indicator whose line appears to react to moves before they happen. Both are the same defect: a formula that reads a bar that had not printed yet.

What happens at the edges

An array has a first bar and a last bar. A shift asks some bars to read a position that does not exist, and AFL fills those positions with Null.

For the backward direction this is documented. The User’s Guide’s evaluation table shows Ref( Volume, -1 ) producing Null at bar 0, because there is no earlier bar to read. Generalising from the table: a shift of -n produces a run of n empty values at the left edge.

For the forward direction the User’s Guide does not state the behaviour in words. The symmetrical answer — n empty values at the right edge — is the obvious expectation, and it is what the diagram above marks with a question mark rather than a value.

∴Research note

Where the documentation is silent, this course says so rather than filling the gap from memory. Verify the forward edge on your own installation before you rely on it, using the procedure below. It takes about a minute and the answer is then a fact about your build rather than a belief.

Whatever fills the edges matters, because Null propagates. A Null in an arithmetic expression produces Null; so does a Null in a comparison; so does a Null in a logical AND. The User’s Guide’s table follows exactly that chain, ending with a Buy array that is Null — not zero — wherever either of its two inputs was empty. The leading empty run of a strategy is as long as the longest warm-up anywhere in its dependency chain, and Ref() contributes to that chain.

Measuring the edge yourself

Two functions turn the question into a number:

Fragment — not a complete formula

Shifted = Ref( Close, -14 );

LeadingEmpties  = NullCount( Shifted, 1 );   // mode 1: leading Nulls
TrailingEmpties = NullCount( Shifted, 2 );   // mode 2: trailing Nulls
MissingHere     = IsNull( Shifted );         // per-bar test, one value per bar

NullCount() returns a single number and is documented with four modes: 1 counts consecutive empties at the beginning, 2 at the end, 3 at both ends, 0 counts every empty value anywhere in the array. IsNull() is the per-bar test, and it is the only reliable one — comparing a value with Null produces Null, which is not true, so x == Null never works.

A laboratory for the sign convention

Goal

Settle the sign convention permanently by looking at it. Rather than memorising which direction is which, we put three columns side by side in an Exploration — this bar’s close, the close n bars earlier, and the close n bars later — and read the relationship straight off the screen.

The complete formula

Complete runnable AFL

ref-shift-laboratory.afl

// ref-shift-laboratory.afl
// Part 9 - Referencing Past Bars with Ref()
//
// An Exploration that puts the two directions of Ref() side by side so that the
// sign convention can be read off the screen instead of memorised:
//
//   Close             - the close of this bar
//   Ref( Close, -1 )  - the close one bar EARLIER
//   Ref( Close,  1 )  - the close one bar LATER
//
// Run it on a single symbol with the range set to "All quotations", then compare
// any row with the rows immediately above and below it. The -1 column repeats
// the row above; the +1 column repeats the row below. That is the whole
// convention, demonstrated rather than asserted.
//
// Assumptions:
//   - One symbol at a time. The output is one row per bar, so a watch list of
//     500 symbols would produce an unreadable report.
//   - The +1 column exists here only to be looked at. It reads a bar that had
//     not happened yet, so it must never appear in a trading rule.
//   - Nothing here is a trading signal: no Buy or Sell variable is set.

ShiftBars = 1;   // change to 5 or 14 and re-run to watch both edges grow

BarNo        = BarIndex();
EarlierClose = Ref( Close, -ShiftBars );
LaterClose   = Ref( Close,  ShiftBars );

// IsNull() is the only reliable Null test. Comparing a value with Null yields
// Null, which is not "true", so "x == Null" never works.
EarlierMissing = IsNull( EarlierClose );
LaterMissing   = IsNull( LaterClose );

// How long is the unusable run at each end of the array? NullCount() returns a
// single number: mode 1 counts leading Nulls, mode 2 counts trailing ones.
LeadingNulls  = NullCount( EarlierClose, 1 );
TrailingNulls = NullCount( LaterClose,   2 );

Filter = 1;   // report every bar, so the shift is visible row by row

AddColumn( BarNo,          "Bar index",                    1.0 );
AddColumn( Close,          "Close (this bar)",             1.4 );
AddColumn( EarlierClose,   "Ref(C,-n): n bars earlier",    1.4 );
AddColumn( LaterClose,     "Ref(C,+n): n bars later",      1.4 );
AddColumn( EarlierMissing, "Backward shift is Null (1=yes)", 1.0 );
AddColumn( LaterMissing,   "Forward shift is Null (1=yes)",  1.0 );
AddColumn( LeadingNulls,   "Leading Nulls in whole array",   1.0 );
AddColumn( TrailingNulls,  "Trailing Nulls in whole array",  1.0 );

Download ref-shift-laboratory.afl48 lines

How it works

The formula does three things. It builds the two shifted arrays with Ref() using the same magnitude and opposite signs, so the only difference between the columns is direction. It then tests each shifted array for emptiness in two ways: IsNull() gives a per-bar answer, and NullCount() gives the length of the empty run at each end of the whole array. Finally it sets Filter = 1, which is the Exploration’s way of saying “report every bar” — the point here is to see the rows next to each other, so nothing is filtered out.

ShiftBars is a single named constant at the top rather than a literal buried inside two calls, so that changing 1 to 14 changes both directions at once and keeps them comparable.

Key functions

• Ref( ARRAY, period ) — the shift itself. Negative goes back, positive goes forward.
• BarIndex() — the zero-based bar number as an array. It is documented as equivalent to Cum(1) - 1 and as being much faster in indicators. Including it as a column makes the row-to-row comparison unambiguous.
• IsNull( x ) — true where a value is empty. Documented as a synonym of IsEmpty(), and the official page recommends IsNull in new formulas.
• NullCount( array, mode ) — the length of an empty run, as one number.

Expected result

Open the Analysis window, select a single symbol, set the range to all quotations, and press Explore. You should get one row per bar with eight columns.

=Expected result

On any row in the middle of the report, the “n bars earlier” cell repeats the close from the row above it, and the “n bars later” cell repeats the close from the row below it. The first row has an empty backward cell. The last row has an empty forward cell. With ShiftBars = 1 the leading and trailing counts should both read 1; set ShiftBars = 5 and both should read 5.

Test it

The check that would catch the formula being wrong is a comparison you can do by eye and then by arithmetic. Pick any row k in the middle of the report. Its “earlier” cell should be identical to the close printed on row k−1, and its “later” cell identical to the close on row k+1. If the two columns are the other way round, the shift is not doing what this lesson claims and you should stop and find out why before writing anything that depends on it.

Then change ShiftBars to 5 and re-run. Every row should now reach five rows up and five rows down, and the two NullCount columns should both report 5.

Common errors

• Running it over a watch list. Filter = 1 reports every bar of every symbol. On five hundred symbols that is a report nobody can read and a slow wait. Select one symbol.
• Expecting the forward column to be useful. It is here to be looked at, not used. The moment it appears in a rule, the rule is untradeable.
• Reading an empty cell as zero. An empty cell is Null. It is not zero, and arithmetic on it yields Null rather than a number.

Extension

Add a column for Close - Ref( Close, -ShiftBars ), the change over the shift window in points, and another for the same quantity as a percentage. That is the rate-of-change calculation the official Ref page gives as its worked example, and it is the shape of every momentum measure you will meet later in the course.

The plus sign that invalidates a backtest

Consider two rules that differ by one character:

Fragment — not a complete formula

// Rule A - tradeable
Buy = Close > Ref( Close, -20 );

// Rule B - not tradeable, and it will not tell you
Buy = Close > Ref( Close, 20 );

Rule A says “the close is above where it was twenty bars ago”. Every input was knowable at the moment the bar closed. Rule B says “the close is below where it will be twenty bars from now” — the comparison is with a bar that had not happened. AmiBroker will run it. The backtester will produce a report. The report will look extraordinary, because the rule buys only when it already knows price rises afterwards.

This is look-ahead bias: using information in a decision that was not available when the decision had to be made. Part 30 gives it a full lesson, including the other ways it creeps in — the trade-delay settings, LastValue(), multi-timeframe expansion, survivorship in the universe. What matters here is the narrow, mechanical case: a positive period in Ref() inside Buy, Sell, Short, Cover, a Filter, or anything they depend on, is look-ahead bias, and the documentation says as much in plain words.

⚠Warning

Look-ahead bias does not produce a warning, an error or a suspicious-looking chart. It produces an unusually good result, which is precisely the kind of result nobody feels like interrogating. Grep your own formulas for Ref( and check every sign before you believe a backtest.

There is a second, subtler version of the same mistake that does not involve a positive sign at all. If a rule uses today’s close to decide a trade, and the backtester is configured to fill that trade at today’s close, then the decision and the execution happen at the same instant — which is not something you could do in practice. That is a trade-delay question rather than a Ref() question, and Part 28 covers it where it belongs. Mentioning it now is worth the detour, because “no positive Ref()” is a necessary condition for a tradeable rule and not a sufficient one.

When a positive period is legitimate

Positive Ref() is not forbidden. It is forbidden in decisions. There are two honest uses.

The first is drawing. Shifting a plotted line to the right so that it does not sit on top of the price, or projecting a level forward past the last bar, is a presentation choice with no decision attached. Plot() and PlotShapes() both accept their own XShift argument for exactly this, which is usually cleaner than shifting the data itself.

The second is measurement — and it is the important one, because it is how this course tests claims. To ask “what happened in the twenty bars after this event?” you must read twenty bars into the future. That is legitimate precisely because the answer is a description of recorded history and is never fed back into a rule. Every reality-check page in this course rests on this technique.

Goal

Measure the distribution of what followed an event, and — just as important — measure the same thing from every bar, so that the first number has something to be compared with.

The complete formula

Complete runnable AFL

forward-return-study.afl

// forward-return-study.afl
// Part 9 - Referencing Past Bars with Ref()
//
// The one legitimate use of a POSITIVE Ref(): measuring what happened after an
// event, in a study whose output describes recorded history and is never a
// signal.
//
// The event studied here is deliberately ordinary - the close crossing above its
// 50-bar simple moving average. The interesting question is not "what happened
// after the event" on its own, but "how did that compare with the same
// measurement taken from every bar?" Set ShowAllBars to 1 to produce the
// comparison set. Part 7 and Part 30 develop the method properly; this formula
// shows only the array mechanics.
//
// Assumptions:
//   - A forward return is unknowable at the time of the event. This is a
//     post-hoc measurement of history, not a rule anyone could have traded.
//   - The final HorizonBars bars have no forward value at all. They are excluded
//     rather than quietly counted as zero.
//   - No costs, no slippage, no position sizing, no compounding. The numbers
//     describe price changes, not the result of trading them.
//   - Percentage changes computed on unadjusted data are wrong across splits.
//     Use an adjusted series, or expect nonsense on split dates (Part 2).

HorizonBars = 20;
MaPeriod    = 50;
ShowAllBars = 0;   // 0 = event bars only.  1 = every bar, for the baseline run

Average = MA( Close, MaPeriod );
Trigger = Cross( Close, Average );

// A POSITIVE period reads forward in time. That is exactly what a forward-return
// measurement needs, and exactly what a trading rule must never contain.
FutureClose = Ref( Close, HorizonBars );
ForwardPct  = 100 * ( FutureClose - Close ) / Close;

// The last HorizonBars bars cannot have a forward value, so they are not
// measurable and must not be reported as though they were.
Measurable = NOT IsNull( FutureClose );

Filter = Measurable AND ( ShowAllBars OR Trigger );

AddColumn( BarIndex(),  "Bar index",                                       1.0 );
AddColumn( Trigger,     "Is an event bar (1=yes)",                         1.0 );
AddColumn( Close,       "Close on this bar",                               1.4 );
AddColumn( FutureClose, StrFormat( "Close %g bars later", HorizonBars ),    1.4 );
AddColumn( ForwardPct,  StrFormat( "Change over %g bars %%", HorizonBars ), 1.2 );

Download forward-return-study.afl47 lines

How it works

The event is a crossing of the close above a fifty-bar average; that choice is arbitrary and is meant to be replaced. Ref( Close, HorizonBars ) reads the close twenty bars later, and ForwardPct expresses the difference as a percentage of the price at the event.

The two remaining ideas are what separate a study from an anecdote. Measurable excludes the final twenty bars, where no forward value exists — reporting those rows would silently mix “no data” into the results. ShowAllBars switches the Filter between event bars only and every measurable bar, so that the same formula produces both the sample and its comparison set. Without that second run a figure like “+1.8% over twenty bars” is unreadable: if the instrument rose about 1.8% in a typical twenty-bar stretch anyway, the event told you nothing.

Key functions

• Ref( ARRAY, period ) with a positive period — used here as a measurement instrument and confined to the reporting columns.
• Cross( ARRAY1, ARRAY2 ) — the event detector, introduced properly two lessons from now.
• NOT, AND, OR — AFL’s documented logical operators, applied element by element across the arrays.
• StrFormat( formatstr, ... ) — builds the column captions so that changing HorizonBars relabels the report. Use %g for numbers; the documentation notes that %d does not work, because AFL has no integer type.

Expected result

=Expected result

With ShowAllBars = 0 you get one row per crossing, each with the close at the event, the close twenty bars later and the percentage change. With ShowAllBars = 1 you get a row for nearly every bar in the history. Export both to CSV and compare the two distributions — not just the averages, which a handful of extreme values can dominate.

Test it

Set HorizonBars = 0 temporarily. Ref( Close, 0 ) should return the array unshifted, so the “later” column should equal the close column on every row and the percentage change should be zero throughout. If it is not, your understanding of the shift is wrong somewhere and this is the cheapest place to find out.

Then set HorizonBars back to 20 and check the tail: the last twenty bars of the symbol should not appear in the report at all.

Common errors

• Letting the forward column leak into a rule. If you later extend this formula, keep FutureClose out of anything that resembles a signal. The safest habit is to keep measurement formulas and rule formulas in separate files.
• Skipping the baseline run. A forward-return number without a comparison set is not evidence. It is a number.
• Running it on unadjusted data. A split shows up as a huge negative forward return that has nothing to do with the event. Part 2 covers why.
• Treating the average as the finding. Report the spread as well; Part 7 and Part 30 develop this properly.

Extension

Add a column for the lowest low over the same forward window using LLV( Ref( Low, HorizonBars ), HorizonBars ), which measures the worst excursion during the horizon rather than only its end point. Two events with identical twenty-bar returns can have entirely different paths, and the path is what a trader actually has to sit through.

Periods that vary bar by bar

Ref() accepts a period that is itself an array, so the shift can differ on every bar. The User’s Guide lists it among the functions that take a time-variant period argument.

Fragment — not a complete formula

// Shift back to the bar on which the condition last became true.
Age     = BarsSince( Condition );
AtEvent = Ref( Close, -Age );

The minus sign in front of Age is doing critical work. BarsSince() returns a positive count, and a positive period reads forward. Feeding an unnegated age straight into Ref() produces a formula that looks past the current bar by however long ago the last event was — a spectacular, silent look-ahead.

In practice ValueWhen() expresses this idea more directly and is covered two lessons from now. Variable-period Ref() is worth knowing mainly so that you can read other people’s code without being surprised by it.

Recap

Ref() shifts an array along the time axis. The sign of the second argument is an offset applied to the bar index, so a negative period reads the past and a positive period reads the future — the official documentation states this explicitly and calls the positive case looking up the future.

Shifting creates empty values at one edge of the array, and those empties propagate through arithmetic, comparisons and logical operators until something converts them. The backward edge is documented; the forward edge is not, and this course would rather you verified it than took its word.

A positive period inside any decision is look-ahead bias. A positive period inside a measurement, kept firmly out of the decisions, is the only way to ask what happened next — and asking that question honestly, with a baseline to compare against, is most of what the research half of this course consists of.

Check your understanding

Question 1. What does the highlighted expression return on the bar for 3 June?

Yesterday = Ref( Close, -1 );

• The close on 4 June
• The close on 2 June
• The average close of the last bar
• A single number, not an array

Show the answer and why

Answer: The close on 2 June

A negative period reads backwards, so the value is the close of the previous bar. It is still an array: one shifted value for every bar on the chart.

Question 2. Which of these belong in a tradeable Buy rule? Select all that apply.

• Ref( Close, -5 )
• Ref( Close, 5 )
• Ref( HHV( High, 20 ), -1 )
• Ref( Close, -BarsSince( Condition ) )
• Ref( Close, BarsSince( Condition ) )

Show the answer and why

Answer: Ref( Close, -5 ), Ref( HHV( High, 20 ), -1 ), Ref( Close, -BarsSince( Condition ) )

Any positive period reads a bar that had not printed. BarsSince() returns a positive count, so it must be negated before being used as a period - the unnegated form looks forward by a varying amount, which is the hardest kind of look-ahead to notice.

Question 3. A formula contains Diff = Close - Ref( Close, -14 ). What is the value of Diff on bar 3 of the chart?

• 0
• The close on bar 3
• Null
• The close on bar 17

Show the answer and why

Answer: Null

The shift needs a bar 14 positions earlier, which does not exist before bar 14. That produces Null, and arithmetic involving Null yields Null - so the first fourteen bars of Diff are empty, not zero.

Question 4. True or false: a formula with no positive Ref() anywhere in it cannot suffer from look-ahead bias.

• True
• False

Show the answer and why

Answer: False

False. A positive Ref() is one route to look-ahead bias, not the only one. Deciding on the close and filling at the same close, LastValue(), careless multi-timeframe expansion and a survivorship-filtered universe all produce the same defect without a single plus sign. Part 30 covers the full list.

Question 5. You want the report to exclude the final twenty bars, where a twenty-bar forward return cannot exist. Which test does that?

FutureClose = Ref( Close, 20 );

• FutureClose != Null
• FutureClose > 0
• NOT IsNull( FutureClose )
• NullCount( FutureClose, 2 )

Show the answer and why

Answer: NOT IsNull( FutureClose )

Comparing anything with Null yields Null rather than true or false, so the first option silently fails. NullCount returns one number for the whole array, not a per-bar test. IsNull() is the documented per-bar emptiness test.

Check my answersClear