Cleaning Signals: ExRem, ExRemSpan and Flip

The previous lesson ended with a rule that produced fourteen hundred buy signals where it should have produced forty-six. The standard response, reached for within about thirty seconds by most people who hit it, is:

Fragment — not a complete formula

Buy  = ExRem( Buy,  Sell );
Sell = ExRem( Sell, Buy );

Those two lines are the official example on the ExRem page, and they usually work. This lesson is about understanding them precisely enough to know when they are the right answer, and honest enough to recognise the cases where applying them hides a defect instead of fixing one.

What ExRem actually does

Fragment — not a complete formula

exrem( ARRAY1, ARRAY2 )

The documented behaviour, stated exactly: it returns 1 on the first occurrence of a true signal in ARRAY1, then returns 0 for every subsequent true signal in ARRAY1 until ARRAY2 becomes true.

Three things follow from that definition, and all three matter.

It keeps the first signal of each run — not the best one, not the strongest, not a confirmed one. First.

It needs ARRAY2 to reset it. Without a true value in the second array, the first signal is the only signal that will ever pass, for the rest of the data.

It discards information. After ExRem, the array no longer records whether the underlying condition was still true on bar 900. If you need that later, keep the raw condition in a separate variable.

ExRem thinning a pair of state-shaped rules, and Flip putting the state back

ExRem keeps the first true of each run and suppresses the rest until the opposing array fires. Flip reverses the process exactly, which the official Flip page states in as many words.

Bar	0	1	2	3	4	5	6	7	8	9
RawBuy (a state)5 true bars	0	1	1	1	0	0	0	1	1	0
RawSell (a state)5 true bars	1	0	0	0	1	1	1	0	0	1
CleanBuy = ExRem( RawBuy, RawSell )2 true bars	0	1	0	0	0	0	0	1	0	0
CleanSell = ExRem( RawSell, CleanBuy )3 true bars	1	0	0	0	1	0	0	0	0	1
Flip( CleanBuy, CleanSell )identical to RawBuy here	0	1	1	1	0	0	0	1	1	0

ExRem keeps the first true of each run and suppresses the rest until the opposing array fires. Flip reverses the process exactly, which the official Flip page states in as many words.

The order of the two lines is part of the idiom

Look carefully at the second line of the official example:

Fragment — not a complete formula

buy  = ExRem( buy, sell );
sell = ExRem( sell, buy );

By the time the second line runs, buy has already been overwritten by the thinned version. The sell cleaning therefore resets against the cleaned buy array, not the raw one. That is deliberate, and it is why the two lines produce a neatly alternating buy-sell-buy-sell sequence.

Reversing the two lines, or computing both from copies of the originals, gives different arrays. Neither is illegal; both are different from what the documentation’s idiom produces. If you write it any way other than the documented one, write a comment saying why.

Flip is the documented inverse

Fragment — not a complete formula

flip( ARRAY1, ARRAY2 )

A latch: 1 from the first true in ARRAY1 until a true occurs in ARRAY2, which resets it to 0 until the next true in ARRAY1. The Flip page’s own example is the round trip:

Fragment — not a complete formula

buy = ExRem( buy, sell );
buy = Flip( buy, sell );   // multiple signals are back again

That pairing is worth holding on to, because it tells you exactly what the two functions are: ExRem collapses a state into its leading edges, Flip expands leading edges back into a state. Neither adds information. They convert between two representations of the same thing, and the conversion is lossy in the ExRem direction.

Flip is the tool for anything that needs to know “am I notionally in a position?” — shading a chart, computing exposure, gating a second rule, or counting bars in trade. Two properties deserve attention:

• It is sticky. If the reset array never becomes true, the state stays on to the end of the data. That is correct, and it is the argument for putting a fallback exit — a stop, a time limit — into the reset array rather than relying on the primary exit rule alone.
• Simultaneous set and reset is undocumented. The official page does not say which wins when both arrays are true on the same bar. If your two conditions can coincide, test it rather than assuming.

ExRemSpan, and what the documentation says about it

Fragment — not a complete formula

exremspan( ARRAY1, numbars )

The first non-zero bar passes through; all subsequent non-zero bars are suppressed until numbars bars have elapsed since the initial signal, after which a new signal may pass. There is no second array — nothing resets it except the passage of time.

The official page then says something a course has an obligation to repeat:

This function is marked as obsolete. To implement N-bar stop you should use ApplyStop function instead.

!Important

Any material presenting ExRemSpan() as the way to build an N-bar exit is contradicting the official documentation. It is a signal filter: it stops new entry signals from being reported for a while. It does not close a position, it does not affect position management, and it is not an exit rule. Learn it so you can read older code; use ApplyStop() — covered in Part 28 — for time-based exits.

There is also no worked example on the official page: the EXAMPLE field is empty. That is worth knowing before you go looking for one.

What the backtester does with unclean signals

This is the part that changes how most people think about ExRem, and it is documented rather than folklore.

AmiBroker’s default backtest mode already removes redundant entry signals. The portfolio-backtesting chapter of the User’s Guide describes it directly: buy and sell signals are matched into trades, and an entry signal arriving after an entry but before the matching exit is ignored. The chapter then states that this process of removing excess signals is the same as the ExRem() function provides.

The Equity() page says the same thing from the other side: Equity( 1 ) updates the buy, sell, short and cover arrays so that all redundant signals are removed “exactly as it is done internally by the backtester”.

So for a plain, default-mode backtest, applying ExRem yourself typically changes nothing at all. Where it does matter is everywhere else:

• Chart arrows. PlotShapes() marks every true bar. Without thinning you get a band instead of a marker.
• Scans and Explorations. Each true bar becomes a report row when the range covers more than one quotation.
• Alerts. One notification per true bar, for the duration of the state.
• Non-default backtest modes. And here the relationship reverses — see below.

⚠Warning

SetBacktestMode( backtestRegularRaw ) and backtestRegularRawMulti exist precisely so that repeated entry signals are not discarded: the raw mode acts on any entry signal that passes ranking and cash checks, and the multi mode opens several positions on the same symbol when the entry signal stays true. If you apply ExRem to your arrays and then select one of those modes, you have already thrown away the signals the mode was chosen to use. Part 28 covers the modes; the point here is that cleaning is not universally harmless.

When ExRem is the wrong fix

The honest version of this lesson is that ExRem is a presentation fix that is often mistaken for a logic fix. Five situations where reaching for it makes things worse:

The rule is a state and should have been an event. If you wrote Buy = Close > MA( Close, 50 ) and meant “when price moves above the average”, ExRem will tidy the output and leave the rule saying the wrong thing. The two are not equivalent: Cross() fires when the relationship changes, while ExRem-of-a-state fires on the first bar of a run that follows a true sell signal — which, if the sell rule is also wrong or rare, can be a completely different bar. Fix the rule.

You wanted a different signal than the first. ExRem keeps the earliest true bar in each run. If your intention was “the first bar with confirming volume”, or “the strongest setup this week”, ExRem silently substitutes a different selection rule for yours. Express the selection you actually want, then thin.

The reset never fires. ExRem( Buy, Sell ) where Sell is rare, or where the exit is handled entirely by ApplyStop() and so never appears in the Sell array, produces exactly one entry signal in the whole history. The symptom — one trade per symbol — looks like a data problem and is not.

The underlying rate is the real problem. A condition true on forty per cent of bars is not a trigger, and thinning it produces a tidy-looking array built on a meaningless rule. The occurrence-rate measurement from earlier in this part is the check that catches this; it takes a minute and it asks the right question.

Duplicate alerts have a different cause. Repeated alerts during a live session are often produced by repeat scanning re-evaluating the same bar many times, not by a state-shaped array at all. Part 25 deals with alert suppression properly, and ExRem is not the tool for it.

✦Tip

A decision procedure that takes about two minutes.

1. Measure the array: how many bars is it true, and what is its longest unbroken run? (The last lesson gave the three lines that answer this.)
2. If the longest run is 1, the array is already event-shaped. Do not thin it — find out why you thought you needed to.
3. If the longest run is large, ask whether the rule should have been an event. If yes, rewrite the rule with Cross() or an explicit rising edge, and stop.
4. If the rule is genuinely a state and you want one signal per run, thin it — and check that the resetting array actually fires.

Verifying that a signal array is clean

Goal

Replace “I applied ExRem, so it should be fine” with a number. The formula counts raw signals against cleaned signals, rebuilds the state with Flip, and reports how far the rebuild disagrees with the original — so you can see what was removed and confirm that nothing else changed.

The complete formula

Complete runnable AFL

signal-cleaning-workbench.afl

// signal-cleaning-workbench.afl
// Part 9 - Cleaning Signals: ExRem, ExRemSpan and Flip
//
// Counts raw signals against cleaned signals, so that what ExRem() removed is a
// number on the screen rather than a belief. It also rebuilds the state with
// Flip() and reports how far the rebuilt state differs from the original - the
// cheapest available check that the cleaning did what you expected.
//
// One row per symbol. Run it over a watch list to find the instruments where a
// rule is firing hundreds of times instead of a handful.
//
// Assumptions:
//   - The rules below are state-shaped on purpose. They are the kind of rule
//     that produces signal storms and sends people reaching for ExRem.
//   - The second ExRem line consumes the ALREADY THINNED first array. That
//     ordering is the documented idiom; swapping the two lines changes the
//     result, so do not reorder them casually.
//   - Counts cover the bars delivered to this run.
//   - Cleaning changes what a chart and a scan report. It does not by itself
//     change what the portfolio backtester does in its default mode, which
//     already discards redundant same-direction signals.

SetBarsRequired( sbrAll );

MaPeriod = 50;
Average  = MA( Close, MaPeriod );

// ---------------------------------------------------------------------------
// Deliberately state-shaped rules: true on EVERY bar the condition holds
// ---------------------------------------------------------------------------

RawBuy  = Close > Average;
RawSell = Close < Average;

// ---------------------------------------------------------------------------
// Cleaning
// ---------------------------------------------------------------------------

// ExRem( a, b ): keep the first true in a, then suppress further trues in a
// until b is true. Note the second line uses CleanBuy, not RawBuy.
CleanBuy  = ExRem( RawBuy,  RawSell );
CleanSell = ExRem( RawSell, CleanBuy );

// Flip( set, reset ): latch on from the first clean entry until a clean exit.
// This puts back the state that ExRem discarded.
Holding = Flip( CleanBuy, CleanSell );

// ---------------------------------------------------------------------------
// Counting
// ---------------------------------------------------------------------------

BarsInRange    = Cum( 1 );
RawBuyCount    = Cum( IsTrue( RawBuy ) );
CleanBuyCount  = Cum( IsTrue( CleanBuy ) );
RawSellCount   = Cum( IsTrue( RawSell ) );
CleanSellCount = Cum( IsTrue( CleanSell ) );
HoldingBars    = Cum( IsTrue( Holding ) );

// Where the rebuilt state and the original state disagree. Expect a small
// number, concentrated at the left edge before the first signal has occurred.
RebuildGap = Cum( IsTrue( Holding ) != IsTrue( RawBuy ) );

Filter = Status( "lastbarinrange" );

AddColumn( BarsInRange,    "Bars in range",                          1.0 );
AddColumn( RawBuyCount,    "Raw buy trues (= bars the state held)",  1.0 );
AddColumn( CleanBuyCount,  "Buy signals after ExRem",                1.0 );
AddColumn( RawSellCount,   "Raw sell trues",                         1.0 );
AddColumn( CleanSellCount, "Sell signals after ExRem",               1.0 );
AddColumn( HoldingBars,    "Bars inside the Flip state",             1.0 );
AddColumn( RebuildGap,     "Bars where the rebuild disagrees",       1.0 );
AddColumn( 100 * HoldingBars / BarsInRange, "Time in state, % of bars", 1.2 );

Download signal-cleaning-workbench.afl72 lines

How it works

The rules at the top are deliberately state-shaped: price above and below a fifty-bar average. That guarantees the signal storm the lesson is about, and it makes the counts easy to interpret — for a state-shaped rule, the number of “raw signals” and the number of bars the state was true are the same number, which is itself the diagnosis.

The cleaning block follows the documented idiom exactly, including the ordering that makes the second call consume the already-thinned first array. The Flip call rebuilds the state, and RebuildGap counts the bars where the rebuild and the original state disagree — the direct check that the ExRem and Flip pair round-tripped.

Filter = Status( "lastbarinrange" ) gives one row per symbol, and SetBarsRequired( sbrAll ) makes the cumulative counts cover the whole delivered history rather than a QuickAFL slice.

Key functions

• ExRem( ARRAY1, ARRAY2 ) — keeps the first true of each run, resets on the second array.
• Flip( ARRAY1, ARRAY2 ) — the latch that reverses the process.
• IsTrue( ARRAY ) — maps Null to 0 before counting, so warm-up bars cannot poison a total.
• Cum( ARRAY ) — running totals, read at the last bar via the filter.
• Status( "lastbarinrange" ) — one row per symbol.

Expected result

=Expected result

On a few years of daily data, the raw buy count should be in the hundreds or low thousands, and the count after ExRem in the tens. The rebuild disagreement should be small — typically a handful of bars at the very start, before the first clean signal has occurred. The “time in state” percentage should be somewhere near half for a symbol with no strong long-term drift, and noticeably above half for one that trended upwards over the period.

Test it

Two checks, both quick.

First, the round trip. If RebuildGap is large — hundreds of bars rather than a handful — the ExRem and Flip pair did not reverse each other, which usually means the two rules can be true on the same bar. Add a column for Cum( IsTrue( RawBuy ) AND IsTrue( RawSell ) ) and see whether it is non-zero.

Second, the reset. Temporarily replace RawSell with Close < 0, which is never true on a price series. The cleaned buy count should collapse to 1 for every symbol — a vivid demonstration of what happens when the resetting array never fires, and the shape of a bug that produces exactly one trade per symbol.

Common errors

• Cleaning arrays that were already events. The counts will be almost identical before and after, which is the signal that the thinning was unnecessary.
• Reversing the two ExRem lines. The output changes. If you need a different ordering, comment the reason.
• Reading the “time in state” figure as an exposure estimate. It is the fraction of bars the condition held, not the fraction of capital deployed. Real exposure comes from the backtest report, in Part 29.

Extension

Add a column comparing ExRem( RawBuy, RawSell ) with ExRemSpan( RawBuy, 20 ) to see how differently a reset-based filter and a time-based filter thin the same array. Then read the note on the ExRemSpan page again and decide, for a rule of your own, whether what you want is really an entry filter or really an exit.

Recap

ExRem( a, b ) keeps the first true value of each run in a and suppresses the rest until b fires. Flip( a, b ) is the documented inverse: it turns a pair of edges back into a continuous state. ExRemSpan() thins by elapsed bars instead of by a reset array, and its own page marks it obsolete and points to ApplyStop() for time-based exits.

The backtester’s default mode already discards redundant entry signals, in the same way ExRem does. Cleaning therefore matters most for charts, scans, explorations and alerts — and it actively works against the raw backtest modes, which exist to use the signals ExRem throws away.

Above all, ExRem tidies output. It does not correct a rule that says the wrong thing. Measure the array first; if the longest run of true bars is long and the rule was meant to fire on a moment, the fix belongs in the rule.

Check your understanding

Question 1. What does ExRem( Buy, Sell ) return on the third consecutive bar of a run of true Buy values, with no Sell in between?

• 1
• 0
• Null
• The same as Buy

Show the answer and why

Answer: 0

Only the first true value of the run passes. Everything after it is suppressed until the second array becomes true, which is the documented behaviour and the whole purpose of the function.

Question 2. A system uses ApplyStop() for all exits and never assigns anything to Sell. What happens to ExRem( Buy, Sell )?

• Nothing - it works normally
• It returns exactly one signal per symbol, because the resetting array never becomes true
• It raises an error
• It returns the raw Buy array unchanged

Show the answer and why

Answer: It returns exactly one signal per symbol, because the resetting array never becomes true

ExRem needs a true value in the second array to re-arm. With a Sell array that is never true, the first signal passes and everything after it is suppressed for the rest of the history - producing one trade per symbol and a very confusing report.

Question 3. Which statement about the backtester is correct?

• ExRem must be applied or the backtester will open a new position on every true bar
• The default backtest mode already ignores entry signals that arrive while a position is open
• The backtester ignores ExRem entirely
• ExRem is required for ApplyStop to work

Show the answer and why

Answer: The default backtest mode already ignores entry signals that arrive while a position is open

The User's Guide states that in the default regular mode, entry signals arriving after an entry and before the matching exit are ignored, by the same process ExRem performs. Cleaning matters for charts, scans and alerts - and it removes the signals the raw modes are chosen to act on.

Question 4. You want an array that is 1 for every bar a position would be open, starting from a cleaned entry signal. Which is right?

• ExRem( Buy, Sell )
• ExRemSpan( Buy, 20 )
• Flip( Buy, Sell )
• Cum( Buy )

Show the answer and why

Answer: Flip( Buy, Sell )

Flip is the latch: on from the first entry until an exit. ExRem does the opposite job, ExRemSpan thins by elapsed bars, and Cum produces a rising count rather than a Boolean state.

Question 5. A rule fires on 38% of all bars. After ExRem it produces a tidy 40 signals over ten years. What is the right conclusion?

• The rule is now clean and ready to backtest
• ExRem fixed the problem
• The underlying condition is a state that is true on more than a third of all bars, and thinning it has hidden that rather than addressed it
• The occurrence rate is irrelevant once ExRem is applied

Show the answer and why

Answer: The underlying condition is a state that is true on more than a third of all bars, and thinning it has hidden that rather than addressed it

A condition true on 38% of bars describes the market rather than selecting a moment in it. ExRem produces a tidy array, but the selection rule is now "the first bar of each run following a sell", which is unlikely to be what anyone intended.

Question 6. True or false: the official documentation recommends ExRemSpan() for building an N-bar exit.

• True
• False

Show the answer and why

Answer: False

False. The ExRemSpan page marks the function obsolete and says to use ApplyStop() for an N-bar stop. ExRemSpan filters entry signals; it does not close positions.

Check my answersClear