Skip to content
Level 1 · Chart ReaderReality checkPart 07 · page 5 of 540 min
40Minutes
15AFL functions
10Sources
StandardRequires
AFL functions taught here15

Reality Check: Can We Test Whether a Pattern Predicted Anything?

Everything in this part has been building to a single procedure. You are going to take one named candlestick pattern, turn it into arithmetic with every threshold visible, measure what followed it on your own data, and compare that with two different benchmarks drawn from the same sample. Then you are going to read the answer without deciding in advance what you want it to be.

Forty minutes, no paid data, no Professional edition. The output is a table you could hand to someone who disagrees with you.

The claim, stated so that it could be wrong

Section titled “The claim, stated so that it could be wrong”

That sentence contains four things that have to be separated before anything can be measured: a geometry, a context (“after a decline”), a mechanism (“buyers taking control”) and an outcome claim (“followed by a rise more often than usual”). The mechanism is not testable from bar data, as the first lesson explained. The other three are.

Written as a hypothesis that could fail:

Over the instruments and period defined below, the mean percentage change of the close over the following 10 bars, measured on bars where the pattern occurred, is higher than the same quantity measured on all bars that met the same context condition — by an amount larger than the sampling error of the difference.

Note what that formulation excludes. It does not say the pattern will work in future. It does not say a system built on it would be profitable. It makes one comparison, on one sample, and it can come out against the claim.

Here is the whole geometry, with every threshold named:

Condition Written as
Today closed up Close > Open
Yesterday closed down PrevClose < PrevOpen
Today’s body covers the bottom of yesterday’s BodyBottom <= Min( PrevOpen, PrevClose )
Today’s body covers the top of yesterday’s BodyTop >= Max( PrevOpen, PrevClose )
Today’s body is not trivially small BodyPct >= MinBodyPct

Five conditions, one free number: MinBodyPct, defaulting to 30% of the bar’s own range. Ties count as covering, because the comparisons use <= and >=. That is a choice and it is written down.

This is the body definition of engulfing from the second lesson, not the range definition. Choosing between them is itself a variant, and picking one after seeing which gave the better result would be exactly the multiple-comparison mistake. Pick first.

Decision 2: the forward measurement window

Section titled “Decision 2: the forward measurement window”

The measurement is the percentage change of the close over the next 10 bars:

Fragment — not a complete formula

// Ref() with a POSITIVE period reads bars that had not printed at the time.
// In a study that is what we want: we are measuring what followed. In a rule
// it is a look-ahead bug. It must never appear inside a Buy statement.
FwdReturn = 100 * SafeDivide( Ref( Close, Horizon ) - Close, Close, Null );

Four things about that choice, each of which is a decision someone could have made differently:

  • Ten bars is arbitrary. It is short enough that the pattern’s supposed effect should still be present and long enough for the measurement not to be dominated by a single bar’s noise. It is not the “right” horizon; there is not one.
  • Close to close ignores everything that happened inside the window. A pattern followed by a 20% fall and a full recovery scores zero here. That is a real limitation and it is listed again at the end.
  • No costs, no slippage, no spread, no position sizing. This is a measurement of the market, not a simulation of trading it. Adding costs would be meaningless without also specifying entry timing, which would make this a backtest.
  • The last ten bars of every series have no forward window. The formula excludes them by bar number rather than trusting what Ref() returns past the end of the array.

This is where this reality check goes further than the one in Part 6, where you compared bars above an RSI threshold with all bars. Here the pattern carries a context condition, so one benchmark is not enough.

What each comparison actually answers

  1. Fwd % on all measurable barsThe unconditional base rate. What happens anyway.
  2. Fwd % on context barsBars where the previous close was below its 20-bar moving average. What being in a pullback alone was worth.
  3. Fwd % on pattern barsContext AND the engulfing geometry.
  4. Pattern minus contextThe number the hypothesis is actually about: what the shape added.

The context condition is stated now, before any result exists:

Fragment — not a complete formula

// "After a decline", made precise using only bars that had already closed.
Context = Ref( Close, -1 ) < Ref( MA( Close, ContextMA ), -1 );

Nothing about that is uniquely correct. It is one defensible reading of “after a decline”, it uses no future data, and it is fixed before the measurement.

Use a watch list of ten to thirty liquid instruments with at least ten years of daily history — the sort of universe the Part 3 lab assembled. Set the Analysis range to All quotations, so every bar in the database contributes.

Write this down before running anything. A worked version:

I will treat the claim as unsupported on this sample if the mean difference between pattern bars and context bars is smaller than twice its standard error on the pooled result, or if the sign of the difference is inconsistent across the majority of instruments. I will not rerun the test with a different horizon, a different body threshold or a different context definition and report the best of them as though it were the first thing I tried. If I do run variants, I will report how many.

That paragraph is the difference between a test and a search. Keep it with the result.

One exploration that produces one row per symbol containing everything needed to evaluate the hypothesis: the size of each group, the mean forward return in each group, the difference that matters, its standard error, and the up-rates alongside the means so that a couple of extreme moves cannot masquerade as an effect.

Complete runnable AFL

pattern-base-rate.afl
// pattern-base-rate.afl
// Part 7 - Reality Check: Can We Test Whether a Pattern Predicted Anything?
//
// Run this as an EXPLORATION in the Analysis window, with the range set to
// "All quotations". It produces one row per symbol comparing what followed a
// precisely defined bullish engulfing bar with what followed every other bar in
// the same symbol over the same period.
//
// WHAT IS BEING ASSUMED - all of these are choices, and all of them move the
// answer, which is why they are listed before the code rather than buried in it:
//
// Pattern A body-engulfing up bar after a down bar, with a body of at
// least MinBodyPct of its own range, occurring while the previous
// close was below its ContextMA-bar simple moving average.
// Measure Plain percentage change of the close over Horizon bars. No
// costs, no slippage, no spread, no position sizing, no stops.
// Benchmarks Two of them. "All bars" is the unconditional base rate.
// "Context bars" is the base rate among bars that met the context
// condition alone - the comparison that closes the context
// escape hatch, because it asks what the SHAPE added.
// Independence Forward windows overlap, so consecutive observations are
// correlated. The standard error printed below assumes they are
// independent, so it is optimistic. Treat the t ratio as a rough
// screen, never as a p-value.
// Universe Whatever you point the Analysis window at, with all of that
// universe's survivorship problems intact.
//
// This measures an association in one sample of the past. It is not a backtest,
// it is not evidence of a tradable edge, and it says nothing about the future.
// -2 means "require ALL past and future bars", which switches QuickAFL off so
// that the running totals below cover the whole loaded history.
SetBarsRequired( -2, -2 );
Horizon = Param("Forward window (bars)", 10, 1, 60, 1 );
MinBodyPct = Param("Minimum body as % of bar range", 30, 0, 80, 5 );
ContextMA = Param("Context: prior close below MA of", 20, 5, 200, 5 );
MinPrice = Param("Minimum close", 1, 0, 100, 0.5 );
MinTurnover = Param("Minimum 50-bar average turnover", 0, 0, 5000000, 100000 );
// -------------------------------------------------- 1. which bars we may use
// Filter the sample once, up front, and apply the same filter to the pattern
// group and to both benchmark groups. A benchmark computed over a different
// set of bars from the pattern is not a benchmark.
Turnover = MA( Close * Volume, 50 );
Usable = Close >= MinPrice
AND Volume > 0
AND High > Low
AND Turnover >= MinTurnover;
// ------------------------------------------------ 2. the forward measurement
// Ref() with a POSITIVE period reads bars that had not printed yet. In a study
// that is precisely what we want, because we are measuring what followed. In a
// trading rule the same line is a look-ahead bug. It must never reach Buy.
FwdReturn = 100 * SafeDivide( Ref( Close, Horizon ) - Close, Close, Null );
// The final Horizon bars have no forward window. Rather than depend on what
// Ref() returns past the end of the array, exclude them by bar number.
LastBarIndex = LastValue( BarIndex() );
HasForwardWindow = BarIndex() <= LastBarIndex - Horizon;
Measurable = Usable
AND HasForwardWindow
AND NOT IsNull( FwdReturn )
AND Status("barinrange");
// -------------------------------------------------- 3. the context condition
// "After a decline" made precise, using only bars that had already closed.
Context = Ref( Close, -1 ) < Ref( MA( Close, ContextMA ), -1 );
// ---------------------------------------------------- 4. the shape condition
PrevOpen = Ref( Open, -1 );
PrevClose = Ref( Close, -1 );
BodyTop = Max( Open, Close );
BodyBottom = Min( Open, Close );
BodyPct = 100 * SafeDivide( BodyTop - BodyBottom, High - Low, 0 );
ShapeCondition = Close > Open
AND PrevClose < PrevOpen
AND BodyBottom <= Min( PrevOpen, PrevClose )
AND BodyTop >= Max( PrevOpen, PrevClose )
AND BodyPct >= MinBodyPct;
Pattern = ShapeCondition AND Context;
// ------------------------------------------------- 5. three groups of bars
GroupAll = Measurable;
GroupContext = Measurable AND Context;
GroupPattern = Measurable AND Pattern;
// Nz() is used deliberately: FwdReturn is Null on the excluded bars, and a
// single Null leaking into a running total would silently destroy every number
// below it.
ReturnAll = IIf( GroupAll, Nz( FwdReturn ), 0 );
ReturnCtx = IIf( GroupContext, Nz( FwdReturn ), 0 );
ReturnPat = IIf( GroupPattern, Nz( FwdReturn ), 0 );
CountAll = Cum( GroupAll );
SumAll = Cum( ReturnAll );
UpAll = Cum( GroupAll AND Nz( FwdReturn ) > 0 );
CountCtx = Cum( GroupContext );
SumCtx = Cum( ReturnCtx );
SqCtx = Cum( ReturnCtx * ReturnCtx );
UpCtx = Cum( GroupContext AND Nz( FwdReturn ) > 0 );
CountPat = Cum( GroupPattern );
SumPat = Cum( ReturnPat );
SqPat = Cum( ReturnPat * ReturnPat );
UpPat = Cum( GroupPattern AND Nz( FwdReturn ) > 0 );
MeanAll = SafeDivide( SumAll, CountAll, Null );
MeanCtx = SafeDivide( SumCtx, CountCtx, Null );
MeanPat = SafeDivide( SumPat, CountPat, Null );
// ------------------------------------- 6. how large is the difference, really
// Sample variance from running sums. Adequate for a screen at this scale; a
// serious study would compute it in one pass over the raw observations.
VarCtx = SafeDivide( SqCtx - CountCtx * MeanCtx * MeanCtx, CountCtx - 1, Null );
VarPat = SafeDivide( SqPat - CountPat * MeanPat * MeanPat, CountPat - 1, Null );
StdErrDiff = sqrt( SafeDivide( VarPat, CountPat, Null )
+ SafeDivide( VarCtx, CountCtx, Null ) );
Difference = MeanPat - MeanCtx;
TRatio = SafeDivide( Difference, StdErrDiff, Null );
// ------------------------------------------------------------- 7. the report
// One row per symbol, written on the last bar of the analysis range. Symbols
// where the pattern never occurred are dropped: they carry no information about
// the pattern, and leaving them in makes the table look busier than the
// evidence is.
Filter = Status("lastbarinrange") AND CountPat > 0;
AddColumn( CountAll, "Bars measured", 1.0 );
AddColumn( CountCtx, "Context bars", 1.0 );
AddColumn( CountPat, "Pattern bars", 1.0 );
AddColumn( MeanAll, "Fwd % all bars", 1.2 );
AddColumn( MeanCtx, "Fwd % context bars", 1.2 );
AddColumn( MeanPat, "Fwd % pattern bars", 1.2 );
AddColumn( Difference, "Pattern - context", 1.2 );
AddColumn( StdErrDiff, "Std error", 1.2 );
AddColumn( TRatio, "Diff / std error", 1.2 );
AddColumn( 100 * SafeDivide( UpAll, CountAll, Null ), "Up-rate all %", 1.1 );
AddColumn( 100 * SafeDivide( UpCtx, CountCtx, Null ), "Up-rate context %", 1.1 );
AddColumn( 100 * SafeDivide( UpPat, CountPat, Null ), "Up-rate pattern %", 1.1 );
// COUNT and AVERAGE rows. The AVERAGE row weights every symbol equally, so a
// symbol with six occurrences counts as much as one with six hundred. To pool
// properly, export the table and weight each mean by its own Pattern bars.
AddSummaryRows( 2 | 16, 1.2 );

Download pattern-base-rate.afl154 lines

Seven numbered sections, in the order the decisions above were made.

Section 1 defines which bars are eligible at all — minimum price, non-zero volume, a real range, and an optional turnover floor. The same filter applies to every group, because a benchmark computed over a different set of bars from the pattern is not a benchmark.

Section 2 builds the forward measurement and excludes the tail of the series that has no forward window. Section 3 states the context, section 4 the geometry, and section 5 forms the three groups and accumulates them.

The accumulation is the part worth understanding. Cum() produces a running total from the first computed bar, so Cum( GroupPattern ) at any bar is the number of pattern occurrences up to that bar. At the last bar of the analysis range, that running total is the total for the whole range — which is why the exploration outputs a single row per symbol, selected with Status( "lastbarinrange" ), and why every column is a running total evaluated at that one bar.

Section 6 turns the running sums into means, sample variances and the standard error of the difference between two means. Section 7 writes the columns.

  • SetBarsRequired( -2, -2 ) requires all past and future bars, which turns QuickAFL off. This matters because Cum() has not forced whole-history calculation since AmiBroker 5.30, so without this line the totals would depend on how many bars AmiBroker decided to load.
  • Status( "barinrange" ) is true on bars inside the Analysis window’s date range; Status( "lastbarinrange" ) is true only on the last one. The first restricts the accumulation, the second selects the output row.
  • Cum( array ) is the running total. Applied to a Boolean condition it counts occurrences; applied to a value it sums them.
  • Nz( x ) replaces Null with zero. It is used deliberately here: a single Null leaking into a running total would destroy every number after it, silently.
  • SafeDivide( x, y, valueifzerodiv ) protects every ratio, returning Null where a group is empty rather than producing a misleading zero.
  • BarIndex() and LastValue() are used to exclude the final Horizon bars by position, rather than depending on undocumented edge behaviour of Ref().
  • AddColumn( array, name, format ) adds a numeric column; the format 1.2 gives two decimal places, 1.0 gives none.
  • AddSummaryRows( flags, format ) adds summary rows: flag 2 is AVERAGE and flag 16 is COUNT, combined here as 2 | 16.

Three checks, in this order, before you look at the numbers you care about.

Check the count against the chart. Apply engulfing-and-bars.afl from the second lesson to one of the symbols, with the Body definition selected and the same MinBodyPct. Its title reports a bullish engulfing count for the whole history. The exploration’s “Pattern bars” for that symbol will be lower, because the exploration also requires the context condition and the eligibility filter. If it is higher, something is wrong: the same geometry cannot occur more often once conditions are added.

Check the arithmetic on one symbol. Set the turnover floor to zero and the minimum body to zero. “Context bars” should now be close to the number of bars whose previous close was below its 20-bar average — usually a little under half. If it is near zero or near the total, the context condition is not doing what you think.

Check that the horizon behaves. Change the forward window from 10 bars to 1. Every mean should shrink towards roughly a tenth of its former size, because you are now measuring a tenth as much time. If a mean does not move at all, the parameter is not reaching the calculation.

  • Every column shows Null or the exploration returns nothing. The range is not set to All quotations, or “Apply to” is pointing at a single symbol with too little history. Check the range first.
  • Counts change between runs. The SetBarsRequired( -2, -2 ) line was removed or the range was changed. Running totals are only reproducible when the same bars are computed.
  • Enormous means on one symbol. Almost always a data defect: a bad tick, an unadjusted split, or a stale price. Raise the minimum price, then look at the chart for that symbol before doing anything else.
  • Comparing the pattern mean against zero. Zero is not the benchmark. The benchmark is in the next column along.
  • Treating the AVERAGE row as the pooled result. It weights every symbol equally regardless of how many occurrences it contributed. To pool honestly, export the table and weight each symbol’s mean by its own Pattern bars count.

Add two more columns: the mean forward return on bars that met the shape condition without the context, and its count. You will then have all four cells of the table — shape and context, shape only, context only, neither — and you can see directly whether the two conditions interact at all, or whether one of them is carrying the whole result.

From formula to table

  1. Formula EditorPaste the formula, give it a name, save it.
  2. Tools → Send to AnalysisOpens an Analysis window with this formula applied.
  3. Apply toFilter, then use the Filter button to choose your watch list. Or All symbols.
  4. RangeAll quotations.
  5. ExploreOne row per symbol appears in the result list.
  6. File → Export HTML/CSVFor pooling and for the written record.

Here is an invented output, used only to practise the reading. These are not measurements and they are not from any market. They are the shape of table you will get, with numbers chosen to make the reading exercise worthwhile.

Symbol Bars measured Context bars Pattern bars Fwd % all Fwd % context Fwd % pattern Pattern − context Std error Diff / SE
AAA 5,120 1,940 96 0.62 1.18 1.41 +0.23 0.74 0.31
BBB 5,120 2,150 141 0.55 0.98 0.71 −0.27 0.61 −0.44
CCC 3,240 1,290 38 0.71 1.05 2.44 +1.39 1.51 0.92
DDD 5,120 2,010 187 0.48 0.84 1.02 +0.18 0.52 0.35

Four questions, asked in this order.

Is the sample large enough to say anything? CCC has 38 occurrences. Look back at the standard-error table in the previous lesson: at that count almost nothing is distinguishable from nothing. CCC also has the largest apparent effect, which is what small samples do — they produce the extremes in both directions. The two facts belong together in the reading.

Which benchmark am I using? Compare the pattern column with the all bars column and every symbol looks positive: +0.79, +0.16, +1.73, +0.54. Compare it with the context column and the picture changes: +0.23, −0.27, +1.39, +0.18, with one symbol against the claim. Most of what the naive comparison attributed to the pattern was attributable to the context, because the context column is higher than the all-bars column on every row. That is itself a finding — about the context, not about the shape.

Is the difference larger than its own uncertainty? No row reaches a ratio of 2. The largest is 0.92, on the smallest sample. On this table, the honest summary is that the difference between pattern bars and context bars is not distinguishable from zero.

Is the sign consistent? Three positive, one negative, none large. If a pattern carried a real effect of the size the claim implies, you would expect the sign to be more consistent across instruments than that.

The write-up for a table like this is short and it is a real result:

Bullish engulfing, body definition, minimum body 30% of range, context defined as the previous close below its 20-bar simple moving average, measured over a 10-bar forward window on four instruments and about 18,600 measurable bars. Mean forward return on pattern bars exceeded the within-context benchmark on three of four instruments, but no difference reached twice its standard error, and the largest difference occurred on the smallest sample. On this sample the claim is unsupported. One variant tested. Universe subject to survivorship bias; opens not independently verified.

Being clear about the limits is not a disclaimer, it is part of the finding.

  • It is one sample. One universe, one period, one horizon, one definition. A different combination might behave differently, which is why the count of combinations tried has to travel with the result.
  • It is not a backtest. No entry timing, no exit rule, no costs, no spread, no slippage, no position sizing, no portfolio constraints. A positive difference here would be a reason to build a system and test it properly, not a system.
  • A mean hides the path. Close-to-close over ten bars scores a violent round trip the same as a flat drift. Two instruments with identical means can have completely different experiences of holding through the window.
  • A mean hides the distribution. A handful of large moves can carry an average. The up-rate columns are there as a check: if the mean is positive and the up-rate is below the benchmark’s, the effect lives in a few outliers.
  • Overlapping windows overstate certainty. Occurrences within ten bars of each other share most of their measurement window, so the printed standard error is optimistic. The extension in the previous lesson — keeping only occurrences separated by at least the horizon — is the fix.
  • Association is not mechanism. Even a clean, large, well-benchmarked difference would tell you that two things moved together in the past, not why, and not that they will continue to.
  • Your data has its own defects. Survivorship in the universe, unverified opens, unadjusted corporate actions. Part 2 catalogues them; none of them are fixed by running this formula.

You have just done, in a small way, the thing the rest of this course is about: taken a claim, made it precise enough to be wrong, measured it against a fair comparison, and written down what came back. Everything from here is that loop with better tools.

The immediate limitation you will have felt is the language. You ran a formula rather than writing one; when you wanted to change the horizon or add a fourth group, you were editing someone else’s code rather than expressing your own idea. Part 8 fixes that from the ground up — what AFL is, why one line of it performs thousands of calculations, and how Boolean arrays work — and Part 9 gives you the array toolkit this formula quietly depended on: Ref(), running extremes, counting with Cum(), carrying values forward, and the state-versus-event distinction that decides whether a formula does what you meant.

After that, Part 12 turns explorations from a thing you run into a thing you design, and Parts 27 to 33 take a measurement like this one and put it through the machinery that separates a genuine finding from a well-dressed accident.

Check your understanding

Question 1. The formula uses Ref( Close, Horizon ) with a positive period. Why is that acceptable here?
Show the answer and why

Answer: Because the formula measures what already happened rather than deciding what to do next; the same expression in a Buy rule would be a look-ahead bug

Reading forward is exactly right for a study of what followed an event, and exactly wrong for a rule that must act using only information available at the time. The distinction is the use, not the function.

Question 2. On the illustrative table, why is comparing the pattern column with the "all bars" column misleading?
Show the answer and why

Answer: Because the pattern requires a context condition, and the context bars already had higher forward returns than all bars — so the naive comparison credits the shape with the context’s contribution

When a pattern carries a context condition, the benchmark must share it. Otherwise the measured difference mixes the value of the context with the value of the shape, and on that table the context accounted for most of it.

Question 3. Symbol CCC shows the largest difference (+1.39) on 38 occurrences. What is the correct reading?
Show the answer and why

Answer: A difference of 1.39 with a standard error of 1.51 is not distinguishable from zero, and small samples produce the extremes in both directions

The ratio of difference to standard error is below 1. Small samples generate the largest apparent effects in both directions, which is why the count column has to be read before the effect column.

Question 4. Which of these are things this exploration deliberately does NOT do? Select all that apply.
Show the answer and why

Answer: Apply commissions and slippage, Model the path taken inside the forward window, Adjust the standard error for overlapping forward windows

The within-context benchmark is the one thing it does add over a simpler test. Costs, path dependence and the overlap adjustment are all outside its scope, and each is listed in the limitations rather than left for the reader to discover.

Question 5. Why does the formula call SetBarsRequired( -2, -2 )?
Show the answer and why

Answer: Because Cum() no longer forces whole-history calculation, so without it the running totals depend on how many bars AmiBroker chose to compute

The reference for Cum() records that from AmiBroker 5.30 it stopped forcing all bars to be processed, so QuickAFL can shorten the computed range. Requiring all past and future bars makes the totals reproducible.

Sources for this lesson

10 verified · checked 2026-08-31

  1. 01AmiBroker User's Guide — How to create your own explorationamibroker.com/guide/h_exploration.html2026-08-31
  2. 02AmiBroker User's Guide — Analysis windowamibroker.com/guide/h_newanalysis.html2026-08-31
  3. 03AFL Function Reference — AddColumnamibroker.com/guide/afl/addcolumn.html2026-08-31
  4. 04AFL Function Reference — AddSummaryRowsamibroker.com/guide/afl/addsummaryrows.html2026-08-31
  5. 05AFL Function Reference — Statusamibroker.com/guide/afl/status.html2026-08-31
  6. 06AFL Function Reference — Refamibroker.com/guide/afl/ref.html2026-08-31
  7. 07AFL Function Reference — Cumamibroker.com/guide/afl/cum.html2026-08-31
  8. 08AFL Function Reference — SetBarsRequiredamibroker.com/guide/afl/setbarsrequired.html2026-08-31
  9. 09AFL Function Reference — SafeDivideamibroker.com/guide/afl/safedivide.html2026-08-31
  10. 10AFL Function Reference — Nzamibroker.com/guide/afl/nz.html2026-08-31

Every technical claim on this page was checked against the official AmiBroker documentation on the date shown. Where the course disagrees with folklore, the source is how you can tell which one to trust.