The Research Lifecycle End to End

A research process is not a collection of good habits. It is a pipeline with a fixed order, in which every stage consumes a specific artefact produced by the stage before it and hands on a specific artefact of its own. When it goes wrong — and it does — the order is what lets you find out where.

By the end of this lesson you should be able to draw the whole pipeline from memory, name what each stage consumes and produces, say which part of this course taught it, put a defensible time budget against each stage, and, given a symptom, name the stage that owns it.

The pipeline, end to end

From market data to a reviewed decision

1. 1Market dataBars arrive from a vendor or a download. Nothing has been checked yet.
2. 2Data validationStaleness, integrity, holes, adjustments. The gate that everything downstream depends on.
3. 3Market regimeOne classification of the benchmark, computed once, applied to every symbol.
4. 4UniverseWhich instruments are eligible today, by liquidity, price and history.
5. 5ScreenSetup and trigger applied across the universe. An Exploration, not a judgement.
6. 6RankCross-sectional ordering, because more candidates appear than you can act on.
7. 7Candidate listThe hand-off: a short, dated, reproducible list of instruments and numbers.
8. 8Chart reviewA human looks, in a fixed order, at a fixed set of questions.
9. 9Risk assessmentStop level, position size, participation, concentration, event exposure.
10. 10Trading decisionTake, skip or defer - and the reason, recorded before the outcome exists.
11. 11JournalWhat the rules said, what you did, and why they differed.
12. 12Performance evaluationTwo separate measurements on a schedule, feeding back into the rules.

Twelve stages. The three shaded ones are the human's; the course stops at the tenth.

Three features of that diagram do more work than the stages themselves.

It is directed. Regime is decided before the universe is filtered, and the universe is filtered before anything is screened, because each of those choices narrows the next. Running them in a different order produces different answers, and running them in no particular order produces answers you cannot reproduce.

It has a seam. Stages 1 to 7 are mechanical and should be identical on Tuesday and on Thursday. Stages 8 to 10 are human. Stages 11 and 12 exist entirely to measure the seam. Most of the trouble people have with systematic trading lives at that boundary, and almost none of the published material about it distinguishes the two sides.

It ends in a loop that is slower than it looks. Performance evaluation feeds back into the rules, but not every evening. Lesson 4 argues that the feedback arrow should be traversed on a schedule measured in months, not in reaction to the last five trades.

What each stage consumes and produces

Stage	Consumes	Produces	Taught in
Market data	A vendor feed or a download	Bars in the database	Parts 2, 18, 19
Data validation	Bars in the database	A pass/fail row per symbol	Part 2, and this lesson
Market regime	The benchmark’s history	One classification, dated	Parts 16, 35
Universe	Every symbol, the regime	A dated eligibility list	Parts 12, 35
Screen	The universe, the rules	Signal rows from an Exploration	Parts 12, 27
Rank	Signal rows	An ordered candidate list	Part 13
Candidate list	The ranked rows	An exported, dated file	Parts 12, 35
Chart review	The candidate list	Accept or reject, with a note	Parts 4, 5, 7
Risk assessment	Accepted candidates	Stop level and planned size	Part 34
Trading decision	The plan	Take, skip or defer, with a reason	Part 35
Journal	The decision and the plan	One row per decision	Part 35
Performance evaluation	Journal plus backtest	Two measurements and a verdict	Parts 29, 32, 33, 35

Read the middle column again. Every stage produces a thing, and every one of those things can be stored, dated and shown to somebody else. That is not administrative overhead; it is the entire difference between a research process and a routine.

Data and universe: stages 1 to 4

These four are the ones people skip, and they are the ones that silently invalidate everything downstream. A stale symbol produces a signal on last week’s price. A symbol with a hole filled by a repeated close produces a breakout that never happened. A universe defined by today’s index membership produces a backtest of companies selected for having survived.

The output of stage 4 is a list, and the list should be dated. “The universe” is not a property of your database; it is a property of your database on a given evening.

Selection: stages 5 to 7

The screen is an Exploration in AmiBroker’s exact sense — a table defined by Filter and AddColumn() — not a Scan, and the distinction matters because you want the numbers, not just the tickers. The rank is a separate operation and answers a different question: the screen asks does this instrument qualify, the rank asks of the instruments that qualify, which ones do I look at first when there are more than I can act on.

Stage 7, the candidate list, is where the pipeline hands over. It should be small enough to review properly. If the screen routinely produces forty candidates and you can review eight, the constraint is real and the fix belongs at stage 5 or 6, not in reviewing faster.

Judgement: stages 8 to 10

The human stages exist because there are things a formula cannot see: a chart whose recent history is a single gap, an instrument whose data looks fine and whose behaviour does not, a concentration you are already carrying in three correlated positions, an event on the calendar that end-of-day OHLCV does not carry.

They are also where discipline leaks. Lesson 3 gives them a protocol precisely because “looking at the chart” without one is an invitation to find a reason for whatever you were going to do anyway.

Memory: stages 11 and 12

The journal is written at the moment of decision, not afterwards. Performance evaluation reads it later. Lesson 4 is entirely about why those two must be separate measurements and what you can learn from the difference between them.

Two clocks

The twelve stages do not all run at the same frequency, and conflating their clocks is a common and expensive mistake.

The operating loop runs every evening, or every week if you trade weekly bars. It goes from stage 1 to stage 11 and stops. Nothing in the operating loop is allowed to change a rule. Its job is to apply the current rules to today’s data and record what happened.

The research loop runs on a much slower schedule and covers stage 12 and the arrow back into the rules. It is where a hypothesis is formed, a backtest is run, a walk-forward is performed and a rule is changed. It uses the same twelve stages, but in the mode Parts 27 to 33 describe rather than as a daily routine.

!Important

Keep the two loops in separate calendar slots and, ideally, separate Analysis windows and separate formula files. The single most common way a good process degrades is that a bad evening in the operating loop turns into an unscheduled research session, and a rule gets changed on the strength of one trade.

How long each stage should take

The figures below are design targets for a daily end-of-day process over a universe of a few hundred symbols. They are budgets this course recommends, not measurements of anything, and your machine, universe size and interval will move them. Time yourself for a fortnight and replace them with your own numbers.

Stage	Budget	What it means if you exceed it
Data download and import	2–5 min, unattended	Batch it; do not sit and watch it
Data validation	Under 1 min to run, 2 min to read	Too many flagged symbols means the universe or the source needs work
Regime and universe	Under 1 min, combined	A slow run here usually means QuickAFL is off when it need not be
Screen and rank	1–3 min	Consider a narrower Range; the screen rarely needs all quotations
Chart review	2–3 min per candidate, hard-capped	If you cannot finish, the candidate list is too long, not your reading too slow
Risk assessment	2 min per accepted candidate	Longer usually means the sizing rule is not actually decided
Decision and journal	1 min per decision	Longer means you are writing narrative instead of fields
Weekly review	20–30 min	—
Quarterly system review	Half a day	—

The important number in that table is the hard cap on chart review. A process whose human stage has no time budget is a process that will be abandoned within a month, and the failure will be attributed to the strategy.

The artefacts are the deliverable

Each stage produces a file, and the files are what make the process auditable months later. The practical minimum:

• The data-validation exploration, exported, one file per evening.
• The candidate list, exported, one file per evening.
• The journal, one row per decision, appended.
• For every backtest that ever influenced a rule: the report, with the formula and the settings embedded in it. AmiBroker can store both inside the report, and the Report Explorer lists every report you have generated. Turning that on is the single most useful habit for auditing a decision you made six months ago, because a report without its formula is a set of numbers whose provenance you will not remember.

∴Research note

Name the exported files with an ISO date and the rule-set version, not with a description. 2026-08-31-candidates-v1.2.csv is a fact. candidates-good-ones.csv is an opinion, and it is the opinion of somebody who did not yet know how the trades turned out.

Using the pipeline as a diagnostic

The reason to memorise the order is that it converts a vague complaint into a located one.

Symptom	The stage that owns it
A signal fired on a price that never traded	Data validation
The screen returns nothing, on a day when it should	Universe, or the regime gate
The screen returns two hundred rows	Screen: the rules are not selective
The candidate list is dominated by tiny illiquid symbols	Universe: the liquidity floor is too low or absent
You keep skipping candidates for reasons you cannot articulate	Chart review: there is no protocol
Position sizes vary for no reason you can reconstruct	Risk assessment
Live results diverge from the backtest	Journal and evaluation — and until you have both, you cannot say which

That last row is the whole argument for Lesson 4. “The system stopped working” is not a diagnosis. It is three different diagnoses wearing one sentence.

The data-validation gate

Goal

Stage 2 needs a formula that answers one question per symbol: is this symbol’s data fit to be used tonight? It must be fast enough to run before everything else, blunt enough that its output is a verdict rather than a research project, and honest enough to distinguish an arithmetic impossibility from a suspicious pattern.

Complete formula

Complete runnable AFL

pipeline-data-check.afl

// pipeline-data-check.afl
// Part 35 - The Research Lifecycle End to End
//
// Stage two of the research pipeline: data validation. Run this as an
// Exploration over your whole universe before anything downstream is allowed
// to look at a price. It produces one row per symbol describing the state of
// that symbol's data and nothing else. It generates no signals, ranks nothing
// and expresses no opinion about any instrument.
//
// Run settings this formula assumes:
//   Apply to: the watch list that defines your universe
//   Range:    All quotations - the counts below need history, not one bar
//
// Assumptions:
//   - End-of-day daily bars. On intraday data "one bar behind" means something
//     quite different and the staleness column has to be re-read accordingly.
//   - Staleness is measured against the clock of the machine running the
//     formula, so a wrong system date produces a wrong staleness figure.
//   - A repeated close and a zero-volume bar are symptoms, not verdicts. A
//     genuinely untraded day and a vendor filling a hole by carrying the last
//     price forward look identical in an OHLCV file. This formula reports the
//     symptom; deciding what it means is your job.
//   - Nothing here checks whether the prices are adjusted for splits and
//     dividends, because an OHLCV file does not carry the information needed
//     to answer that. Part 2 covers what adjustment does and does not do.

_SECTION_BEGIN("Pipeline data check");

// -2 is the documented "all bars" value for SetBarsRequired, and passing it
// turns QuickAFL's partial-array evaluation off. Without it, the history count
// below would report how many bars this run happened to be given rather than
// how many bars the database holds.
AllBars = -2;
SetBarsRequired( AllBars, AllBars );

StaleDaysLimit = Param( "Flag as stale after (calendar days)", 4, 1, 30, 1 );
RecentWindow   = Param( "Recent window for symptom counts (bars)", 20, 5, 250, 1 );
MinHistoryBars = Param( "Minimum history required (bars)", 250, 20, 5000, 10 );

// ---- What the database actually holds for this symbol --------------------

BarNumber   = Cum( 1 );
BarsHeld    = LastValue( BarNumber );
LastBarDate = LastValue( DateTime() );

// Now(5) returns the current date and time as a DateTime number. DateTimeDiff
// returns the difference between two DateTime values in seconds, so dividing
// by 86400 converts it to calendar days.
AgeInDays = DateTimeDiff( Now( 5 ), LastBarDate ) / 86400;

// ---- Integrity checks ----------------------------------------------------
// These are arithmetic impossibilities rather than judgements. A bar that
// fails one of them is corrupt whatever the vendor's status page says.

HighBelowLow    = High < Low;
CloseOutOfRange = Close > High OR Close < Low;
OpenOutOfRange  = Open > High OR Open < Low;

BadBar      = HighBelowLow OR CloseOutOfRange OR OpenOutOfRange;
BadBarCount = LastValue( Cum( BadBar ) );

// ---- Symptoms worth a human look ----------------------------------------

// A close identical to the previous close with no volume behind it is the
// classic shape of a hole filled by carrying the last price forward.
RepeatedClose = Close == Ref( Close, -1 ) AND Nz( Volume ) == 0;
RepeatedCount = LastValue( Sum( RepeatedClose, RecentWindow ) );

ZeroVolume      = Nz( Volume ) <= 0;
ZeroVolumeCount = LastValue( Sum( ZeroVolume, RecentWindow ) );

// A calendar gap between consecutive bars that is wider than a long weekend.
// Ref() is Null on the first bar, so Nz() floors it and BarNumber guards it.
GapDays      = Nz( DateTimeDiff( DateTime(), Ref( DateTime(), -1 ) ) ) / 86400;
LongGap      = BarNumber > 1 AND GapDays > StaleDaysLimit;
LongGapCount = LastValue( Sum( LongGap, RecentWindow ) );

// ---- The verdict ---------------------------------------------------------

IsStale   = AgeInDays > StaleDaysLimit;
IsShort   = BarsHeld < MinHistoryBars;
IsCorrupt = BadBarCount > 0;
Usable    = NOT IsStale AND NOT IsShort AND NOT IsCorrupt;

// 0 usable, 1 short history, 2 stale, 3 failed integrity. The nested IIf runs
// worst-first so the most serious problem is the one reported, and the words
// are printed as words - the verdict is never carried by colour alone.
VerdictCode = IIf( IsCorrupt, 3, IIf( IsStale, 2, IIf( IsShort, 1, 0 ) ) );
VerdictList = "usable\nhistory too short\nstale\nfailed integrity check";

Filter = Status( "lastbarinrange" );

AddTextColumn( FullName(), "Name", 40 );
AddMultiTextColumn( VerdictCode, VerdictList, "Verdict", 24 );
AddColumn( DateTime(), "Last bar", formatDateTimeISO );
AddColumn( AgeInDays, "Age (days)", 1.1 );
AddColumn( BarsHeld, "Bars held", 1.0 );
AddColumn( BadBarCount, "Impossible bars", 1.0 );
AddColumn( RepeatedCount, "Repeated closes", 1.0 );
AddColumn( ZeroVolumeCount, "Zero-volume bars", 1.0 );
AddColumn( LongGapCount, "Calendar gaps", 1.0 );
AddColumn( Usable, "Usable", 1.0 );

// Column numbers are one-based and they count the two default columns
// (Ticker and Date/Time), so "Usable" is column 12 and "Age (days)" is
// column 6. Unusable symbols first, then the stalest of them at the top.
SetSortColumns( 12, -6 );

_SECTION_END();

Download pipeline-data-check.afl109 lines

How it works

The formula has four sections and no cleverness anywhere.

What the database holds establishes the two facts everything else is measured against: how many bars exist for this symbol, and when the last one is dated. Now(5) returns the current date and time as a DateTime number, and DateTimeDiff() returns the difference between two DateTime values in seconds, so dividing by 86,400 gives the age of the last bar in calendar days.

Integrity checks test arithmetic impossibilities: a High below the Low, a Close or an Open outside the bar’s own range. These are not judgements about quality. A bar that fails one of them is corrupt, and no amount of context makes it not corrupt.

Symptoms count patterns that are consistent with a defect but do not prove one: a close identical to the previous close with no volume behind it, zero-volume bars, and calendar gaps between consecutive bars that are wider than a long weekend. Each is reported as a count over a recent window rather than as a flag, because the count is what tells you whether it is an occasional quirk or a systematic property of the feed.

The verdict collapses the checks into one of four words, chosen worst-first so that a symbol failing several tests reports the most serious failure. The words are printed as words. A verdict carried only by a colour is unreadable to a screen reader and to anyone who exports the file.

Key functions

• SetBarsRequired( -2, -2 ) — the documented way to require all bars, which turns off QuickAFL’s partial-array evaluation. Without it the history count reports how many bars this run was handed, not how many the database holds. This is the one place in the whole pipeline where turning QuickAFL off is the right answer.
• DateTimeDiff( arg1, arg2 ) — difference between two DateTime values, in seconds. It is the supported way to compare DateTime values, which are a bitset rather than a plain number and cannot be reliably compared with < or >.
• Now( 5 ) — the current system date and time as a DateTime number. Formats 0 to 10 return quite different things; 5 is the one that pairs with DateTimeDiff().
• AddMultiTextColumn( array, textlist, caption, ... ) — a text column whose text is chosen per bar by a numeric selector. Note the argument order: the caption is third, because the newline-separated text list is second. AddTextColumn() cannot do this, because it takes a single string that cannot vary bar by bar.
• SetSortColumns( col1, col2 ) — one-based column numbers, negative for descending, and they count the two default Ticker and Date/Time columns.

Expected result

Set Apply to your universe watch list, Range to All quotations, and press Explore. You should get exactly one row per symbol, sorted so that unusable symbols appear first and the stalest of those at the top.

On a well-maintained end-of-day database most rows read usable, with an age of one to three days depending on when you last downloaded and whether the market was open. Symbols recently added to the database appear as history too short. A handful of thinly traded symbols will show non-zero repeated-close and zero-volume counts, which is information, not an error.

=Expected result

The row you are looking for is not the flagged one. It is the symbol you expected to be flagged and which came back usable — that is the case where your idea of the data and the data disagree, and the data is right.

Test it

Run it against a symbol you can verify by hand.

1. Pick any symbol and note the last bar date from its chart.
2. Run the exploration and confirm the Last bar column matches that date exactly.
3. Change your machine’s clock forward by a week and re-run. Every symbol should now report stale. Change it back. This proves the staleness column is measuring what you think it is, and it also demonstrates its weakness: it trusts the system clock.
4. Set the history floor above the longest history in your database. Everything should report history too short. Nothing else should change.

Common errors

• Every symbol reports history too short. The Range is narrow, or SetBarsRequired() was removed. QuickAFL gave the formula a slice of the array and the bar count is counting the slice.
• The exploration produces one row per bar per symbol. Filter was set to 1 instead of Status("lastbarinrange"). With All quotations selected this is how a twenty-year database turns into millions of rows.
• The Age (days) column is negative. The system clock is behind the data, which happens with a manually imported file that carries a future date, and is worth knowing about.
• A symbol shows a large calendar-gap count and is otherwise clean. Very often correct: a suspended instrument, a market holiday sequence, or a symbol that only recently began trading daily. The column is a prompt to look, not a verdict.

Extension

Add a column that reports the largest single-bar percentage change over the recent window. Prices that jump by a factor consistent with a split are the most common symptom of an adjustment that has not been applied, and the check costs one line. Resist the temptation to turn it into a verdict: a genuine 40 per cent move and an unadjusted 2-for-1 split are distinguishable only with corporate-action data the OHLCV file does not carry.

Recap

You now have the pipeline as an object rather than as a set of habits: twelve ordered stages, each consuming a named artefact and producing another, with a clearly marked seam where the human takes over and a slower loop that feeds evaluation back into the rules. You have a budget for each stage that you can hold yourself to, a diagnostic table that turns a symptom into a located stage, and a working data-validation gate that stops bad bars from becoming signals.

The next lesson takes the two stages that constrain everything downstream — the universe and the regime — and makes both of them explicit enough to be criticised.

Check your understanding

Question 1. The screen returns two hundred rows every evening and you can review eight candidates. Which stage owns the problem?

• Chart review - you need to review faster
• Screen or rank - the rules are not selective enough, or the ordering is not being used
• Data validation - too many symbols are passing the gate
• Performance evaluation - the strategy is not working

Show the answer and why

Answer: Screen or rank - the rules are not selective enough, or the ordering is not being used

A candidate list longer than the review budget is a selection problem, not a reading-speed problem. Either the setup and trigger admit too much, or the ranking exists and is not being used to cut the list. Trying to fix it at the review stage means reviewing badly, which is worse than reviewing fewer candidates properly.

Question 2. Why does the data-validation formula call SetBarsRequired with -2 for both arguments?

AllBars = -2;
SetBarsRequired( AllBars, AllBars );

• To make the exploration run faster
• To require all bars, so the history count reports what the database holds rather than what QuickAFL handed the formula
• To force AmiBroker to re-download the data
• Because AddMultiTextColumn needs it

Show the answer and why

Answer: To require all bars, so the history count reports what the database holds rather than what QuickAFL handed the formula

-2 is the documented "all bars" value. QuickAFL evaluates only the part of the array it thinks is needed, which is normally a benefit and is exactly wrong here: a bar count taken from a partial array reports the size of the slice. This is one of the few places where turning QuickAFL off is correct, and it costs run time to do it.

Question 3. Which of these belong to the operating loop rather than the research loop? Select all that apply.

• Running the data-validation exploration
• Changing the breakout lookback from 50 to 40 bars because last night’s candidate went nowhere
• Exporting the candidate list
• Writing the journal row for a decision to skip a candidate

Show the answer and why

Answer: Running the data-validation exploration, Exporting the candidate list, Writing the journal row for a decision to skip a candidate

The operating loop applies the current rules to today’s data and records what happened. Changing a parameter is a research-loop activity, and doing it in reaction to one outcome is the specific failure mode the two-clock separation exists to prevent. It also destroys the comparability of every journal row written before the change.

Question 4. A symbol reports six repeated closes and six zero-volume bars in the last twenty bars, and passes every integrity check. What does the formula let you conclude?

• The data is corrupt and the symbol must be excluded
• The data is fine, because no integrity check failed
• That the pattern is consistent both with a genuinely untraded instrument and with holes filled by carrying the last price forward, and the file cannot distinguish them
• That the vendor has stopped supporting the symbol

Show the answer and why

Answer: That the pattern is consistent both with a genuinely untraded instrument and with holes filled by carrying the last price forward, and the file cannot distinguish them

Repeated closes with zero volume are a symptom with at least two causes, and an OHLCV file contains no field that separates them. The formula reports the count precisely so that a human decides. What it does tell you unambiguously is that a breakout rule reading those bars is reading something other than trading.

Question 5. Why should a stored backtest report include the formula and the settings that produced it?

• Because AmiBroker refuses to generate a report otherwise
• Because the report is the only artefact that survives, and numbers without their formula and settings cannot be reproduced or audited later
• Because it makes the report smaller
• Because the Report Explorer sorts by formula name

Show the answer and why

Answer: Because the report is the only artefact that survives, and numbers without their formula and settings cannot be reproduced or audited later

Every stage of the pipeline produces an artefact whose value depends on being reproducible. Two reports run with different periodicity, different cost assumptions or different rate settings on the Report tab are not comparable, and six months later the only record of which was which is the copy stored inside the report itself.

Check my answersClear