Component 9: The Research Report
This is the deliverable. Everything else was built to make it possible.
Thirteen questions. Each one has an honest answer, an evidence requirement, and a characteristic weak answer. Write all thirteen, attach the evidence, and be willing to have the whole thing read by someone who is trying to find the hole in it — because the person most likely to read it that way is you, in six months, with money at stake.
The thirteen questions
Section titled “The thirteen questions”1. What exactly is the strategy?
Section titled “1. What exactly is the strategy?”How to answer honestly. The complete specification from Component 6: hypothesis, universe, regime, entry, exit, stop, sizing, constraints, costs. In words, precisely enough that a competent reader could implement it and get your trade list.
Evidence to attach. The specification template, filled in, plus the .afl file with its header
intact.
A weak answer says “a trend-following breakout system with a 3 ATR stop”. That is a genre, not a strategy.
2. What is the hypothesis, and what would refute it?
Section titled “2. What is the hypothesis, and what would refute it?”How to answer honestly. One sentence of belief, stated as a claim that could be false. Then the refutation criterion as you wrote it before the first run, with its date.
Evidence to attach. The dated log entry.
A weak answer is a refutation criterion invented after the results. The date is what makes it checkable, which is the whole reason for writing it in advance.
3. What universe was it tested on, and how was that list built?
Section titled “3. What universe was it tested on, and how was that list built?”How to answer honestly. The instruments, the source of the list, the date the list was constructed, and — the load-bearing part — whether it is point-in-time or current membership.
Evidence to attach. The list itself, and a note on what is missing from it.
A weak answer says “40 liquid stocks”. A strong one says: “the current constituents of index X as at March 2026, which excludes every company delisted, merged or taken over during the test period. I could not build a point-in-time list, so the results are biased upward by an amount I cannot quantify.”
4. What period, and what kinds of market did it contain?
Section titled “4. What period, and what kinds of market did it contain?”How to answer honestly. The dates, and a description of the environments inside them. Then the concentration figure from the regime split exploration: what share of your signals fell into the single most populated regime?
Evidence to attach. The regime split output.
A weak answer gives dates and stops. Fifteen years that were mostly one kind of market is one observation of a market, however many bars it contains.
5. What were the execution assumptions?
Section titled “5. What were the execution assumptions?”How to answer honestly. Delay, fill price, slippage, commission, stop fill assumption
(ExitAtStop value), participation cap. Each with its value.
Evidence to attach. The formula’s assumptions block and the execution audit table.
A weak answer says “realistic costs were included”. Which costs, at what level, entering where?
6. Were the trades physically possible?
Section titled “6. Were the trades physically possible?”How to answer honestly. From the execution audit: the average same-bar leak avoided, the largest participation percentages, and how often the bar after a fill gapped through the stop.
Evidence to attach. The audit table, sorted by participation.
A weak answer omits this question entirely, which is what most published backtests do.
7. How many specifications did you evaluate?
Section titled “7. How many specifications did you evaluate?”How to answer honestly. A number, from the research log. Every parameter combination, every rule added or removed, every universe, every date range, every abandoned attempt.
Evidence to attach. The log.
A weak answer says “a few”. If you cannot produce a number, the answer to this question is “unknown”, and that is what you should write.
8. What does the parameter surface look like?
Section titled “8. What does the parameter surface look like?”How to answer honestly. Plateau, spike or flat — with the optimisation table, and the reported result’s position on the surface. Not the best cell.
Evidence to attach. The optimisation output.
A weak answer reports the best cell as the result. That is the fault Part 30’s challenge exists to teach.
9. What happened out-of-sample?
Section titled “9. What happened out-of-sample?”How to answer honestly. The result on data you had not looked at, the date you first examined it, and whether you subsequently changed anything.
Evidence to attach. The out-of-sample report and the log entry.
A weak answer describes an out-of-sample test that was run several times with adjustments in between. That is in-sample testing with extra steps, and saying so is better than not.
10. What happened in walk-forward, and were the parameters stable?
Section titled “10. What happened in walk-forward, and were the parameters stable?”How to answer honestly. The concatenated out-of-sample result, the degradation ratio, and whether the selected parameters were stable across steps.
Evidence to attach. The walk-forward summary and the per-step parameter selections.
A weak answer quotes the performance and ignores the stability. Parameters that jump to a different corner of the grid at every step are telling you the optimiser is fitting noise — often the single most informative output of the whole exercise.
11. How much of the result came from the rules?
Section titled “11. How much of the result came from the rules?”How to answer honestly. Compare against the things that are not your rules: the matched buy-and-hold benchmark, the random-signal null model, and the regime-gate-off variant.
Evidence to attach. All three comparison results.
A weak answer compares against a broad index over a different universe. That measures the difference between two universes at least as much as the strategy.
12. What is not modelled, and in which direction does each omission bias the result?
Section titled “12. What is not modelled, and in which direction does each omission bias the result?”How to answer honestly. The “explicitly not modelled” list, with a direction against each item. Borrow cost — worse. Dividends — better for buy-and-hold, better for you while holding. Taxes — worse. Market impact beyond the cap — worse. Queue position — worse.
Evidence to attach. The list.
A weak answer lists the omissions without directions. The directions are what let a reader adjust your conclusion.
13. What would you do next, and what would make you stop?
Section titled “13. What would you do next, and what would make you stop?”How to answer honestly. The next test, and the result that would end the line of investigation.
Evidence to attach. None — this is judgement, and it is the question that shows whether you have one.
A weak answer is “trade it small”. Trading is not a test; it is the most expensive and slowest way to gather evidence, and it should follow the evidence rather than substitute for it.
A worked example answer
Section titled “A worked example answer”Question 4, answered properly, from a hypothetical capstone:
Period: 4 January 2016 to 31 December 2025, daily bars.
The range contains one extended rise (2016–2019), one sharp decline and recovery (2020), one further rise (2021), one broad decline (2022) and one recovery (2023–2025). It does not contain a multi-year sideways market, and it does not contain a sustained high-inflation environment.
The regime split exploration classifies signals on two axes — benchmark above/below its 200-bar average, benchmark ATR percentile above/below its own median. Of 412 signals, 351 (85%) fell into the up-trend/low-volatility cell. The three remaining cells contain 61 signals between them, which is not enough to say anything about the strategy’s behaviour in those environments.
What this means for the conclusion: the strategy has effectively been tested in one kind of market. Its behaviour in a declining or high-volatility market is unknown, not favourable-and-small. The regime gate is doing most of the work of keeping it out of those environments, which is by design — but it also means the test has no information about what the entry rules do when the gate is open in a volatile market, because that combination barely occurred.
Notice what makes it strong: it quantifies the concentration, it states what the sample cannot support, and it draws the distinction between “measured and small” and “not measured”.
What a weak answer looks like
Section titled “What a weak answer looks like”The same question, answered the way most reports answer it:
Period: 2016 to 2025, which covers a variety of market conditions including the 2020 crash and the 2022 bear market. The strategy performed well across the full period.
Three specific failures, none of them dishonest:
“A variety of market conditions” is asserted, not measured. The regime split takes ten minutes and replaces the assertion with a number.
Naming two events implies the strategy was tested through them. It may have been flat throughout both, because the regime gate held it out — which means it was not tested through them at all.
“Performed well” has no assumptions attached, so it is not a claim anybody could check.
Self-assessment rubric
Section titled “Self-assessment rubric”Score each question 0, 1 or 2. Be strict — the point is to find the weak answers, not to pass.
| Score | Meaning |
|---|---|
| 0 | Not answered, or answered without evidence |
| 1 | Answered with evidence, but with a hedge, an assertion, or a missing qualifier |
| 2 | Answered with evidence, with the limitation stated, in a form a hostile reader could check |
| Total | What it means |
|---|---|
| 22–26 | A defensible piece of research. You know what you found and what you did not. |
| 16–21 | Sound work with identified gaps. Name them in the report rather than hoping nobody notices. |
| 10–15 | The machinery works and the reasoning does not yet. Re-read questions 3, 7, 8 and 11. |
| Below 10 | You have a backtest, not research. Nothing here is wasted; the analysis is the part still to do. |
Where this ends
Section titled “Where this ends”Alert → human review → decision, and no further
- The researchComponents 1–8. Evidence about a hypothesis, with its limitations stated.
- The reportThirteen questions, answered. This is where the course ends as a course.
- A decision about the researchContinue investigating, park it, or abandon it. All three are legitimate outcomes.
- If you go further: alert → human review → decisionThe workspace raises an alert. A person reads it. A person decides.
- Not part of this course: the orderPlaced by you, through your broker, on your own judgement and your own responsibility.
The last thing to write
Section titled “The last thing to write”After the thirteen answers, one paragraph, in plain language, that somebody who does not trade could read:
What did I set out to find, what did I actually find, and how confident am I — and why?
If that paragraph is hard to write, it is usually because the honest answer is less impressive than the work felt. Write it anyway. That paragraph is the difference between having done research and having produced output.
Thirteen questions, each with an honest answer, an evidence requirement and a characteristic weak form. The hardest are question 3 (how the universe was built), question 7 (how many specifications you evaluated) and question 11 (how much of the result came from your rules rather than from the market being up). Score yourself strictly, because a low score is the rubric working. And the chain ends where it has ended throughout: alert, human review, decision — made by a person who has read their own evidence and knows exactly how much it is worth.
Check your understanding
Sources for this lesson
4 verified · checked 2026-08-31
- 01AmiBroker User's Guide — Backtest reportamibroker.com/guide/w_report.html2026-08-31
- 02AmiBroker User's Guide — Walk-forward testingamibroker.com/guide/h_walkforward.html2026-08-31
- 03AmiBroker User's Guide — Portfolio-level backtestingamibroker.com/guide/h_portfolio.html2026-08-31
- 04AmiBroker User's Guide — Monte Carlo simulationamibroker.com/guide/h_montecarlo.html2026-08-31
Every technical claim on this page was checked against the official AmiBroker documentation on the date shown. Where the course disagrees with folklore, the source is how you can tell which one to trust.