A backtest is a historical simulation of a declared process
A backtest asks: if these rules had been applied to this historical dataset under these execution and cost assumptions, what sequence of positions and outcomes would the model have produced?
That is a useful question. It turns an idea into something inspectable. It can reveal whether rules are internally consistent, how often they act, which markets and periods drive the result, how much exposure they require, and where the historical path becomes uncomfortable.
It is not the same question as “will this strategy make money?” Historical data is one realized path. The rules may have been chosen after seeing it. The fill model may simplify the market. Costs, competition, and behavior can change. A backtest is evidence about a specified model on specified data—not a guarantee and not a substitute for a research design.
What is inside the result
A backtest combines several objects:
| Object | Example | Why it matters |
|---|---|---|
| Strategy version | Entry, exit, sizing, and parameters | Defines the decisions |
| Dataset | Instrument, dates, fields, adjustments | Defines the historical information |
| Timing | Bar close, next event, session cutoff | Determines what was knowable |
| Execution model | Vector, bar, trade, quote, or depth logic | Determines modeled fills |
| Economics | Fees, spread, slippage, financing | Converts paper decisions into net results |
| Evaluation | Metrics, baselines, holdout, uncertainty | Determines how the outcome is judged |
Change any one and the question changes. A strategy that looks attractive on adjusted daily bars with next-close fills is not the same experiment as the same visible rules on trades with event-level costs.
Correct calculation, realistic simulation, and credible inference are separate
A backtest can succeed at one level and fail at another:
| Level | The question | A failure at this level |
|---|---|---|
| Calculation | Did the software apply the declared rules and accounting correctly? | A return is shifted, a fee has the wrong sign, or a position is counted twice |
| Simulation | Could the modeled decisions and fills have occurred with the available data and market mechanics? | A completed-bar signal fills at the same close or a passive order fills without queue evidence |
| Inference | Does the historical result support the claim being made beyond this sample and model? | The winner was selected from many variants or depends on three unusual trades |
Passing the calculation level does not establish realistic execution. Passing both does not establish persistence. This is why code verification, fidelity review, and out-of-sample evaluation are complementary rather than competing ways to “validate a backtest.”
What a backtest can do well
Make the rule concrete
Ambiguous language has to become a decision. “Buy strength” must specify a measurement, threshold, decision event, entry, size, and exit. That translation often exposes assumptions before any metric is calculated.
Describe the historical path
The result can show equity, drawdown, turnover, holding periods, exposure, trade distribution, costs, and failure periods. It can identify whether one event or one instrument produced most of the outcome.
Compare controlled alternatives
When two versions use the same data, timing, costs, and risk basis, a backtest can isolate the effect of one change. A breakout with and without a volume filter is more informative than a filtered breakout compared with cash.
Falsify simple claims
If a strategy cannot survive a basic timing correction, realistic fees, or a reasonable parameter neighborhood, the backtest has found a reason not to continue. Rejection is productive research.
What it cannot establish by itself
That the selected rule was independent of the sample
If the researcher tried many ideas and retained the best, the final output contains selection. A clean chart does not reveal the discarded alternatives.
That modeled fills were available
A bar crossing a price does not prove a passive order filled, a queue was reached, or the position could be traded at scale. The claim cannot exceed the data and execution model.
That the market mechanism will persist
Participants adapt, costs change, capacity fills, regulations shift, and regimes differ. A plausible mechanism is more durable evidence than a pattern alone, but it is still not permanence.
That the estimate is precise
One return path can produce a noisy Sharpe, drawdown, and win rate. Few trades or clustered outcomes make the uncertainty larger than the number of rows may suggest.
That operational execution is ready
Research code, a machine-readable strategy definition, and a historical result do not certify broker behavior, live risk controls, monitoring, or failure recovery.
A result can be correct and still misleading
Suppose an illustrative daily strategy earns most of its return from three trades. The code and arithmetic may be correct. The conclusion “this rule has a stable edge” may still be unsupported because the estimate depends on three events, one cost model, and one chosen window.
The remedy is not to distrust all backtests. It is to narrow the claim: “Under these assumptions, three historical events produced most of the positive result.” That statement points directly to the next research tasks: inspect the trades, test nearby definitions, use later data, and challenge the execution assumptions.
Build evidence in layers
- Rule integrity — confirm timing, data, calculations, and reproducibility.
- Baseline comparison — compare with a simpler credible alternative.
- Sensitivity — inspect nearby parameters, markets, and windows.
- Cost and fidelity — add the economics and data needed by the claim.
- Holdout or walk-forward — evaluate decisions on later data.
- Selection awareness — account for how many alternatives were tried.
- Portfolio context — test overlap, capital, and risk with other strategies.
These layers answer different objections. A sophisticated statistical test cannot repair lookahead. Tick data cannot repair a strategy selected from the test set. More realism is useful only when it addresses the current weakness.
Questions to ask before believing the chart
- What exact rule and version produced it?
- Which dataset and point-in-time policy were used?
- When was each decision knowable, and when could it fill?
- Which costs are included?
- What is the simplest fair baseline?
- How many variants were tried?
- Which trades and periods dominate?
- Does the result survive later data and reasonable nearby settings?
- What evidence label does this run actually support?
Further reading
- White, “A Reality Check for Data Snooping” (2000) — Defines data snooping as reusing one history for model search and inference, then develops a benchmark-relative test for the best searched model. It addresses selection bias, not execution realism.
- Bailey et al., “The Probability of Backtest Overfitting” (2016) — Defines the probability that an in-sample-selected strategy ranks poorly out of sample and presents combinatorially symmetric cross-validation as one way to estimate it. It is a diagnostic of selection risk, not proof that a backtest is valid.
- Peng, “Reproducible Research in Computational Science” (2011) — Explains why computational findings require traceable data, code, parameters, and environment. Reproducibility enables inspection but does not establish that a market effect will persist.