Baselines, costs, and fair comparisons

A strategy is useful only relative to a credible alternative

A baseline is the simpler decision process a proposed strategy must improve upon. It turns “the backtest made money” into a sharper question: did this specific idea add value beyond what we could have obtained more simply?

The right baseline depends on the claim:

• A market-timing rule may be compared with buy-and-hold and cash.
• A portfolio optimizer may be compared with equal weighting.
• A confirmation filter should be compared with the same entry rule without the filter.
• A new exit should be compared with the same strategy using the old exit.
• A predictor may be compared with a constant, historical mean, or simple observable forecast.

A deliberately weak baseline makes almost any complexity look impressive. A baseline that answers a different question can be equally misleading.

Baseline, benchmark, control, and ablation are different jobs

These terms are often used interchangeably, but they answer different questions:

One object can serve more than one job, but the label should follow the question. Buy-and-hold may be a useful benchmark for market exposure while remaining a poor ablation for an intraday entry filter. A strategy should not receive credit merely because it beat the easiest available comparison.

Fair means changing one thing at a time

Two alternatives should see the same:

• eligible instruments and dates;
• information available at each decision;
• exposure or risk basis;
• execution and fill assumptions;
• fees, spread, slippage, financing, and borrow treatment;
• capital and constraint policy;
• evaluation window and metric definitions.

If a complex strategy receives close fills and free turnover while the baseline pays next-event slippage, the comparison measures the assumptions rather than the strategy.

A filter example

Illustrative example. Suppose a breakout rule enters after price closes above a 20-day high. A researcher adds a relative-volume filter and reports a higher Sharpe.

The clean comparison keeps the breakout, holding rule, sizing, dataset, and cost model fixed. The only difference is whether the volume condition must be satisfied.

Then report:

• how many candidate breakouts the filter rejects;
• whether it delays entries that still occur;
• the return distribution of accepted and rejected opportunities;
• turnover and cost differences;
• whether improvement persists on later data and nearby thresholds.

Comparing the filtered rule with cash cannot isolate the filter. Comparing the filtered strategy net of costs with the unfiltered strategy gross of costs rewards lower turnover twice.

Normalize the risk question

A leveraged low-volatility strategy and an unleveraged high-volatility strategy may have different returns simply because they take different risk. Depending on the question, compare at equal capital, equal volatility, equal gross exposure, or another declared basis.

No normalization is neutral. Equal-volatility comparison requires an estimated volatility and a leverage policy. Equal capital can leave one strategy dominating risk. Report the native result and the normalized comparison when both are useful.

Costs belong on both sides

Implementation shortfall is the gap between the value implied at the investment decision and the value actually realized after delay, spread, impact, fees, and missed trades. A backtest rarely observes every component directly, but it should not omit the ones central to the strategy.

Cost relevance differs:

Use the same economics for the baseline unless the proposed strategy genuinely changes the implementation. If it does, that change is part of the comparison.

Statistical significance is not the first question

Before asking whether a difference is statistically significant, ask whether the comparison is coherent and economically meaningful. A precise estimate of an unfair comparison is not useful.

Uncertainty still matters. A small improvement driven by a few trades may not survive sampling variation. Multiple baselines and many tried settings create selection risk. There is no universal test that every comparison must run; the method should match dependence, sample size, and the decision being made.

Build a baseline ladder

A useful sequence often contains more than one alternative:

1. Null baseline — cash, no signal, or randomized timing where appropriate.
2. Simple economic baseline — buy-and-hold, equal weight, or the parent rule.
3. Ablation baseline — the proposed strategy with one component removed.
4. Operational baseline — the current process the new design would replace.

The ladder shows where value enters. If a complex model beats cash but not the simple parent rule, the added complexity has not earned its place.

Common comparison mistakes

• Choosing a benchmark because it is easy to beat.
• Changing universe, timing, and risk along with the feature being tested.
• Applying costs to only one alternative.
• Comparing gross returns with net returns.
• Treating equal capital as equal risk without checking contribution.
• Reporting only the winning metric from many comparisons.
• Calling a tiny in-sample improvement economically meaningful.

Further reading

• DeMiguel, Garlappi, and Uppal, “Optimal Versus Naive Diversification” (2009) — Tests fourteen estimated portfolio rules against equal weighting across seven datasets and finds no method that consistently dominates it out of sample. It illustrates why a simple baseline can remain demanding when estimation error is substantial.
• Perold, “The Implementation Shortfall: Paper Versus Reality” (1988) — Defines implementation shortfall from the investment decision through execution and missed trades. It supports applying the same economic accounting to both a proposed strategy and its comparison.
• White, “A Reality Check for Data Snooping” (2000) — Tests the best model from a searched family relative to a declared benchmark, showing why the benchmark and the set of alternatives examined are both part of a fair comparison.