A null model describes what “no useful signal” would look like
A strategy result becomes more informative when it is compared with an alternative designed to contain no version of the claimed edge.
That alternative is a null model. It keeps selected features of the original problem—such as holding periods, risk per trade, market exposure, or return dependence—while removing or randomizing the information the strategy claims to use.
If the strategy cannot reliably distinguish itself from a credible null, the historical outcome may reflect ordinary market drift, exposure, trade count, or selection luck rather than the proposed signal.
A null must match the question
There is no universal random strategy. Consider three claims:
- “The entry timing identifies favorable moments.”
- “The exit improves risk after any entry.”
- “The whole long strategy adds value beyond market drift.”
Randomizing entry time may address the first claim. It does not isolate the exit if entry, holding period, and risk all change. Buy-and-hold may address the third claim but says little about whether one tactical entry is better than another.
A useful null removes the target information while preserving the nuisance features that could otherwise explain performance.
Nulls, benchmarks, and stresses are not interchangeable
| Comparison | Main question |
|---|---|
| Null model | What would the chosen statistic look like if the claimed information were absent? |
| Economic benchmark | Did the proposed policy improve on a credible alternative use of capital? |
| Ablation | Did one specific component add value to the otherwise unchanged strategy? |
| Stress | What happens when a declared adverse assumption is imposed? |
A randomized-entry null can test whether timing was unusual without being a sensible investment benchmark. Buy-and-hold can be a credible benchmark without representing “no timing information.” A spread shock can reveal fragility without estimating a p-value. The comparison must match the claim.
Matched randomized entry
A matched randomized-entry baseline places entries at random eligible times while preserving important features such as:
- number of trades;
- long/short mix;
- holding-period or exit policy;
- risk or size distribution;
- instrument and session eligibility;
- cost model.
Illustrative example. Suppose a strategy makes 80 long entries and holds each for five days. A weak null makes 1,000 unconstrained intraday trades. A matched null also makes 80 long five-day trades in eligible periods. The comparison then focuses more closely on whether the strategy’s timing mattered.
Matching is never perfect. If exit levels depend on the original setup, a random entry may not have equivalent stop and target context. State what was preserved and what had to be approximated.
Permutation tests break a declared relationship
A permutation test rearranges labels, signal values, or outcomes to represent the world in which the tested association is absent while other data features remain fixed.
The core assumption is exchangeability under the null: after the claimed relationship is removed, the values being permuted must be legitimately swappable under the chosen grouping. If volatility regime, date, sector, or holding-period overlap makes some observations structurally different, an unrestricted permutation constructs the wrong comparison.
For example, to test whether a cross-sectional score predicts next-period returns, the score labels can be permuted across instruments within each date. This keeps the date’s return distribution and portfolio size but breaks the mapping between each score and its subsequent return.
Repeating the permutation produces a null distribution for the chosen statistic. The p-value is the proportion of permuted statistics at least as extreme as the observed one under that procedure. It is not the probability that the strategy is false or the probability it will lose.
Time series require dependence-aware randomization
Shuffling every return independently can destroy autocorrelation, volatility clustering, seasonality, and overlapping positions. The null may then be much easier to beat than the real problem.
Alternatives include:
- block or stationary bootstrap;
- circular time shifts;
- within-session or within-regime randomization;
- sign flips under a symmetry assumption;
- randomized entry among matched eligible states.
Each method preserves some structure and destroys another. The right choice follows from the null hypothesis, not from which one produces the smallest p-value.
Passive and cost-only baselines are null-like controls
A passive market baseline asks whether active rules add value beyond exposure. A cost-only baseline asks whether the apparent edge is larger than the economic friction needed to trade it. A shifted-signal baseline asks whether timing survives when the signal is deliberately misaligned.
These controls may be more interpretable than a sophisticated randomization. Use the simplest null that removes the claimed advantage without changing the problem beyond recognition.
A worked interpretation
Illustrative example. Assume a strategy has a net profit of 10 under a chosen metric. Nine hundred ninety-nine matched randomized-entry paths are generated, and 60 produce a metric of at least 10.
Under that exact matching and randomization procedure, the estimated upper-tail proportion is approximately 6%. That suggests the observed timing is unusual relative to the null, but it does not establish a 94% chance of a true edge. The result still depends on matching quality, costs, sample choice, selection among strategies, and whether dependence was preserved.
If the strategy was selected from hundreds of alternatives, a single 6% result is weaker than it appears because the family search created many chances for an unusual outcome.
Common null-model mistakes
- Randomizing so much structure that the null is trivial to beat.
- Preserving the signal accidentally.
- Changing trade count, exposure, or cost along with entry timing.
- Using an independent shuffle for dependent time-series observations.
- Interpreting a p-value as the probability the strategy is true.
- Reporting only the most favorable of several null models.
- Ignoring the search that selected the strategy before the null test.
Further reading
- Ernst, “Permutation Methods: A Basis for Exact Inference” (2004) — Explains the reasoning behind permutation inference and the conditions under which it is exact and distribution-free.
- White, “A Reality Check for Data Snooping” (2000) — Shows how to test the best result from a searched model family against a benchmark rather than treating the reported winner as the only comparison.
- Politis and Romano, “The Stationary Bootstrap” (1994) — Supplies a dependence-aware resampling method for stationary time series when independent shuffling would destroy relevant structure.