A result inherits the limits of the engine that produced it
Execution fidelity describes how much market-event detail a backtest uses when turning strategy decisions into fills. Provenance is the retained identity of the data, strategy, settings, engine, and artifacts that produced the result.
Fidelity tells you what kind of execution claim may be reasonable. Provenance lets you trace and reproduce that claim. A result without either can still show a number, but the number is difficult to interpret.
Four trust questions use four different words
| Concept | The question it answers |
|---|---|
| Fidelity | How much of the relevant market-event process does the model represent? |
| Validity | Does the model support the particular conclusion being drawn? |
| Provenance | Which exact inputs, versions, methods, and artifacts produced this result? |
| Reproducibility | Can the declared process be rerun to obtain the same result within its stated tolerances? |
Higher fidelity does not guarantee validity if the model is wrong. Complete provenance does not make a biased study credible; it makes the bias traceable. A result can be reproducible and still answer the wrong question.
Discovery and execution are different jobs
A fast vector calculation can test many signal and parameter combinations. It is valuable for discovery because it makes broad comparison practical. It may use simplified position transitions and cost assumptions.
A bar or event-driven evaluator processes decisions through a more explicit sequence. Trade, quote, or depth data can support narrower questions about event order, spread, available liquidity, and queue behavior.
Higher fidelity is not a quality badge attached to the same result. It changes the model, data requirements, runtime, and sometimes the strategy behavior.
What common fidelity levels can support
| Fidelity | Typical use | Claims that still need caution |
|---|---|---|
| Vector | Broad signal and parameter discovery | Intrabar order, path-dependent fills |
| Bar | Completed-bar decisions and bar-aware fills | Which price occurred first inside the bar |
| Tick/trades | Transaction sequence and trade-driven logic | Displayed liquidity and passive queue |
| Quote | Bid/offer state and spread-aware execution | Deeper liquidity and exact queue position |
| Depth | Price-level liquidity and queue models | Hidden liquidity, latency, venue-specific order priority |
A slow strategy may not need event-level data. A market-making result does. The right fidelity is the least expensive model that preserves the information needed by the claim.
Requested and effective fidelity must agree
A study can request one fidelity and encounter missing data or unsupported runtime capability. A trustworthy system either executes the supported request or blocks it. It should not silently fall back to a weaker model while keeping the stronger label.
Results should retain both requested and effective fidelity, any fallback or blocker status, and warnings. If the two differ under an explicitly allowed policy, the claim follows the effective path.
A simple fill example
Illustrative example. Assume a strategy decides to buy after a completed bar closes at 100.
- A vector model may apply a next-period return or price convention.
- A bar model may fill at the next bar’s open plus declared slippage.
- A trade model may wait for the first eligible transaction after the decision.
- A quote model may use the offer available when the order arrives.
- A depth model may simulate available size and queue under a declared policy.
These are not five estimates of one known fill. They are five different execution models. Their disagreement is information about the assumptions.
Provenance answers “which exact result?”
A reproducible result should identify:
- dataset, instrument, coverage, and checksum;
- derived-input or signal-output identity;
- strategy definition and immutable version;
- parameter and binding values;
- study definition and window;
- requested and effective execution fidelity;
- retained result families and output choices;
- cost and fill model;
- engine and schema versions;
- warnings, blockers, and artifact checksums.
“The volume-weighted average price (VWAP) strategy backtest” is not enough. Two runs with the same visible strategy name can use different data, versions, parameters, or runtimes.
Artifacts are evidence only when they belong to the run
A chart file or metric table needs lineage to the owning result. Reusing a stale artifact, regenerating it from different data, or attaching a study-level statistic to an isolated run can create a persuasive but incoherent page.
Artifact manifests, hashes, bounded readers, and status fields help distinguish:
- present and validated evidence;
- missing evidence;
- malformed or stale evidence;
- a method that was not requested;
- a capability that is not established.
Missing evidence should remain visibly missing. An empty panel is more honest than a substitute calculated from incompatible inputs.
Fidelity validation is not live certification
Even a depth-aware historical replay does not establish broker acceptance, network latency, queue priority in every venue, operational monitoring, or live risk behavior. Research fidelity narrows one source of model error. It does not collapse the boundary between backtest and deployment.
Common provenance mistakes
- Comparing results without checking Strategy version and Dataset.
- Reading requested fidelity while ignoring effective fidelity.
- Calling a vector discovery result an execution backtest.
- Assuming more granular data means the evaluator used it.
- Detaching charts or metrics from their owning artifact manifest.
- Recomputing a metric later without recording the new method and inputs.
- Treating missing evidence as zero or passed.
- Assuming replay fidelity certifies live operation.
Further reading
- Peng, “Reproducible Research in Computational Science” (2011) — Presents reproducibility as a minimum standard for evaluating computational evidence when full independent replication is unavailable.
- Sargent, “Verification and Validation of Simulation Models” (2013) — Distinguishes model verification, conceptual-model validity, data validity, and operational validity, separating correct implementation from a credible conclusion.
- W3C, PROV Model Primer (2013) — Introduces a structured model for connecting data and artifacts to the activities, agents, inputs, generation steps, and derivations that produced them.