Benchmarks

What Benchmarks is for

Benchmarks runs a bounded workload on the current computer so you can compare local configurations and notice meaningful performance changes. It is not a prediction of live-order latency, provider speed, or the runtime of every possible Study.

Configure a benchmark run

Choose the available fixture type, fixture row count, and parallel worker count. These fields define the workload being measured:

• Fixture Type selects the supported input shape used by the suite.
• Fixture Rows controls the bounded amount of test data.
• Parallel Workers controls the comparison between serial and parallel execution where the case supports it.

Use the same values when comparing machines or settings. Increasing several inputs at once makes it difficult to identify what changed.

Select Run Benchmark to queue the suite. The run appears in Jobs while it is active and becomes available in the comparison matrix when its retained result is complete.

Compare runs fairly

The Comparison Matrix places retained runs side by side. Select a run to inspect the exact suite version, fixture type, row count, trial count, worker configuration, timestamps, case status, and device profile.

Compare like with like:

1. use the same Arizmic build and benchmark suite;
2. use the same fixture type and row count;
3. keep worker settings explicit;
4. avoid comparing a busy, thermally constrained machine with an idle one; and
5. repeat a run before treating one outlier as a regression.

The median is normally more useful than the fastest single trial. Review the case spread as well: a similar median can hide unstable trials, and a single fast result can hide a slow tail.

Read valid, skipped, and flagged cases

A benchmark case can be valid, skipped, or flagged. A skipped case did not have the supported prerequisites for a meaningful measurement. A flagged or invalid case should not be used in the speed comparison until its diagnostic explains why it is comparable.

The device panel records the host, operating system, processor, core counts, memory, results-disk profile, Python version, and application version associated with the run. Keep that context with any exported result.

Save, export, or remove a run

Use the Run Detail actions to retain a useful comparison, export it for your own records, or delete an obsolete benchmark record. Deleting a benchmark run does not change application settings or remove product Results.

For a change to worker or storage settings, run a small representative Job as well as a benchmark. A synthetic fixture can reveal a local regression, but it cannot establish that a particular Dataset, strategy, provider, or Live connection will behave the same way.