Studies
Study Types, Optimization, and Walk-Forward
Choose between Solo, Grid, Random, and Optuna Studies, then match the sampler, walk-forward method, fidelity, and analysis depth to the research question.
App path
- Studies -> Study Design -> Study Mode
Choose the Study that matches the question
Arizmic separates what you want to compare from how each candidate is simulated. Start by choosing a Study Type and, for an optimization, a search method. Then choose the Execution Fidelity and Analysis Depth that make the result useful.
| Research question | Starting choice | Main benefit |
|---|---|---|
| Does this exact Strategy setup run as intended? | Solo Run | One clear baseline with no parameter search |
| What happens at every value in a small, deliberate search space? | Grid Optimization | Complete coverage of every declared combination |
| Which parts of a large search space deserve attention? | Random Optimization | Broad, replayable sampling without evaluating the whole grid |
| Can the search spend more trials in promising regions? | Optuna | Adaptive, sampler-driven search under a fixed trial budget |
| Does a selected setup remain useful through changing periods? | Walk Forward | Repeated train-and-validation folds with explicit temporal separation |
| Does a discovery candidate survive more realistic fills? | A higher Execution Fidelity | Tests the same decision under richer execution data and timing |
The available combinations are resolved from the current workstation. A method, sampler, fidelity, or analysis depth can be disabled when its data, runtime, optional package, or Strategy requirements are not satisfied. Treat the explanations below as a selection guide and Readiness as the final authority for the current Study.
Solo Run
Solo Run evaluates one concrete parameter point and one concrete set of input bindings. It produces an individual Run Output rather than an optimization leaderboard.
Use it to:
- establish a baseline before searching;
- verify that entries, exits, sizing, and costs behave as expected;
- reproduce one previously selected candidate;
- test one candidate at a higher execution fidelity; or
- isolate whether a change came from the Strategy or the optimizer.
The main benefit is interpretability. There is no search process to confuse with the Strategy result. A Solo Run is normally the right first step and the right final confirmation step, even when optimization is part of the research in between.
Optimization Studies
An Optimization Study evaluates multiple candidate configurations. Study Axes define the values that may change. The objective metric and direction define how completed candidates are ranked, but a leading score is not proof that the parameters will generalize.
Arizmic offers three optimization families.
Grid Matrix Array
Grid Optimization enumerates every declared combination of parameter values. For numeric axes, the declared minimum, maximum, and increment determine the points on the grid.
Use Grid when:
- the search space is small enough to evaluate completely;
- every combination has a clear research meaning;
- you want a parameter surface rather than only a winning point; or
- you need to prove that no declared combination was skipped.
Its benefit is completeness. Its cost grows multiplicatively: five values on each of four axes already produce 625 candidates before additional assets, bindings, execution sets, or walk-forward folds are counted. Check the current estimate rather than multiplying only the visible Strategy parameters.
Random Matrix Array
Random Optimization draws a fixed number of candidates from the declared parameter ranges. Sample Count controls the budget, and Seed makes the same sample matrix reproducible.
Use Random when:
- a complete grid would be too large;
- you want broad coverage before choosing a narrower search;
- only a few axes may matter and you do not know which ones yet; or
- you need a neutral benchmark for an adaptive optimizer.
Its benefit is budget control. It can cover more dimensions than a comparable grid without requiring every cross-combination. It does not learn from earlier trials, so it may spend candidates in weak regions; that is also what makes it a useful baseline against Optuna.
Optuna
Optuna uses a selected sampler to propose trials under Max Trials. It also records the objective, seed, sampler settings, pruning policy, and retained trial count.
Use Optuna when:
- the search space is too large for exhaustive Grid;
- later trials should use information from earlier results;
- the space contains a mix of numeric and categorical decisions;
- individual trials are expensive enough that sample efficiency matters; or
- the research has an explicit multi-objective tradeoff.
The benefit depends on choosing a sampler that matches the shape of the search space. An adaptive search can find promising regions with fewer trials, but it also concentrates observations. Always compare the selected region with the baseline, neighboring parameters, and retained validation evidence.
Optuna samplers
The Sampler menu is driven by the live capability matrix. Some samplers use optional optimizer or OptunaHub packages and may appear blocked until those dependencies are installed. The menu message and Readiness finding explain the current requirement.
| Sampler | Best fit | Benefit and important boundary |
|---|---|---|
| TPE | Mixed numeric and categorical spaces | Models stronger and weaker trials, then samples where improvement looks more likely. It is the normal first adaptive optimizer. Multivariate TPE can model parameter interactions. |
| Random | Noisy spaces and optimizer baselines | Samples each trial independently through Optuna. It is easy to reason about and useful for checking whether a more complex sampler adds value. |
| AutoSampler | Exploratory spaces where the appropriate sampler is unclear | Chooses an underlying sampler from the objective and search-space shape. Arizmic exposes seed-based configuration; user-supplied constraint functions are not wired. |
| Hierarchical TPE | Conditional or branch-shaped spaces | Extends TPE for parameters that become relevant only under certain choices. Arizmic exposes seed and startup trials; a user-supplied conditional function is not part of the current control. |
| CMA-ES | Dense continuous parameter spaces | Adapts a covariance matrix around promising continuous regions. It is a poor fit for mostly categorical or Boolean axes and can require its optional optimizer package. |
| TuRBO | Continuous, single-objective searches | Uses local trust regions to search difficult continuous spaces. Categorical axes and multi-objective Studies are blocked for this sampler. |
| QMC | Even numeric coverage under a fixed budget | Uses Sobol or Halton low-discrepancy sequences for more even coverage than ordinary random sampling. Categorical axes are sampled independently. |
| Grid | Finite discrete choices that should use Optuna trial accounting | Enumerates the declared discrete choices while preserving Optuna's trial history and retention controls. Max Trials can cap the grid. |
| Brute Force | Fully enumerable bounded spaces | Traverses every branch the search-space contract can enumerate. It is exhaustive and can become very large quickly. |
| Genetic Search | Single-objective spaces that benefit from population search | Evolves candidates through selection, crossover, and mutation using an NSGA-II-backed sampler. Population settings materially change the search. |
| NSGA-II | Two or more competing objectives | Evolves a Pareto population and retains objective-vector evidence instead of forcing every tradeoff into one score. |
| NSGA-III | Many-objective Pareto searches | Extends the evolutionary approach to broader objective vectors where a two-objective frontier is not enough. |
| Gaussian Process | Expensive continuous trials with a relatively small evaluation budget | Fits a surrogate response surface and chooses information-rich trials. It can be valuable when each evaluation is costly and requires the optional Gaussian-process dependencies. |
For TPE, start with enough startup trials to observe the space before the model guides the search. For population samplers, the population size, crossover, and mutation settings are part of the experiment. For QMC, choose Sobol or Halton and retain the seed. For every sampler, keep the full trial denominator and do not report only the retained leaders.
Optuna pruners
A Pruner can stop an unpromising trial before it consumes its full budget. Standard Studies report aggregate trial progress; walk-forward Studies report fold progress. Pruning saves time, but a pruned trial is not a completed candidate and should remain visible in trial accounting.
| Pruner | Useful when | Main benefit or caution |
|---|---|---|
| No Pruner | Trials are short, progress is not comparable, or a complete denominator matters most | Runs every trial to completion and provides the cleanest baseline. |
| Median Pruner | Intermediate results are comparable across trials | Stops trials performing below the median of earlier completed trials after the warmup. |
| Percentile Pruner | You want a stricter or looser cutoff than the median | Uses a declared percentile of earlier performance. The cutoff and minimum-trial settings affect aggressiveness. |
| Threshold Pruner | A known objective boundary makes a trial unacceptable | Stops when reported progress crosses the declared lower or upper threshold. |
| Successive Halving Pruner | Many trials can start cheaply and receive more resources in rounds | Progressively keeps stronger trials. Resource and reduction settings define the competition. |
| Hyperband Pruner | You want several successive-halving brackets with different budgets | Allocates resources across brackets, improving coverage of early-stop schedules at the cost of more complex accounting. |
| Wilcoxon Pruner | Fold or step comparisons support a paired statistical check | Uses a Wilcoxon test to stop trials whose intermediate path is unlikely to improve on the reference. |
| Patient Pruner | Short-term noise would make another pruner stop too early | Wraps a concrete pruner and requires its condition to persist for the declared patience and minimum change. |
Pruner availability can depend on the selected method, execution fidelity, and runtime. If Readiness requires No Pruner, do not reinterpret that as a performance recommendation; it is the supported execution boundary for that Study configuration.
Standard and walk-forward methods
Method controls whether candidates use one research window or a sequence of training and validation folds.
| Method | How the training window moves | Good use case | Main benefit |
|---|---|---|---|
| Standard | One evaluation window | Discovery, baselines, and direct parameter-surface comparison | Simple, fast, and easy to interpret |
| Rolling Walk Forward | A fixed-length training window moves forward by the Step | Markets where older history should expire | Keeps the training horizon recent and consistent |
| Anchored Walk Forward | Training begins at one fixed start and expands each fold | Research where all earlier observations should remain available | Uses increasing history while preserving later validation windows |
| Capped Anchored Walk Forward | Training expands from the anchor but retains no more than the declared cap | Research that benefits from an expanding start but should not let remote history dominate | Balances accumulating evidence with a maximum training-history budget |
Walk-forward is useful for testing temporal stability, not for declaring the validation folds untouched after repeatedly inspecting and changing the Study. If the design is revised using fold outcomes, create a genuinely separate final holdout or rerun under the new declared process.
Walk-forward window controls
| Control | Meaning |
|---|---|
| Window Unit | Bars, Minutes, Hours, Calendar Days, Weeks, or Sessions. Time units resolve UTC boundaries. Bars use the primary clock. Sessions require an unambiguous persisted calendar and compatible input topology. |
| Train | Amount of data used to select or evaluate the candidate before each validation fold. For Anchored, this is the initial training amount; for Capped Anchored, it is the maximum retained history. |
| Validation | Amount of later data used to evaluate the candidate without adding it to that fold's training interval. |
| Step | Distance between consecutive fold starts. A smaller step creates more overlap and more folds. |
| Purge | Authored interval removed between training and validation to reduce direct overlap or label leakage. |
| Embargo | Additional separation after the purge interval. |
Use Bars when the input clocks are deliberately aligned and bar count is the natural horizon. Use time units when elapsed market time matters more than row count. Use Sessions when the market calendar defines the research horizon and Study Design confirms that one unambiguous calendar is available.
The fold estimate is authoritative. Arizmic excludes folds that lack required input coverage instead of treating missing rows as a neutral result.
Execution Fidelity is separate from the search method
Grid, Random, and Optuna decide which candidates are evaluated. Execution Fidelity decides how each candidate is simulated.
| Fidelity | What it represents | Typical use |
|---|---|---|
| Vector | Idealized vector fill model using prepared arrays and lightweight output | Broad discovery and large parameter searches |
| Bar | Conservative bar replay with execution events | Candidate confirmation when bar timing and fills matter |
| Tick / Trades | Source-ordered trade-tick replay | Strategies whose decisions or fills depend on trade sequence |
| Quote | Top-of-book quote-cross replay | Strategies that need bid/ask spread and L1 quote timing |
| Depth | Depth-aware book replay | Strategies whose fills depend on available depth and book interaction |
A fast Vector Study is discovery evidence, not execution evidence. A common workflow is to search broadly with Vector, retain a small set of candidates, then use Solo Runs or a smaller optimization at the supported higher fidelity. Changing fidelity can change the meaning of entries, exits, costs, and fills; it is not merely a slower version of the same result.
Analysis Depth and retained output
| Analysis Depth | Adds | Use it when |
|---|---|---|
| Summary | Point metrics and a bounded basic trade bootstrap from retained Run Output | You need fast screening or a lightweight baseline |
| Robust Statistics | Trade-order and stationary resampling, matched-random tests, Sharpe diagnostics, outlier checks, decomposition, and retained parameter sensitivity | You are narrowing candidates and need stronger statistical diagnostics without rerunning every scenario |
| Full Scenario Suite | Price-path permutations, synthetic markets, parameter neighborhoods, temporal folds, and execution scenarios through Strategy reruns | You are stress-testing a small finalist set and the selected runtime supports it |
Enable Chart Output when the retained finalists need the additional data for interactive charts. The full search still keeps lightweight ranking and sampler evidence; only the declared leading candidates retain the more detailed Run Outputs.
A practical research sequence
- Run a Solo Run with the intended baseline and inspect its decisions, costs, and output.
- Use Grid for a small interpretable space or Random for broad initial coverage. Keep the seed and the complete candidate denominator.
- Use Optuna TPE as the normal adaptive starting point when the search is large enough to benefit from guided trials. Choose a specialist sampler only when the search-space or objective shape justifies it.
- Inspect parameter neighborhoods and tradeoffs, not only the highest score.
- Add Walk Forward when the question concerns stability through time.
- Rerun a small finalist set at the highest supported Execution Fidelity that matches the Strategy's execution assumptions.
- Keep a separate final holdout or forward process for evidence that was not repeatedly used to redesign the Study.
Before queueing, select Estimate Study and review the candidate or trial count, folds, preparation plan, fidelity, optional dependency findings, resource estimate, and retained-output policy. Any material edit makes the estimate stale and requires a new estimate.
Next
Use Strategy Study Design for the full Strategy workflow or Portfolio Study Design for coordinated Portfolio research. When an optimization finishes, continue to Strategy Study Results or Portfolio Study Results.