Data formats, Instruments, and canonical shapes

Start with meaning, not file extension

Arizmic imports heterogeneous source files and provider responses into immutable canonical Datasets. A supported extension means the source can be inspected by an adapter; it does not prove the selected schema, Instrument, timestamps, ordering, or event semantics are correct.

The safe chain is:

source bytes + adapter
  -> explicit mapping and Instrument/session resolution
  -> bounded typed normalization
  -> canonical partitions + quality/coverage evidence + checksums
  -> candidate Dataset version
  -> operator review and activation

Source bytes are never edited in place by import. Correct a bad conversion or mapping by producing a new reviewed source or Dataset version.

Supported local source families

The capability registry lists these source families for the Local Import platform. Actual support is adapter-version and schema specific. A folder is a collection of source files, not proof that coverage is non-overlapping or that every file belongs to the same Instrument.

Canonical shape summary

Choose the shape that describes the source record—not the downstream fidelity you want. A bar file cannot become trades because it has volume. A price-level depth file cannot become MBO because it has a sequence column.

Instrument contract

Every market Dataset resolves these governed fields:

The exact invariant is:

tick value = price increment × contract size

If all three are supplied and disagree, resolution blocks. Do not tune economics until a backtest looks familiar.

Keep source, contract, and root identity separate

A vendor is not a venue. A venue symbol is not globally unique. A continuous futures label is not an exchange-traded contract. Multi-contract import keeps each contract unchanged; roll policy belongs to a research view or Study and does not rewrite canonical rows.

Bar shape

Required canonical meaning:

• ts_start: inclusive interval start;
• ts_end: interval completion boundary;
• instrument_id;
• bar_type and interval;
• open, high, low, close;
• source_id; and
• volume/trade count only when the producer truly supplies those measures.

ts_start must precede ts_end, and the interval label must agree with the bounds. A single field named timestamp is ambiguous until the producer says whether it means bar open, close, or another label. Mapping close time as start time shifts the interval and can introduce look-ahead.

OHLC is not execution detail. Bar fidelity uses an authored fill model to reason within/around completed bars; the bar does not reveal individual trades, top-of-book state, queue position, or depth.

Trade / Tick shape

Required canonical fields:

• ts_event;
• instrument_id;
• executed price;
• executed size; and
• source_id.

Optional evidence can include ts_recv, exchange trade ID, aggressor side, classified bid/ask/unknown volume, sequence/channel facts, and source file/row.

Trade size is not quote size, order size, cumulative bar volume, or contract multiplier. Aggressor side is not book side. Multiple legitimate trades can share a timestamp and price; deduplication requires stronger event identity.

A trade stream can support trade/tick replay when ordering, continuity, Instrument, source roles, corrections and the selected execution contract all pass readiness. It does not provide contemporaneous spread or queue state.

Quote shape

Canonical quote observations carry:

• ts_event and optionally ts_recv;
• instrument_id;
• bid price and bid size;
• ask price and ask size;
• source identity; and
• genuine ordering/source fields where available.

One-sided quotes can be represented only under an explicit nullable-field and consumer contract. Crossed or locked states are quality facts; do not silently swap sides or widen the market.

L1/Quote fidelity needs a Prepared quote-replay contract, not merely a table with bid and ask columns. At the reviewed revision, the dedicated L1 replay and state-anchor contract remains planned/internal.

Depth Delta / MBP shape

A price-level delta describes a change to aggregated book state. Core fields include:

• ts_event, instrument_id, source_id;
• action and bid/ask side;
• price and size;
• is_snapshot or equivalent snapshot/reset boundary; and
• level, sequence, channel/publisher and receive time when required by the feed.

The adapter must define add/modify/delete/clear/reset semantics, zero-size behavior, level meaning, snapshot completeness, and ordering. Generic mapping selects a column; it does not translate arbitrary vendor integer codes into a safe action vocabulary.

MBP/L2 aggregates quantity at price levels. It cannot prove individual order identity or true FIFO queue position.

Order Book Event / MBO shape

The richer event contract can retain:

• event type, action and side;
• price, size and level;
• genuine order ID;
• sequence, channel ID and publisher ID;
• depth and source symbol/source Instrument;
• snapshot, reset, trade, fill and bad-book flags; and
• bounded source-specific context.

MBO/L3 requires actual order-level identity and lifecycle semantics. A price- level update does not become MBO because an adapter generates synthetic IDs. Current complete L3 Prepared/runtime/queue-evidence support remains planned and must fail closed.

Timestamp contract

All canonical market timestamps are stored at nanosecond precision in UTC.

The current flexible Local Import mapping has no integer unit selector. Convert epoch seconds, milliseconds, or microseconds in a new source before import:

Retain the original and conversion provenance. Check overflow and known earliest and latest UTC examples. A timestamp near 1970 or far in the future usually signals a unit problem before a timezone problem.

Time fields are not interchangeable

• ts_event: when the market/provider event occurred; primary event ordering.
• ts_recv: when the collection/feed system received it; latency and feed-order evidence, not market time.
• ts_start: inclusive Bar interval start.
• ts_end: Bar completion boundary.

Mapping receive time as event time adds transport latency to the market timeline. Mapping a bar close label to its start can make a Strategy consume future interval information.

Negative receive latency, out-of-order timestamps, and event/receive inversions are quality evidence. They are not repaired by relabeling fields.

Deterministic ordering and duplicates

Canonical event replay needs a total or sufficiently defined order. Use the strongest genuine evidence available:

event time -> provider sequence -> channel/publisher -> receive time
  -> source file -> source row

The exact precedence is adapter/Prepared-contract specific. Source row is provenance; it is not automatically exchange order. Do not invent a sequence, order ID, or tie-breaker.

A non_monotonic_timestamp Dataset warning can become a blocker for Tick, Quote, Depth or other deterministic consumers. If a derived sort is justified, create and audit it while retaining the original source/checksum.

Duplicates are semantic:

• trades: exchange trade/event ID where available;
• MBO: order ID plus event/action identity;
• feed events: provider sequence/channel/publisher/event identity;
• repeated input: exact source file and row provenance.

Never deduplicate by timestamp or matching price/size alone.

Side, size, price, and action rules

Required price/size fields must be positive where the shape demands it. A systematic price-versus-ticks disagreement usually indicates wrong scale, tick size, or Instrument identity.

Timezone, calendar, and session identity

Timezone answers how to localize source timestamps. Calendar answers which market session owns a correct UTC instant. They are separate.

Session-aligned workflows retain windows with:

• session ID/name;
• UTC start and end nanoseconds;
• calendar ID and definition version;
• session-profile ID and definition version; and
• a deterministic session-window fingerprint.

Sessions are sorted and non-overlapping. Changing a calendar/profile definition can make prior session-aligned Prepared Data stale even when the display label is unchanged. Epoch-aligned data cannot claim a session fingerprint, and session-aligned data cannot proceed without one.

Calendars affect missing-session checks, intraday grouping/resets, opening ranges, session VWAP, overnight rules, Study windows and futures roll timing. Do not change timezone to imitate session open or choose a nearby calendar to make validation pass.

Import validation and canonical quality

Sample validation checks a bounded subset for mapping/type/schema readiness. Full import accumulates wider evidence including:

• null required fields and invalid casts;
• timestamp bounds, monotonicity and ordering breaks;
• duplicate source-row keys;
• nonpositive price/size;
• OHLC and Bar interval contradictions;
• price/tick disagreement;
• classified-volume inconsistencies;
• negative receive latency and feed-order inversions;
• crossed/invalid book state and snapshot/reset problems; and
• source coverage/file overlap where supported.

Warnings remain evidence. Activation is an explicit review step and does not erase them. Downstream Study validation rechecks whether the exact Dataset can support the selected topology and fidelity.

Format-to-fidelity decision guide

Use the minimum fidelity that answers the question. More columns do not create better evidence when their source semantics or runtime are incomplete.

Technical normalization workflow

adapter discovers source units and fixed/flexible schema
  -> mapping binds canonical fields and source roles
  -> Instrument resolver joins manual, embedded, definition and confirmed-profile evidence
  -> timestamps normalize to UTC ns
  -> records validate and normalize in bounded batches
  -> deterministic partitions and quality accumulators publish atomically
  -> manifest/checksums/catalog candidate retain lineage
  -> activation makes the immutable version selectable

The mapping fingerprint, Instrument/profile fingerprint, source checksums, adapter/version, canonical schema, partitions, quality report, coverage and activation event are all needed for reproduction. A Parquet output alone is not the Dataset contract.

Next

Use Map Columns and Timestamps for step-by-step mapping, Resolve Instruments and Sessions for identity/economics/calendars, and Prepared Fidelity Layers and Blockers for the source roles required by each execution cell.