Skip to content

Run Model

What is a run?

A run is a single, identifiable execution of a trading workflow in either backtest or paper mode. Each run is defined by a spec and produces traceable metadata, journal entries, and artifacts. Example specs are in examples/configs/.

Core concepts

  • run_id: unique identifier for a run instance
  • run spec: declaration of mode, instrument, scenario identity metadata (strategy), and run configuration
  • immutable inputs: exact config/data/code references used for that run
  • metadata: operational facts (timestamps, status, mode, identifiers)
  • hashes: fingerprints for config/data/code inputs to support reproducibility checks
  • artifacts: run outputs in a stable layout
  • journal: operational event log for the run lifecycle
  • report: concise summary of outcome, checks, and notable events

Artifact contract (current implementation)

For a given run_id, the local artifact contract is:

  • artifacts/runs/<run_id>/run_spec.yaml
  • artifacts/runs/<run_id>/metadata.json
  • artifacts/runs/<run_id>/journal.jsonl
  • artifacts/runs/<run_id>/metrics.json
  • artifacts/runs/<run_id>/report.md

Optional/additional outputs depending on commands:

  • artifacts/runs/<run_id>/connectivity_readiness.json
  • artifacts/runs/<run_id>/connectivity_probe.json
  • artifacts/runs/<run_id>/reconciliation_result.json
  • artifacts/runs/<run_id>/drills/*.json
  • artifacts/runs/<run_id>/metrics.prom (when exported to file)

Evidence comparison artifacts are cross-run outputs (not standalone runs):

  • root: artifacts/evidence/
  • pair directory: artifacts/evidence/<backtest_run_id>__<paper_run_id>/
  • files:
  • backtest_vs_paper_evidence.json
  • backtest_vs_paper_evidence.md

Evidence compare reads from the two run directories and writes only under artifacts/evidence/...; it does not mutate compared run directories.

Backtest vs paper evidence schema (high level)

backtest_vs_paper_evidence.json includes operational comparison fields such as:

  • schema_version
  • comparison_status
  • known_gaps
  • journal_summary
  • artifact_presence
  • compared_fields

The markdown companion report summarizes those same operational findings in operator-readable form.

Why reproducibility and traceability matter

  • Reproducibility allows the same run inputs to be rerun with consistent expectations.
  • Traceability allows operators to answer what ran, when, with which inputs, and why a result occurred.
  • Together they support practical debugging, review, and drill validation.

Current strategy/scenario contract

  • The current backtest path is a Nautilus engine smoke run over prepared 1-minute OHLCV candles.
  • strategy.name and strategy.version are used as scenario identity/traceability metadata.
  • Backtest currently supports one built-in scenario mapping: ops_smoke_demo.
  • The backtest path registers the built-in Nautilus strategy for ops_smoke_demo.
  • Scenario execution records deterministic operational counters (strategy_registered, bars_seen, deterministic_action_triggered, orders_submitted, fills_count) in metadata.json, metrics.json, journal.jsonl, and report.md.
  • Scenario metrics are artifact-backed: tc metrics export --run-id <run_id> renders a one-shot Prometheus text snapshot; tc metrics serve exposes the same renderer over HTTP for Prometheus/Grafana.
  • ops_smoke_demo does not submit orders (orders_submitted = 0, fills_count = 0).
  • No PnL/Sharpe/returns/profitability/alpha performance reporting is included in the current scenario contract.
  • Current runs do not dynamically load custom strategy modules from config.
  • strategy.name values such as ops_smoke_demo identify local smoke/demo intent, not alpha claims.

Current reserved config fields

  • venue is a RunSpec venue label in current examples. binance on backtest maps to Nautilus test-instrument/local fixture context, and binance_testnet on paper/readiness is a future probe label only; neither implies active exchange/testnet/live connectivity today.
  • data.fingerprint is stored for metadata/traceability and is not a runtime gate yet.
  • observability.journal, observability.metrics, and observability.report are stored in metadata; current lifecycle paths still write standard artifacts regardless of toggle values.
  • connectivity_readiness is a reserved local-only contract for readiness metadata (env var names only).
  • The connectivity_readiness block participates in config hashing.
  • RunSpec parsing does not perform env lookup and does not perform network calls.
  • Local env presence evaluation runs via tc connectivity readiness --spec <path> (artifact-backed preflight only).
  • The readiness path writes connectivity_readiness.json, patches metadata["connectivity_readiness"], appends connectivity_readiness_evaluated to journal.jsonl, and updates report readiness section only if report.md already exists.
  • No env var values are stored in readiness artifacts, metadata, journal, or reports.
  • Readiness state set is finite and local-only: disabled, missing_credentials, configured, invalid_config, unknown.
  • Readiness metrics are artifact-backed from connectivity_readiness.json:
  • tradingchassis_ops_lab_connectivity_readiness_state
  • tradingchassis_ops_lab_connectivity_readiness_enabled
  • tradingchassis_ops_lab_connectivity_readiness_missing_required_env_total
  • tradingchassis_ops_lab_connectivity_readiness_probe_performed
  • Local loopback connectivity probe runs via tc connectivity probe --spec <path> --url <loopback-url>.
  • Probe validates local-only loopback HTTP targets and executes read-only GET only.
  • Probe writes connectivity_probe.json, patches metadata["connectivity_probe"], appends connectivity_probe_evaluated, and updates report probe section only if report.md exists.
  • Probe states are local artifact outcomes: probe_ok, probe_http_error, probe_timeout, probe_unreachable, probe_unknown.
  • Probe does not store response body and does not perform external exchange/testnet/live connectivity.
  • Probe metrics are artifact-backed from connectivity_probe.json:
  • tradingchassis_ops_lab_connectivity_probe_state
  • tradingchassis_ops_lab_connectivity_probe_performed
  • tradingchassis_ops_lab_connectivity_probe_latency_seconds
  • tradingchassis_ops_lab_connectivity_probe_http_status
  • Metrics caveat remains: the artifact-backed metrics renderer requires metrics.json (applies to both tc metrics export and tc metrics serve); probe-only init+probe runs do not create metrics.json.

Metrics serve vs export

tc metrics export and tc metrics serve use the same artifact-backed renderer.

  • export is a one-shot inspection path (stdout or file).
  • serve is the HTTP path used by Prometheus/Grafana (/metrics).
  • serve does not require a prior export; export does not prepare metrics for serve.
  • serve can include aggregate evidence metrics when --evidence-root points at evidence artifacts.
  • Evidence metrics are aggregate and artifact-backed (no run IDs, run pairs, or config hashes as evidence labels).

Concrete examples

  • Backtest example: examples/configs/btcusdt_backtest.yaml
  • Paper example: examples/configs/btcusdt_paper.yaml

For the end-to-end runnable sequence, see Quickstart and Demo Flow.

For an inventory of how artifact failures, safety blocks, probe outcomes, and drill results surface through artifacts, metrics, and dashboard signals, see Failure modes.