Forward-Test Trading Classifiers with Calibrain Harness

What Is Calibrain Harness?

Calibrain Harness is an open-source forward-testing system for high-speed classification models. It connects a supported model to perpetual futures market data, asks labellable questions on each closed bar, converts the returned probabilities into expected value, and optionally executes qualifying trades on Bybit testnet only.

The project combines two related tools: a trading bot and a model-evaluation harness. Every prediction is saved and later compared with the market outcome, including predictions that did not produce trades. This separates model quality from the effects of fees, slippage, execution, and policy rules.

Important: Calibrain Harness is under active development and is research software, not financial advice. Its trading host is hard-coded to Bybit testnet, and there is no mainnet switch. Interfaces, defaults, and question sets may change between commits.

How the System Works

Instead of asking a model to answer “buy, sell, or hold,” the harness asks questions that can be labelled objectively from later price data. One example is which boundary will be touched first during a fixed horizon: a positive target or a negative stop.

1. Mainnet public market data, news, macro context, order-book information, and derivatives data are collected.
2. The feature builder creates a compact, unit-labelled state and adds historical base rates for the relevant instrument and bracket.
3. Kairos or Jev receives the state and seven typed, labellable questions.
4. The response is validated strictly. An invalid response is recorded as an error rather than interpreted as a negative prediction.
5. Model probabilities are blended with historical base rates through the calibration layer.
6. The policy calculates expected value in R, applies entry gates, and determines risk sizing.
7. If automatic trading is enabled, qualifying orders are sent to Bybit testnet.
8. Requests, responses, labels, runs, and trades are stored in SQLite for evaluation and reporting.

Signals and labels use mainnet data because testnet market data can be synthetic or divergent. Orders are nevertheless placed exclusively on testnet and sized according to its available book depth.

Key Features

Multiple Classification Models

The harness currently supports Kairos, provided by DMind through Minara Cloud, and Jev, provided by TypeSafe. Both use the same System One request style, question set, validation, policy, and evaluation pipeline.

Decisions, calibration weights, evaluations, and reports remain separated by model. A live model switch starts a new run and is permitted only while the bot is flat, preventing results from different models from being mixed.

Bloomberg-Style Terminal

The Next.js terminal receives live updates through one WebSocket. It presents charts, model signals, calibrated and raw expected value, positions, account information, an order book and depth view, market maps, news, liquidations, derivatives data, and a decision blotter containing each model request and response.

Built-In Calibration and Evaluation

Calibrain Harness blends model probabilities with instrument-specific base rates using the formula w · model + (1 − w) · base rates. The weight begins at 0.5 and is refitted every 15 minutes after a question accumulates at least 100 labelled outcomes.

Evaluation includes Brier and ranked probability scores, skill against climatology and supplied base rates, AUC, reliability, expected calibration error, probability integral transform analysis, EV ranking, consistency probes, execution quality, and per-run trading results.

Testnet Execution Controls

Entries use IOC limit orders capped relative to the best testnet price. Targets can rest as reduce-only limits, while managed stops are triggered using mainnet executable prices and closed through capped reduce-only IOC orders on testnet. The system also includes time stops, a mark-price backstop, a daily loss limit, position caps, portfolio risk limits, and a kill switch.

Prerequisites

Before installation, prepare the following:

• Python 3.12.
• Node.js 20 or newer with npm.
• A Kairos API key from Minara Cloud or a Jev API key from the TypeSafe console.
• Bybit testnet API credentials from Bybit testnet API management.

The Bybit key should have Contract permissions for orders and positions. Do not grant withdrawal permission.

Install Calibrain Harness

1. Clone the Repository

git clone https://github.com/zadescoxp/calibrain-harness.git
cd calibrain-harness

2. Create the Python Environment

Create a Python 3.12 virtual environment, install the dependencies, and copy the example configuration:

python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp .env.example .env

Add credentials to .env, or enter them later through the terminal’s KEYS command. The application reads keys from this file, which is gitignored and expected to have owner-only permissions. Keys are not stored in the database or sent in full to the browser.

3. Build the Terminal UI

cd web-next
npm ci
npm run build
cd ..

This creates the static terminal in web-next/out. The Python service serves the UI, API, and WebSocket, so a separate production Node.js process is not required.

4. Start the Service

.venv/bin/python -m ctb.server

Open http://127.0.0.1:8800. The server binds locally to 127.0.0.1 and refuses cross-origin writes.

If the UI has not been built, the address displays a short page containing the build command. The API and live WebSocket remain available.

Configure the Model and Credentials

Select Kairos or Jev with CTB_MODEL_PROVIDER in .env:

CTB_MODEL_PROVIDER=kairos
KAIROS_API_KEY=your_kairos_key

For Jev, use:

CTB_MODEL_PROVIDER=jev
TYPESAFE_API_KEY=your_typesafe_key
CTB_JEV_MODEL=jev-1.14.0

The documented Jev model is pinned to jev-1.14.0, although CTB_JEV_MODEL can override it. Kairos credentials must belong to a Kairos-enabled Minara Cloud group.

Add the Bybit testnet credentials using the variable names supplied by .env.example. Never commit .env or files under data/.

Run in Observe-Only Mode First

The bot starts without placing orders. In this mode, it continues collecting market data, calling the selected model, recording decisions, and labelling forecasts. This is the recommended starting point because it lets you verify connectivity and study model behavior without opening testnet positions.

1. Start the Python service.
2. Open the terminal at 127.0.0.1:8800.
3. Confirm that market data and model decisions appear.
4. Inspect errors and latency on the LOG and EVAL screens.
5. Enable AUTO-TRADE only when the setup is functioning correctly.

Use the Terminal

Main Screens

• MON, F1: Displays the watchlist, chart, model markers, account, positions, market map, order book, news, derivatives, flow data, and decision blotter.
• EVAL, F2: Shows calibration, findings, EV ranking, question scores, consistency probes, operational metrics, P&L, and execution quality. Results can be filtered by model, run, symbol, question set, and timeframe.
• TRADES, F3: Lists individual runs and trades, including reference prices, fills, slippage, and exact R.
• NEWS, F4: Presents headlines and Kairos news_return forecasts. News-return forecasting is available only with Kairos.
• LOG, F5: Displays system events and errors.

Command-Line Examples

Press / to open the terminal command line. Common commands include:

• BTC or ETH to load a symbol.
• ASK or PRB to query the model immediately and run consistency probes.
• 1M through 4H to change the chart timeframe.
• TF 5M to set the trading timeframe.
• UNIV to choose the symbol universe.
• POS 10 to set the maximum number of positions.
• KEYS to manage credentials.
• HELP to display terminal help.

You can switch between Kairos and Jev through MODEL in the function-key bar. The bot must be flat before switching.

Enable Testnet Trading Carefully

After observing decisions and validating the setup, use AUTO-TRADE in the command bar to permit testnet orders. Strict mode trades only when calibrated predictions pass all configured gates, including minimum expected value, setup quality, directional agreement, return-distribution agreement, venue divergence, position status, and short-side permissions.

Exploration mode is disabled by default. Setting CTB_EXPLORATION=1 allows the system to trade the model’s preferred side at reduced risk when strict gates block it. Its purpose is to collect real execution data, not to imply that the strict policy found a positive edge.

Testnet liquidity can be extremely thin and may behave differently from mainnet. The project’s early forward tests found that execution quality could dominate testnet P&L. Treat testnet results as research data rather than evidence of future profitability.

Use the Kill Switch

The terminal’s KILL button closes bot positions, cancels orders, and halts the system. The same action can be triggered from the project directory:

touch KILL

Positions opened manually on testnet are normally displayed but not managed by the bot. The kill switch is the documented exception.

Run Tests and Smoke Checks

Run the unit tests without network access:

.venv/bin/python -m pytest -q tests

Send one real request to the active model and validate its response:

.venv/bin/python scripts/kairos_smoke.py SOLUSDT

Target Jev explicitly:

.venv/bin/python scripts/kairos_smoke.py SOLUSDT --model jev

Test the minimum-size testnet execution path by opening a long position, verifying the stop, and closing it:

.venv/bin/python scripts/testnet_smoke.py SOLUSDT long

These smoke scripts make real external requests, so verify the selected symbol and credentials before running them.

Understand Important Configuration Options

Decision Timing and Brackets

• CTB_DECISION_TF defaults to 5, meaning decisions are made on closed five-minute bars.
• CTB_HORIZON_BARS defaults to 24, producing a two-hour horizon at the default timeframe.
• CTB_ATR_TF defaults to 15 for stop sizing.
• CTB_SL_ATR and CTB_TP_ATR default to 1.5 and 2.5 ATR.
• CTB_MIN_SL_BP and CTB_MAX_SL_BP clamp stop distances to 40–300 basis points by default.

Strict Policy Gates

• CTB_MIN_EDGE_R=0.15 sets the default minimum calibrated edge.
• CTB_MIN_SETUP_QUALITY=1.5 controls the setup-quality threshold.
• CTB_MIN_DIRECTION_MARGIN=0.03 requires calibrated direction to agree with the proposed side by the specified margin.

Risk Limits

• CTB_RISK_PER_TRADE_PCT=0.5 defines the intended percentage of equity lost if a stop is hit.
• CTB_MAX_POSITIONS, CTB_MAX_SAME_SIDE, and CTB_MAX_OPEN_RISK_PCT constrain portfolio exposure.
• CTB_MAX_LEVERAGE=3 and CTB_MAX_NOTIONAL_USDT=5000 provide hard caps by default.
• CTB_DAILY_LOSS_LIMIT_PCT=3 sets the default daily loss limit.

Costs and Execution

• CTB_TAKER_FEE_BP=5.5 and CTB_SLIPPAGE_BP=2 feed the cost model used for expected value and labels.
• CTB_MAX_ENTRY_SLIP_BP=10 sets the default entry-price cap.
• CTB_EXIT_MODE=managed enables the documented managed exit process.

Change one group of settings at a time and start a new run when comparing policies. Each run snapshots its configuration, making later analysis easier.

Advanced Evaluation Tips

Compare the Model with Its Baselines

A profitable-looking prediction is not automatically useful. Review whether the model beats climatology and the exact historical base rates included in its input. The calibration layer reduces the model’s influence when labelled outcomes show that it adds little information.

Separate Forecast Skill from Trading Results

Use EVAL to inspect all labelled decisions, not just trades. Forecast metrics measure the model’s probabilistic skill, while testnet P&L measures the combined effects of the model, policy, costs, risk sizing, and execution.

Account for Overlapping Decisions

Labels begin with the first one-minute bar after each decision, preventing look-ahead. If one bar touches both target and stop, the label counts it as a stop. However, consecutive decisions can have overlapping horizons, so raw sample counts may overstate statistical independence. The README recommends waiting for several hundred labelled decisions per symbol before drawing conclusions.

Inspect Consistency

Use repeat probes to measure determinism and mirror probes to determine whether reversing bullish and bearish market features appropriately changes the model’s answer. Also review long-versus-short coherence rather than relying on one isolated metric.

Export Reports

From EVAL, use EXPORT MINARA REPORT to write a Markdown report containing the evaluation results. Filter by model, run, symbol, question set, or timeframe before interpreting comparisons.

Manage Symbols and Runs

The UNIV command lists up to the top 100 USDT perpetuals by turnover that also have testnet listings. It includes depth and spread information and flags thin books and stock or commodity perpetuals. You can select up to 30 symbols, subject to the model request budget. The service restarts itself to apply a universe change.

Every decision and trade is assigned to an active run. To create a run programmatically, send POST /api/runs/start when no bot trade is open. Starting separate runs is useful when changing models, questions, risk rules, or execution settings.

Work on the UI

After pulling changes or editing the frontend, rebuild the static export:

cd web-next && npm run build

Reload the browser afterward; the Python server does not need to restart.

For live-reload development, keep the Python server running and start Next.js separately:

cd web-next && npm run dev

Open http://localhost:3000. The development UI communicates with the API and WebSocket on 127.0.0.1:8800.

Type-check the frontend with:

cd web-next && npx tsc --noEmit

Safety and Operational Practices

• Confirm that the host remains api-testnet.bybit.com; the project intentionally provides no mainnet trading switch.
• Begin in observe-only mode and review model errors before enabling testnet orders.
• Use Bybit credentials without withdrawal permission.
• Keep .env and the data/ directory out of version control.
• Monitor venue divergence, fill ratios, latency, slippage, and exit reasons rather than considering P&L alone.
• Keep the kill-switch procedure accessible before testing execution.
• Run the Python tests and frontend type-check before submitting changes.

Conclusion

Calibrain Harness provides a structured way to determine whether a fast classification model adds measurable information to perpetual-market forecasts. It combines objective questions, historical base rates, probability calibration, strict validation, testnet execution, and detailed evaluation in one local terminal.

Start by installing the service, configuring either Kairos or Jev, and collecting observe-only decisions. Once the data and model pipeline are stable, test execution cautiously on Bybit testnet, keep runs separate, and judge the model against its baselines over a substantial sample rather than relying on short-term P&L.