Control Super Mario Bros. with TypeSafe Jev and Structured Emulator Telemetry

What TypeSafe Mario Does

TypeSafe Mario is an experimental controller that lets TypeSafe's Jev model choose controller inputs for the original Super Mario Bros. Instead of receiving screenshots, the model receives compact, object-centric JSON derived from emulator telemetry and RAM.

The control loop follows this architecture:

NES emulator -> telemetry/RAM parser -> structured JSON -> Jev Choice -> controller input

After Jev selects a legal action, the emulator advances several frames and generates a new state. This process repeats throughout the episode. The raw local tile grid remains available in debug logs and the dashboard, but it is not duplicated in the model-facing input.

Available controller actions

The initial action space is intentionally limited to seven controller macros:

• noop
• right
• right_jump
• right_run
• right_run_jump
• jump
• left

Key Features

• Structured model input: The parser sends TypeSafe meaningful JSON objects rather than flattened prose or screenshots.
• Multiple judgments: Every request evaluates a controller choice, forward-jump usefulness, and immediate danger.
• Measured reaction timing: The state includes action duration and observation-to-action delay.
• Live dashboard: The default display combines gameplay with action probabilities, confidence, latency, danger, reward, and parsed state.
• Detailed run records: Every decision is saved as a JSON Lines record for later analysis or visualization.
• Debug visibility: Human-readable state text and the local tile grid remain available for inspection.

Requirements and Legal Setup

Before installing the project, make sure your environment satisfies these requirements:

• Python 3.13 or newer
• A TypeSafe API key stored in TYPESAFE_API_KEY
• A legal local setup for Super Mario Bros.

The repository does not contain a Nintendo ROM or other copyrighted game data. You are responsible for obtaining and using your emulator and game files lawfully.

Installation

The README provides PowerShell commands for creating a virtual environment, installing the Mario and development dependencies, and configuring the API key.

1. Create a Python 3.13 virtual environment.
2. Install the project in editable mode with the required extras.
3. Set the TypeSafe API key for the current PowerShell session.

py -3.13 -m venv .venv
.venv\Scripts\python -m pip install -e ".[mario,dev]"
$env:TYPESAFE_API_KEY = "your-key"

Replace your-key with your actual TypeSafe API key. Because the command sets a PowerShell environment variable, run subsequent commands from the same configured environment unless you persist the variable separately.

Inspect the Model Input First

Before launching the game, use state-demo to inspect the exact JSON and text that the project would send to TypeSafe:

.venv\Scripts\typesafe-mario state-demo

This command does not launch the game and does not call the API. It is the safest way to understand the model-facing schema and verify how emulator observations are represented.

Understanding the structured state

The canonical object groups observations by their purpose:

• player: Position, velocity, grounded state, jump phase, and power-up.
• trajectory: Airtime, distance since takeoff, and whether Mario is committed to crossing a gap.
• hazard: Up to three enemies, their projected positions, expected contact timing, and the takeoff deadline.
• terrain: Obstacle and gap geometry, observation reliability, and the last grounded preview.
• reaction_timing: Controller action duration and measured observation-to-action delay.
• recent_control: The previous action, its duration, progress gained, and observed outcome.
• episode: Lives, clock, progress, stalls, death state, and level completion.

The structured JSON is the canonical model state. The fuller text rendering exists only for debugging and the user interface. A debug snapshot can include Mario's coordinates and movement, episode progress, nearby enemies, and a local character-based tile grid.

Run World 1-1

Start the World 1-1 environment and let Jev choose an action every eight emulator steps:

.venv\Scripts\typesafe-mario play --env SuperMarioBros-1-1-v0 --frames-per-decision 8

The --env option selects the environment, while --frames-per-decision 8 determines how many emulator steps occur between model decisions. With the default display, the project opens one recordable window containing both the live game and model telemetry.

Use the dashboard

The dashboard presents the information needed to follow Jev's behavior:

• The selected controller action
• The complete Choice probability distribution
• Model confidence
• Jev request latency
• Forward-jump probability
• Immediate danger score
• Reward
• The parsed game state

Press R or click Restart to begin a fresh episode. The dashboard remains open after Mario dies or completes the level. Press Esc or Q to quit.

Choose a Display Mode

The default combined dashboard is useful when observing or recording a run. Two additional modes support narrower use cases.

Show only the emulator

.venv\Scripts\typesafe-mario play --env SuperMarioBros-1-1-v0 --frames-per-decision 8 --display game

The game display mode shows only the emulator window.

Run without a display

.venv\Scripts\typesafe-mario play --env SuperMarioBros-1-1-v0 --frames-per-decision 8 --display none

The none display mode runs headlessly and is intended for benchmark-style execution.

Understand the Three TypeSafe Judgments

Each request evaluates three independent judgments over the same structured state:

• Choice: Selects the next legal controller macro.
• Noul: Estimates whether a forward jump is useful at the current moment.
• Score: Measures immediate danger for the live visualization.

These judgments have distinct roles. The Choice result controls Mario, while the jump estimate and danger score provide additional interpretation and visualization.

Exact timing calculations remain in code. For example, the parser combines response age, enemy motion, action cadence, and jump-clearance time into a typed jump_must_start_this_decision fact. Jev interprets that fact and still owns the controller choice; the harness does not apply a scripted recovery-action override.

Inspect Recorded Decisions

Each decision is written to a timestamped JSON Lines file:

artifacts/run-<timestamp>.jsonl

The records include:

• Jev latency
• Action probabilities
• Confidence
• The canonical model state
• The raw debug state
• The game outcome

These records provide the data needed to inspect a run after execution. They can also serve as input for a live overlay or a rendered social clip.

Advanced Usage Tips

Validate observations before gameplay

Run state-demo before a live session when you want to review the model-facing representation without starting the emulator or consuming an API call. Pay particular attention to the separation between canonical structured state and human-oriented debug text.

Interpret timing facts alongside action cadence

The controller acts in multi-frame intervals, so reaction timing is part of the state rather than an external assumption. When reviewing a decision, compare the chosen action with response delay, action duration, projected enemy contact, and any takeoff deadline reported by the parser.

Use the appropriate display for the task

• Use the default dashboard when inspecting model reasoning signals and gameplay together.
• Use --display game when only the emulator output is needed.
• Use --display none for headless benchmark runs.

Analyze the full probability distribution

Do not limit analysis to the selected action. The dashboard and JSONL artifacts include the full Choice probability distribution and confidence, which can reveal whether Jev strongly preferred one action or selected among several similarly weighted alternatives.

Run Development Checks

The repository includes formatting, linting, and test commands for contributors. Run them from the configured virtual environment:

.venv\Scripts\ruff format --check src tests
.venv\Scripts\ruff check src tests
.venv\Scripts\python -m pytest -q

The first command checks formatting, the second runs Ruff's lint checks, and the third executes the test suite with concise output.

Conclusion

TypeSafe Mario demonstrates a structured alternative to screenshot-driven game control. It translates emulator telemetry and RAM into canonical, object-centric JSON, asks Jev to select from a compact set of legal actions, and records every decision for live inspection or later analysis. Start with state-demo, run World 1-1 through the dashboard, and use the generated JSONL artifacts to understand how state, timing, confidence, and outcomes relate.