Ultimate SFighterAI Tutorial: Train an Agent to Beat Street Fighter II with PPO

Project Overview

Ultimate SFighterAI is a deep reinforcement learning project for Street Fighter II: Special Champion Edition. It uses Ryu as the player character, trains a NatureCNN model with the PPO algorithm, and extends training across all 12 stages.

The model's decision input comes entirely from the RGB pixels of the game screen. It does not directly read character positions, action states, or health. The training environment uses only the agent_hp and enemy_hp variables in memory to calculate rewards and determine wins and losses.

The project provides trained model weights and training records, so you can watch the AI fight immediately, reproduce the experiment, adjust the number of parallel environments, or continue training from an existing model.

Core Features and Implementation Details

• All 12 stages: Each stage runs in an independent emulator, while all stages jointly train the same NatureCNN model.
• Vision-only decisions: The model selects movement, attacks, and special moves based on processed game frames.
• 27 discrete actions: Includes neutral, movement, normal attacks, crouching attacks, and moves such as Hadouken, Tatsumaki Senpu-kyaku, and Shoryuken facing either direction.
• PPO reinforcement learning: Training is based on the PPO algorithm from Stable-Baselines3.
• Multiple evaluation modes: Supports testing with per-stage save states, best-of-three matches, single-KO evaluation, and continuous progression starting from the first stage.
• Pretrained model: The default weights are located at logs/trained_models/ppo_ryu.zip.
• Training visualization: TensorBoard can be used to view reward changes and model update records.
• Multi-device support: Training and inference support auto, cpu, cuda, and mps.

Frame Preprocessing

The original game frames are center-cropped, processed to extract motion foregrounds, and resized to produce an 84×84×3 model input. Motion foreground extraction uses grayscale differences from three points in time, t-16, t-8, and t. Background brightness is multiplied by 0.35, after which the R, G, and B channels are combined. This processing helps the model perceive character movement and attacks.

Reward Mechanism

Rewards are primarily based on changes in both characters' health, knockouts, and time costs. Damage rewards are calculated as follows:

0.01 × (3 × opponent health lost − own health lost)

The reward for knocking out the opponent is:

0.01 × (176 + own remaining health)

The penalty for being knocked out is:

0.01 × (176 + opponent remaining health)

In addition, each executed action incurs a time penalty, with a default value of 0.0005.

Project Directory

• data/: Stores opening save states for the 12 stages, game memory variable definitions, and scenario configuration.
• main/train.py: Handles parallel training across all stages and continued training.
• main/test.py: Handles per-stage testing and continuous-progression evaluation.
• main/env_utils.py: Handles ROM configuration and environment construction.
• main/custom_wrappers.py: Implements frame processing, action mapping, and reward calculation.
• logs/trained_models/ppo_ryu.zip: The trained model provided by the project.
• logs/: Contains TensorBoard records from the training process.
• requirements.txt: Python dependency list.

Installation and Environment Setup

Step 1: Prepare the Python Environment

Python 3.10 is recommended. The environment documented in the README was macOS with Apple Silicon and Python 3.10.20; both the CPU and Apple MPS paths were verified. For other platforms, refer to the official installation instructions for Stable-Retro and PyTorch.

Create and activate a virtual environment in the project root, then install the dependencies:

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Windows users can activate the virtual environment with:

.venv\Scripts\activate

You can also use Anaconda to create a Python 3.10 environment and then install the dependencies in requirements.txt. The project's main dependencies are Gymnasium, Stable-Retro, and Stable-Baselines3.

Step 2: Prepare the Game ROM

Stable-Retro does not provide game ROMs, and this project does not include a ROM. Obtain a supported US ROM of Street Fighter II: Special Champion Edition through legal means.

After obtaining the ROM, provide its absolute path through an environment variable. Run the following on macOS or Linux:

export SF2_ROM_PATH="/absolute/path/to/your/rom.md"

Windows PowerShell:

$env:SF2_ROM_PATH = "C:\path\to\your\rom.md"

Alternatively, place the ROM at the project's default location:

data/street-fighter-ii-special-champion-edition-usa.md

The training and testing programs will import the ROM automatically. If the ROM has already been imported into Stable-Retro in the current Python environment, you can run the programs directly.

Step 3: Verify the Stage Data

The 12 .state files in the data/ directory store the opening state for stages 1 through 12. data.json and scenario_full_frame.json define the game memory variables and scenario configuration. The program reads these files directly; you do not need to copy them to the Stable-Retro installation directory.

Run the Pretrained Model

Test All 12 Stages Individually

After completing the environment setup, run the following from the project root:

python -m main.test --episodes 1

The program loads the opening save state for stages 1 through 12 in sequence. Each stage uses best-of-three matches by default. --episodes 1 performs one complete evaluation round; if this argument is omitted, the default is 10 rounds.

If the model fails a stage, the program continues testing the remaining stages. The terminal outputs the wins and losses, rewards, stage win rates, and the proportion of evaluation rounds in which all 12 stages were won.

Disable Rendering to Focus on Evaluation Results

If you do not need to watch the matches, disable rendering and explicitly use CPU inference:

python -m main.test --episodes 1 --no-render --device cpu

Disabling rendering is suitable for batch evaluation and reduces the overhead caused by display output.

Progress Continuously from the First Stage

To observe the AI challenge the next opponent according to the game's normal progression, run:

python -m main.test --episodes 1 --no-load-level-saves

This mode loads a save state only at the start of the first stage, after which the game advances through the stages normally. Once the model loses, the current attempt ends. The program outputs the win rate for stages reached and the full-completion rate.

In per-stage testing, “all stages won” means that the model passed evaluation on all 12 independent stage save states. “Full-completion rate” requires the model to clear all 12 stages in one continuous run. These metrics have different meanings.

Common Testing Options

• --model-path PATH: Specifies the PPO model. The default is logs/trained_models/ppo_ryu.zip.
• --episodes N: Sets the number of per-stage evaluation rounds or continuous-progression attempts. The default is 10.
• --no-render: Disables the game display.
• --no-load-level-saves: Does not load save states for each stage; instead, starts from the first stage and progresses continuously.
• --reset-round: Uses a single KO as the evaluation unit in per-stage testing instead of the default best-of-three format.
• --device auto/cpu/cuda/mps: Selects the model inference device. The default is auto.
• --speed N: Speeds up visualization by reducing the number of displayed frames. The default is 4.

Train Your Own Model

Start with a Small-Scale Configuration

The default training configuration requires substantial resources. For the first run, start one emulator per stage and use a smaller batch:

python -m main.train --envs-per-level 1 --batch-size 256 --log-dir runs/ryu

Because the project contains 12 stages, --envs-per-level 1 runs a total of 12 parallel emulators. First verify that the ROM, environment construction, device, and log output all work correctly, then gradually increase the number of parallel environments.

Understand the Default Training Scale

When you run python -m main.train directly, the default is 64 emulators per stage, for a total of 768 parallel environments. Combined with the default n-steps=128, one PPO update collects 98,304 samples, and the full training target is 700 million steps.

The default configuration requires substantial CPU and memory resources and is not suitable as a first test on an ordinary computer. Adjust --envs-per-level, --batch-size, or the total number of training steps according to your hardware.

Common Training Options

• --envs-per-level N: Number of parallel environments per level; defaults to 64 environments per level when unspecified.
• --total-timesteps N: Total training timesteps for this run; when resuming, this indicates the additional timesteps, with a default of 700000000.
• --n-steps N: Number of steps collected by each emulator before an update, with a default of 128.
• --batch-size N: PPO update batch size, with a default of 8192.
• --n-epochs N: Number of training epochs per batch of sampled data, with a default of 3.
• --learning-rate N: Learning rate, with a default of 0.0002.
• --device auto/cpu/cuda/mps: Selects the training device, with a default of mps.
• --log-dir PATH: Output directory for the model and training curves, defaulting to logs/ALL_MOTION_SIMULTANEOUS.

Resume Training from Pretrained Weights

The project supports resuming training from the provided PPO weights. The following command adds 1 million training steps and writes the results to a new directory:

python -m main.train \
  --resume-model logs/trained_models/ppo_ryu.zip \
  --envs-per-level 1 --batch-size 256 \
  --total-timesteps 1000000 --log-dir runs/ryu-resume

When resuming, --total-timesteps specifies the additional training steps for this run, rather than a cumulative target that includes the previous model's training.

Viewing Training Curves with TensorBoard

The project includes training logs. Start TensorBoard with:

tensorboard --logdir=logs

Then open http://localhost:6006/ in a browser to view reward changes and model update curves.

To view logs generated by your own training runs, simply replace the directory:

tensorboard --logdir=runs/ryu

Advanced Usage Recommendations

1. Verify before scaling up: Start with one environment per level and a short training run. Confirm that ROM imports, save-state loading, and log writing work correctly before increasing the number of parallel emulators.
2. Select the device for your platform: Apple Silicon users can try mps; environments with compatible NVIDIA GPUs can use cuda; otherwise, use cpu or auto. The README was validated on macOS with Apple Silicon; other platforms must be configured according to the requirements of PyTorch and Stable-Retro.
3. Distinguish between the two evaluation modes: Per-level save-state testing is suited to analyzing each opponent's performance, while continuous progression is closer to the complete game flow. Specify the mode used when reporting results.
4. Use headless batch evaluation: Add --no-render when running multiple rounds for statistical analysis, and enable rendering when observing the policy's performance.
5. Save experiments in separate directories: Use --log-dir to create directories for different parameter combinations. This prevents training records from overwriting one another and makes comparisons in TensorBoard easier.
6. View all command options: Both the training and testing scripts provide help information.

python -m main.train --help
python -m main.test --help

In addition to the module form, the project also supports executing the scripts directly:

python main/train.py
python main/test.py

Conclusion

Ultimate SFighterAI demonstrates how pixel input, action mapping, reward shaping, PPO, and parallel environments across multiple levels can be combined into a complete fighting game agent. The most reliable way to get started is to configure the ROM and run the pretrained model first, then reproduce the training with low parallelism, and finally expand the scale of experiments step by step through continued training, device selection, and TensorBoard analysis.

The project is released under the Apache License 2.0 and uses open-source components including Gymnasium, Stable-Retro, and Stable-Baselines3.