H3-World Operational

H3-World is an action-controllable world model released by Tencent in collaboration with the National University of Singapore and The Hong Kong Polytechnic University, fine-tuned from the 33B MiniMax-H3 video generator. The model translates keyboard operations into concise English instructions, injects them through the pretrained text pathway, and uses temporal attention routing to bind each instruction to its corresponding video frames, enabling precise temporally aligned control. H3-World uses only 8,000 game videos and a rank-32 LoRA to fine-tune 0.199% of the parameters, allowing the model to generate controllable visuals in real time based on keyboard input and generalize to unseen scenarios. It represents an important step toward interactive generative worlds.

Keyboard-to-video control: Converts keyboard-based character movement and camera operations into corresponding changes in video content, enabling an interactive world simulation where actions take effect immediately.
Precise temporal control: Executes different action instructions during different segments of the same video, precisely controlling when each action occurs.
Generalization across environments: Trained on game footage, the model generalizes to diverse visual environments such as neon cities, desert oases, and fantasy churches while maintaining stable scene content.
Compositional action generalization: Combines character and camera action primitives seen during training into novel action pairs that never appeared in the training data, enabling compositional control generalization.
Long-horizon generation: Supports longer continuous controllable video generation, with action instructions continuously driving the evolution of the scene.

Follow WeChat and reply “Open Source” to join the AI open-source project group
Action language representation: Translates the keyboard operations at each frame into a short English instruction, such as “strafe left,” representing actions in natural language.
Reuse of the pretrained text pathway: Instructions are injected through MiniMax-H3’s original text encoding path, directly reusing the semantic representations learned during video pretraining. This allows the model to understand action meanings naturally without learning an action encoding from scratch.
Temporal attention routing: A targeted attention mask binds each instruction to its corresponding video latent frames, restricting the instruction’s effect to a predefined time interval and preventing control leakage between adjacent actions. This is the key to precise temporal control.
Lightweight LoRA fine-tuning: A rank-32 LoRA is trained only on H3’s self-attention projections. Adaptation is completed with 0.199% trainable parameters, 8,000 game videos, and 10,000 optimization steps, substantially reducing the cost of converting a video generator into a world model.

Method 1: Online experience
Method 2: Local deployment
Open the interactive demo Space on Hugging Face (hugging-apps/h3-world-action-demo) to experience keyboard-driven control without downloading the 135GB weights.
Upload or select an initial-frame image and enter a scene-description prompt.
Provide control instructions through preset actions or keyboard input, including character movement and camera operations.
The model generates controllable video clips of approximately 5.2 seconds at 832×480 in real time.
Environment setup: Create a conda environment with Python 3.10 and CUDA 12.8, then install PyTorch 2.10 and the dependencies in requirements.txt.
Get the code: Clone the official H3-World repository and clone DiffSynth-Studio into the specified directory, switch to the specified commit, and apply the official attention patch. This step is required; otherwise, the LoRA will be ineffective.
Download the weights: Download the MiniMax-H3 base weights from Hugging Face, approximately 135GB, and the H3-World LoRA (step-10000.safetensors), placing them in the corresponding repository directories.
Configure the cache: Run env.sh to direct the Hugging Face, Torch, and Triton caches to locations inside the repository.
Run inference: Provide a first-frame image, static semantic conditions, and a timetable-based action sequence, with each segment corresponding to a character or camera instruction.
Generate results: The model generates the corresponding temporal video latents according to the action timetable and decodes them into output video. Longer horizons can be extended autoregressively.

Elegant design with no additional modules: Reuses the pretrained text pathway, turning the action-control problem into a language-understanding problem the model is already good at solving.
Extremely lightweight training cost: Uses only 8,000 game videos and 10,000 optimization steps to fine-tune 0.199% of the 33B model’s parameters with a rank-32 LoRA, unlocking interactive control capabilities.
Precise temporal control: Temporal attention routing allows different actions to be executed according to a timetable within the same video, addressing the inability of global prompts to precisely control action timing.
Significantly better than traditional conditioning methods: Compared with direct action-conditioning methods such as additive bias and FiLM, language grounding produces coordinated character and camera motion without disrupting scene content.
Powerful compositional generalization: Combines action primitives seen during training into novel action pairs and generalizes to out-of-distribution visual environments.
Highly versatile: A single model handles multiple control types, including first- and third-person perspectives, character movement, and camera motion, without separate training for each task.

Official website: https://danzer1xxxxchan.github.io/H3-World/
GitHub repository: https://github.com/Danzer1xxxxChan/H3-World
Hugging Face model repository: https://modelscope.cn/models/DANNY621/H3-World
Technical paper: https://modelscope.cn/papers/2609.01560

Game prototyping: Designers can enter an initial scene and immediately “play” the virtual environment in real time, quickly validating level concepts and cinematic language without first building a complete game engine.
AI agent training sandbox: Provides embodied agents with an interactive virtual environment. Agents explore the world through keyboard actions and receive visual feedback, reducing the cost of real-world training.
Film and animation previsualization: Directors control virtual camera movements with the keyboard, including panning, tracking, and raising or lowering, to generate storyboard previews in real time and significantly shorten the early previz production cycle.
Robot manipulation strategy validation: Maps the movement instructions of robotic arms or drones to action language and tests the temporal validity of action sequences in the generated simulation environment.
Immersive content creation: Combined with VR and game interaction, it serves as a UGC tool that lets users drive the evolution of personalized virtual worlds through natural key presses, allowing anyone to generate “playable videos.”