DuoMatching Operational

DuoMatching is a few-step video generation distillation framework jointly developed by CUHK MMLab and ByteDance. While the video teacher ensures temporal consistency, DuoMatching introduces a frozen image model to supervise per-frame visual quality and semantic alignment. It connects latent spaces through LatentBridge and uses LVS to select key supervision frames, improving visual details and prompt adherence. DuoMatching adds no inference overhead, and its 2-step generation achieves a total VBench score of 83.53.

Improve per-frame visual quality: Addresses blurry or missing details such as hair, textures, and materials in few-step video generation.
Enhance semantic alignment: Improves the model's ability to follow prompts and reduces issues such as missing objects and incorrect colors.
Maintain temporal performance: Optimizes per-frame quality while largely preserving cross-frame consistency and motion capabilities.
Support few-step generation: Supports low-latency video generation in 2 steps, 4 steps, and more, meeting the needs of real-time or interactive applications.
Add no inference overhead: The image teacher is used only during training, so the deployed student model does not need to run an additional image model.

Follow WeChat and reply with “开源” to join the AI open-source project discussion group
Joint and marginal distribution matching: In addition to conventional joint DMD, DuoMatching adds marginal DMD. The video teacher matches the joint distribution of the entire video to preserve temporal and motion quality, while the frozen image teacher matches the marginal distribution of sampled frames to improve per-frame visual quality and semantic alignment. The two losses jointly train the student model, with a default marginal loss weight of 0.4.
LatentBridge connects video and image latent spaces: Because the latent structures of video VAE and image VAE are incompatible, DuoMatching uses LatentBridge, with approximately 10.75 million parameters, to directly map video latent slices into the image teacher's latent space. The module is pretrained with an L1 reconstruction loss, avoiding the high memory cost of decoding to RGB and then encoding again, and enabling the image teacher to effectively supervise the video student.
LVS optimizes frame-level supervision sampling: Latent Variation Sampling divides video segments according to the variation intensity between adjacent latent slices, then selects supervision frames from each segment. By default, 4 latent slices are sampled per video. Compared with random sampling, this reduces the concentration of supervision on static scenes and allocates the limited frame-level supervision budget to more dynamic content.

Prepare the runtime environment: Prepare an NVIDIA GPU with CUDA support and configure a Python 3.10 environment.
Download and install the project: Clone the DuoMatching repository and install PyTorch, FlashAttention, and the dependencies required by the project.
Download the model files: Obtain the Wan2.1-T2V-1.3B base components and the official DuoMatching weights, and place them in the repository's default directories.
Prepare prompts: Create a UTF-8 text file with one prompt per line, or use the official prompts/demos.txt directly.
Start video generation: Run inference.py and specify the inference configuration, model weights, prompt file, and output directory in the command.
View the generated results: Retrieve the videos from the output directory. The official example generates 21 frames by default; the output frame count and random seed can be configured.

Significantly improved per-frame visual quality: Independent frame-level supervision improves the loss of details such as hair, textures, and materials in few-step generation.
More accurate prompt adherence: Image-model semantic priors reduce missing objects, incorrect colors, and composition deviations.
No added inference overhead: The image teacher is used only during training, so the deployed student model requires no additional inference steps or computation.
Balances visual quality and motion capability: While improving visual quality and semantic alignment, it largely preserves temporal consistency and motion performance.
Can benefit continuously from advances in image models: Replacing the image teacher with a stronger model may further improve the student model, and typically requires retraining the corresponding LatentBridge.

Official website: https://johnzhan2023.github.io/DuoMatching/
GitHub repository: https://github.com/JohnZhan2023/DuoMatching
Hugging Face model repository: https://huggingface.co/JohnZhan/DuoMatching
arXiv paper: https://arxiv.org/pdf/2610.03543

Real-time interaction and livestream generation: Use its few-step, low-latency capabilities to quickly generate motion and expression videos for digital humans, virtual streamers, and interactive livestreams.
Advertising and e-commerce content production: Improve the accuracy of generated product materials, colors, and specified objects, reducing inconsistencies between advertising visuals and creative prompts.
Short dramas and animation previsualization: Quickly generate storyboards and animated samples, helping creators validate character movements, shot design, and visual effects at low cost.
Game and virtual world content generation: Provide rapid video prototypes for game cutscenes, character movements, and virtual environments, improving the generation efficiency of interactive content.
High-concurrency video generation services: Improve the visual quality of few-step models without additional inference overhead, making them suitable for platforms that need to balance generation speed, service costs, and output quality.