InstructAV2AV – BAAI and Peking University’s Open-Source Joint Audio-Video Editing Model Operational

InstructAV2AV is a joint audio-video editing model released by the Beijing Academy of Artificial Intelligence and Peking University. With just one natural-language instruction, users can edit the video visuals and corresponding audio simultaneously during end-to-end generation, without manual masks or step-by-step processing. The model supports dialogue modification, character replacement, object insertion and removal, while preserving background and ambient sounds and maintaining audio-visual synchronization.

Identity-preserving speech editing: Replace only what a person says while preserving their appearance.
Audio-video instance replacement: Replace a specified person in the scene together with their voice and dialogue.
Instance insertion: Add a new object to the scene and automatically generate synchronized audio for it.
Instance removal: Remove a specified object and its corresponding sound from the audio track.
Compositional attribute editing: Independently modify and freely combine a person’s appearance, spoken content, and voice timbre.

Follow WeChat and reply “开源” to join the AI open-source project discussion group.
Source latent concatenation: The latent variables of the source video and source audio are concatenated with noise along the channel dimension, making the source content a structural condition throughout the denoising process. This constrains the model to preserve the background, irrelevant objects, and ambient sounds, avoiding the need to “modify one target while redrawing the entire world.”
SIGA gated attention: Source-Instruction Gated Attention enables each token to interact with both source features and instruction features. Learnable soft gates determine, position by position, how much to modify and how much to preserve, achieving a dynamic balance between content preservation and instruction execution.
Dual-stream interaction architecture: Based on Ovi’s symmetric dual-stream diffusion Transformer, the video and audio branches first model their respective modalities through self-attention, then exchange features through bidirectional cross-modal attention, allowing visual motion and sound events to mutually constrain each other and remain synchronized.
Two-stage training: Cross-modal attention is disabled to separately train single-modality editing capabilities, after which the complete architecture is restored for joint fine-tuning to learn fine-grained correspondences between audio and visuals.
Automated data pipeline: The InsAVE-80K dataset is built through three stages—material filtering, editing-target synthesis, and five-dimensional automated evaluation followed by human review—addressing the scarcity of paired training data.

Clone the code: Clone the InstructAV2AV code repository from GitHub to your local machine.
Set up the environment: Configure the Python environment and install the required dependencies according to the repository instructions.
Download the weights: Download the InstructAV2AV model weights from Hugging Face.
Prepare the source material: Prepare a video with audio as the input.
Enter an instruction: Enter a natural-language editing instruction describing what you want to modify.
Run inference: Run the inference script and wait for the model to generate the edited video and audio.
Save the results: Review the output and save the completed audio-video file.

Project website: https://hjzheng.net/projects/InstructAV2AV/
GitHub repository: https://github.com/suimuc/InstructAV2AV
Hugging Face model: https://huggingface.co/suimu/InstructAV2AV
arXiv paper: https://arxiv.org/pdf/2605.18467

Film and television post-production: Post-production professionals can replace actors’ dialogue and synchronize the lip movements, reducing the cost of reshoots.
Short-video creation: Short-video creators can modify characters and voices in their footage with a single instruction, improving creative efficiency.
Virtual humans: Virtual human operation teams can adjust digital characters’ appearance, voice timbre, and spoken content.
Advertising ideation: Advertising professionals can generate promotional videos featuring different characters and dialogue versions in batches.
Interactive content production: Game and education professionals can dynamically generate multimedia assets with consistent audio and visuals.