AuK Operational

AuK is Tencent Hunyuan's open-source 1.5-billion-parameter foundation model for speech generation and editing. It uses a flow-matching diffusion architecture rather than an autoregressive approach. Through a unified natural-language instruction interface, the model supports 16 speech tasks, including zero-shot/instruction-based TTS, speech content editing, acoustic adjustment, paralinguistic editing, denoising, and voice separation. The model provides AuK-Flash, a fast version with 4-step inference that is approximately 4.5 times faster.

Zero-shot TTS: Clone a voice from a few seconds of reference audio and synthesize new speech.
Instruct TTS: Generate speech with specified vocal characteristics using only a textual description, without reference audio.
Speech content editing: Replace, insert, or delete words in a recording without changing the voice or intonation.
Lyrics editing: Rewrite lyrics while preserving the melody and vocal characteristics.
Pitch/speed/volume editing: Precisely adjust acoustic attributes in semitones, playback speed, and decibels.
Emotion and timbre editing: Change the emotional tone without changing the words, or preserve the content while converting the voice timbre.
Accent editing and denoising: Remove accents, reduce noise and reverberation, and improve clarity.
Paralinguistic event editing: Add or remove breaths, laughter, and coughs, enabling conversion between normal speech and whispers.
Speech separation: Split multi-speaker audio by speaking order, or extract a target speaker based on a content description.
Music separation: Extract vocals from songs and remove the accompaniment.

Follow WeChat and reply “开源” to join the AI open-source project discussion group
Overall paradigm: AuK uses a flow-matching diffusion architecture to generate latent representations of audio, making it naturally suited to editing tasks. It uses the latent representation of the original audio as the denoising starting point and modifies only the parts specified by the instruction, thereby better preserving the original timbre, melody, and unedited content. This is the technical foundation that enables it to cover the entire family of editing tasks.
Three-component pipeline architecture: The system consists of three parts: a multimodal large model reads natural-language instructions and reference audio and generates semantic conditions; an audio VAE encodes 24kHz waveforms into a 50Hz acoustic latent space; and the core hybrid Transformer consists of 10 layers of dual-stream MMDiT stacked with 20 layers of single-stream DiT. The model width is 1536, with a total of 1.5 billion parameters.
Large-scale instruction-alignment training: The training data contains approximately 3.03 billion “instruction-audio” pairs, totaling approximately 1.95 million hours of effective supervision. Crucially, all tasks are unified into the same “instruction + input audio → output audio” format, allowing the model to generalize to 16 tasks through a single interface.
Post-training and human preference optimization: After base training, the model undergoes human preference optimization and reinforcement-learning post-training (RL post-training) to improve the naturalness, instruction following, and perceptual quality of generated speech.

Confirm the hardware environment: Prepare an NVIDIA GPU with approximately 24GB of VRAM, such as an RTX 3090/4090.
Clone the code repository: Clone the Tencent-Hunyuan/AuK repository from GitHub to your local machine.
Install the dependencies: Configure the Python environment and install the required packages according to the repository instructions.
Download the model weights: Download the AuK weights and the accompanying VAE files from Hugging Face or ModelScope.
Download the semantic conditioning model: Download and configure the Qwen2.5-Omni-3B multimodal large model separately to parse instructions and reference audio.
Prepare the input materials: Prepare the target text, reference audio, or original audio file to be edited.
Write natural-language instructions: Write instructions based on the task requirements, such as describing vocal characteristics or specifying the sentence to replace or the pitch/speed to adjust.
Run inference: Enter the instructions and materials through the command line (CLI), execute the generation or editing process, and wait for the output.
Listen and adjust: Check the resulting audio. If unsatisfied, revise the instruction wording, sampling steps, or CFG strength and generate again.

Unified interface with broad task coverage: Through natural-language instructions, the model handles 16 speech tasks including TTS, editing, enhancement, and separation, without requiring a separate model deployment for each task.
Rare editing capabilities: Supports fine-grained editing such as changing words, inserting sentences, adjusting pitch, and modifying emotions, while preserving the original timbre and intonation as much as possible.
Instruct TTS without reference audio: Generate speech from a textual description of vocal characteristics, lowering the barrier to use.
Flow-matching architecture for greater fidelity: Uses diffusion-based generation rather than token-by-token prediction. During editing, the original audio serves as the denoising starting point, leaving unmodified parts with almost no degradation.

Project website: https://auk-project.github.io/
GitHub repository: https://github.com/Tencent-Hunyuan/AuK
Hugging Face model repository: https://huggingface.co/tencent/AuK
arXiv technical paper: https://arxiv.org/pdf/2609.08936

Dialogue repair for film and short-drama post-production: Directly replace individual words or sentences in a recording when an actor mispronounces a line or the script is slightly revised, without bringing the actor back to the studio, while preserving the original performance emotion.
Podcast and interview production: Remove verbal slips and filler, adjust speech pacing, eliminate background noise and reverberation, and turn rough-cut material into a finished episode in one step.
Audiobooks and AI voiceover production: Use Instruct TTS to generate differentiated voices based on character descriptions. The same narrator's emotion and intonation can also be adjusted in post-production, reducing the cost of producing multiple characters.
Music creation and derivative works: Rewrite cover-song lyrics while preserving the original melody and vocal characteristics, or extract vocals from songs as remix material.
Language training and accessibility applications: Remove accents and adjust speech rate and volume for users with accents or speech disorders, generating clearer and more accessible audio content.