Nemotron 3.5 Transcribe Operational

Nemotron 3.5 Transcribe is an open-source speech model released by NVIDIA. It uses a speech encoder and MoE architecture with approximately 4.3 billion active parameters, supports 40 languages, and can process recordings up to 1 hour long with as many as 8 speakers. Its capabilities include verbatim transcription, speaker diarization, and translation. Nemotron 3.5 Transcribe supports customizing the output format with natural-language instructions, adapting to specialized fields through fine-tuning, and free commercial use.

Multilingual transcription: Supports 40 languages and converts recordings into text while identifying the languages spoken in the audio.
Speaker diarization: Distinguishes up to 8 speakers and attributes each sentence to the correct speaker, making it suitable for meetings and multi-person interviews.
Long-audio processing: Processes recordings up to approximately 1 hour in a single request.
Instruction-based transcription: Use natural language to specify the output format, such as verbatim transcripts, polished text, filler-word removal, and terminology conventions, without retraining.
Speech translation: Translates between 31 languages and English.
Word-level timestamps: Returns the corresponding audio position for each word, making playback and search easier.
Domain fine-tuning: Based on the NeMo Speech toolkit, it can quickly adapt to specialized scenarios on a single GPU with a small amount of data.

Follow WeChat and reply “Open Source” to join the AI open-source project community.
Speech-Augmented Language Model (SALM): The model consists of three components. A Transformer speech encoder extracts audio features, while a frozen Nemotron 3 speaker diarization model provides speaker activity information. Both feed into the Nemotron 3.5 Lightning 30B-A3B hybrid Mamba-Transformer MoE large language model. The model has approximately 34 billion total parameters, but only about 4.3 billion are activated during each inference step, balancing capability and efficiency.
Parallel expert encoder: Speaker activity signals from the frozen diarization model are injected into the ASR encoder representations. This allows the model to perform speaker-attributed transcription within a single LLM architecture without architectural changes, using serialized output training.
Supervised balanced mixture: Training data is sampled by answer-token count rather than sample count. Each ASR sample can be paired with multiple task instructions, allowing new capabilities to be added with a sampling rate of less than 1%. Training consists of supervised fine-tuning followed by a short reinforcement learning stage that directly optimizes transcription accuracy.
MTP speculative decoding: The model is jointly optimized for multi-token prediction (MTP) during training, allowing its built-in MTP layer to serve directly as the draft model for vLLM speculative decoding. A separate DFlash2 draft model is also provided as an alternative. Together, they can achieve a throughput of 2,238 seconds of audio transcribed per second on a single RTX PRO 6000 Blackwell GPU.
CTC timestamp adapter: After decoding, a lightweight CTC adapter aligns the generated text with the audio, producing speaker-attributed word-level timestamps for long audio and heavily overlapping conversations without adding workload to the main model.

Prepare hardware: The BF16 version requires approximately 80 GB of VRAM. FP8/NVFP4 and GGUF versions are available for lower hardware requirements.
Get the model: Search for NVIDIA-Nemotron-3.5-Transcribe-30B-A4B on Hugging Face or ModelScope, then download the weights under the OpenMDW-1.1 license.
Set up the environment: Choose one of three deployment options: Nemotron Transcribe NIM (official container), vLLM (with MTP acceleration), or NeMo-Speech.cpp (local GGUF execution).
Prepare audio: Upload a recording up to 1 hour long and convert it to the format required by the selected framework.
Write a prompt: Describe the task in natural language, such as “preserve the exact spoken words and label speakers” or “remove filler words and translate into English.”
Run inference: Submit the audio and prompt through the API. The model returns the transcription, optionally with speaker labels and word-level timestamps.
(Optional) Try it quickly: If you do not want to deploy the model, try the nvidia/nemotron-transcribe demo space on Hugging Face online.
(Optional) Fine-tune for a domain: Use the NeMo Speech toolkit to prepare labeled data. Adapt the model to your accents, terminology, and formatting in approximately 0.5–2.3 hours on a single GPU.

Instruction-based output: Switch between verbatim transcripts, polished text, translations, and other formats with a single natural-language instruction, without retraining or configuring multiple models.
High transcription accuracy: Achieves an average WER of just 4.71% on the English Open ASR leaderboard and 4.62% on the multilingual leaderboard, ranking first among open-weight models in both.
Leading speaker diarization: For diarized transcription of overlapping multi-speaker audio, it surpasses specialized cascaded systems for the first time while supporting scenarios of up to 1 hour and 8 speakers.
Strong contextual correction: With a list of terms, names, or other entities, the named-entity error rate can be reduced sixfold, so specialized vocabulary no longer depends on guesswork.
Speaker-attributed word-level timestamps: Lightweight CTC alignment makes transcripts searchable and synchronized for playback, directly supporting applications such as lecture replay and meeting search.
High inference efficiency: Native MTP speculative decoding combined with a DFlash2 draft model enables a single RTX PRO 6000 to process 2,238 seconds of audio per second.

Hugging Face model: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
Technical report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Transcribe-Technical-Report.pdf

Meeting minutes and decision extraction: Automatically distinguish speakers, preserve the original discussion, and extract meeting conclusions and action items according to instructions.
Media interviews and news production: Preserve interviewees’ exact words and label speakers to ensure accurate quotations, while removing filler words as needed.
Class and training replay: Generate transcripts with word-level timestamps so students can search for keywords and jump directly to the moment when an instructor explains a concept.
Medical and professional records: Combine terminology lists with domain fine-tuning to convert doctor-patient conversations into medical records in a standardized format, reducing errors in specialized vocabulary.
Multilingual content localization: Directly translate recordings in 40 languages into English transcripts for international meetings, interviews, or video subtitle production.