If you've ever built an enterprise call-transcription service or transcribed hours-long meetup recordings, you know exactly where OpenAI's touted Whisper stumbles. On clean studio speech from a single speaker, sure, it produces decent text. Feed it a real Zoom call where three people talk over each other against an air conditioner hum and keyboard clatter, and the pipeline turns into a nightmare.
Whisper's core flaw is architectural blindness to speakers: it outputs a solid wall of text with zero sense of who said what. Engineers had to cobble together Rube Goldberg workarounds: slicing audio separately, running it through PyAnnote for diarization (speaker identification), and then using a third model to align text timestamps with voice tracks. Any hiccup at the seams caused hallucinations and misattributed quotes.
NVIDIA has put an end to that pain by unveiling its open flagship Nemotron 3.5 Speech family. Leading the pack is Nemotron 3.5 Transcribe, which topped the global Hugging Face Open ASR Leaderboard with an end-to-end architecture featuring built-in diarization and word-level timestamps out of the box. Let's inspect its anatomy and spin up a practical Python pipeline.
SALM Architecture: Merging Fastconformer Acoustics with a Mamba-Transformer LLM
Unlike classic ASR systems that act as simple acoustic sound-to-letter translators, Nemotron 3.5 is built as a SALM (Speech-Augmented Language Model). Its pipeline unites four core blocks:
- Fastconformer acoustic encoder. An optimized convolutional-attention module converting 16 kHz raw audio mel-spectrograms into dense context vectors with 8x temporal compression.
- Built-in diarization module. A frozen Nemotron 3 Diarization sub-network that extracts speaker embeddings for each participant in parallel with spectrogram processing.
- Nemotron 3.5 Lightning 30B-A3B language core. A Mixture-of-Experts (MoE) large language model. Out of 30 billion total parameters, only about 4.3 billion weights activate dynamically per token. Its blocks combine Mamba-2 selective state-space layers with classic transformer attention for deep grasp of grammar and conversational context.
- CTC adapter for word-level timestamps. A dedicated decoder aligning generated text to the audio timeline with millisecond precision.
This hybrid fusion eliminates the silence hallucinations common in older models: the language model "hears" pauses and doesn't try to invent words where only background noise plays.
Hands-On: A Diarized Transcription Pipeline in Python
NVIDIA provides the open-source NeMo Toolkit for working with Nemotron 3.5. A basic script to transcribe an audio file with built-in speaker separation is remarkably concise:
import nemo.collections.asr as nemo_asr
def transcribe_audio_meeting(audio_path: str):
# Load weights for the end-to-end Nemotron 3.5 Transcribe model
model_name = "nvidia/nemotron-3.5-transcribe"
asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(
model_name=model_name
)
# Run batch inference and extract hypotheses
hypotheses = asr_model.transcribe(
paths2audio_files=[audio_path],
batch_size=1,
return_hypotheses=True
)
# Process transcript and speaker labels
for result in hypotheses:
print("=== Full Detected Dialogue ===")
print(result.text)
if hasattr(result, "speaker_labels") and result.speaker_labels:
print("\n=== Speaker Mapping with Timestamps ===")
for segment in result.speaker_labels:
speaker = segment.get("speaker", "Unknown")
start = segment.get("start_time", 0.0)
end = segment.get("end_time", 0.0)
phrase = segment.get("text", "")
print(f"[{start:.2f}s - {end:.2f}s] {speaker}: {phrase}")
if __name__ == "__main__":
transcribe_audio_meeting("board_meeting_call.wav")
The model ingests audio files up to one hour long in a single continuous pass — no need to pre-slice into tiny chunks.
Production Inference: Compiling with TensorRT-LLM
For high-load telephony gateways and contact centers, running through PyTorch leaves GPU utilization on the table. The Nemotron family is fully optimized for compiling into high-speed binary engines via TensorRT-LLM:
# Build an optimized Nemotron 3.5 Speech binary engine
trtllm-build \
--checkpoint_dir ./checkpoints/nemotron-3.5-speech \
--output_dir ./engines/nemotron_trt \
--gemm_plugin float16 \
--max_batch_size 16 \
--max_input_len 8192
# Launch a streaming ASR microservice on Triton Inference Server
tritonserver --model-repository=/models/nemotron_speech_trt/ --grpc-port=8001
For live simultaneous translation and voice assistants, the lineup also includes the compact nemotron-3.5-asr-streaming-0.6b at just 600 million parameters. It supports tunable streaming latency from 80 milliseconds to 1 second, emitting interim words as they are spoken. For more on end-to-end audio models and real-time voice agents, check out our comparison of Hermes Apollo and audio-native Dessa.
Open ASR Leaderboard and Outperforming Whisper
On the authoritative Hugging Face Open ASR Leaderboard, Nemotron 3.5 took an undisputed #1 spot among open-weight models. Average Word Error Rate (WER) came in at:
- 4.71% on English test sets;
- 4.62% across multilingual benchmarks (versus ~6.8% for Whisper Large v3).
The model supports 40 global languages, with zero-config built-in speaker diarization available for the 13 most widely spoken languages. The Mamba-plus-MoE fusion delivers the same efficiency leap seen in NVIDIA's vision models — we previously analyzed that mechanic in multimodal encoder Nemotron 3.5 Super VL.
Hardware Requirements and Operational Bounds
Before dropping Nemotron 3.5 into your production pipeline, note these technical specs:
- VRAM requirements: the base Transcribe model packs a 30B MoE language core (~4.3B active), so smooth FP16 inference calls for a 16–24 GB GPU (RTX 4090, RTX 6000 Ada, or A100). Lighter cards should opt for the 0.6B streaming variant.
- Audio prep: the model strictly expects 16,000 Hz mono audio. Recordings at 8 kHz (standard telephony) or 44.1 kHz must be resampled beforehand via
soxorffmpeg. - Diarization languages: while transcription spans 40 languages, high-accuracy speaker separation is validated on 13 core languages; on rarer tongues, turns may merge under a single speaker ID.
NVIDIA Nemotron 3.5 Speech sets a new bar for open-source speech recognition: an end-to-end architecture finally frees developers from a zoo of fragile libraries, uniting hearing, comprehension, and voice separation in one sleek checkpoint.

