**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**
NVIDIA released Nemotron 3 Diarization, an open-weight model that identifies who speaks when in multi-speaker audio, improving accuracy and supporting up to eight speakers in real time.
NVIDIA introduced Nemotron 3 Diarization, an open-weight model with 100 million parameters designed to identify speaker turns in multi-speaker audio. The model ranks first on VoiceArena’s Diarization-Bench with a 14.72% diarization error rate, outperforming competitors by approximately 24% relative reduction in errors. It supports up to eight speakers and handles overlapping speech, chunked processing, and customizable streaming latency for both live and recorded conversations.
The model processes 16 kHz single-channel audio into Mel-spectrogram features, using a 31-layer Transformer encoder with rotary positional embeddings. It outputs a [T, 8] tensor representing speaker activity probabilities over time, with a default 10 ms stride. The architecture includes arrival-time ordering to stabilize speaker labels and supports multiple latency configurations, from 0.32 to 30.4 seconds input buffers, balancing responsiveness and accuracy.
Training incorporated public and licensed speech data, including multispeaker-annotated real-world conversations from David AI and large-scale simulated mixtures in 21 languages. Adding David AI data reduced the compound diarization error rate by 0.77 absolute points, from 11.19% to 10.42%, at both offline and ultra-low-latency operating points. The model does not identify real-world speaker identities but provides anonymous channel IDs that downstream applications can map to identities using metadata or verification models.
Nemotron 3 Diarization is intended for integration with automatic speech recognition (ASR) to produce speaker-attributed transcripts. It addresses challenges like missed speech, incorrect speaker assignments, and boundary errors, while ASR handles word transcription. The model’s performance was validated across eight public benchmarks at 1.04-second output latency, demonstrating state-of-the-art results compared to NVIDIA’s previous four-speaker Streaming Sortformer baseline.