NVIDIA NeMo 3 Diarization Architecture: How a 100M-Parameter Model Separates 8 Voices Talking Over One Another

Automatic speech recognition (ASR) has achieved near-human accuracy in recent years: models like Whisper and Conformer transcribe individual phrases practically without errors. However, when applied to a real recording of a meeting, medical consultation, or court hearing, the result turns into an entirely unreadable wall of text. Without a precise answer to the question “who exactly said these words?” the transcript loses its practical value. To solve this problem, NVIDIA has made the model NeMo 3 Diarization.

The Core Problem: Why Speaker Diarization Is Harder Than Word Recognition

Speaker diarization is the process of segmenting an audio stream based on the acoustic characteristics of individual speakers. The difficulty lies in three fundamental factors:

1. Acoustic variability: the timbre, volume, and intonation of the same person change depending on their emotions, distance from the microphone, and the room’s background noise.
2. Overlapping Speech: in live meetings, participants regularly speak at the same time, interrupt one another, or insert short acknowledgments (“yes,” “uh-huh,” “exactly”). Most classic pipelines merge the two voices into a single random profile at this point.
3. Unknown number of speakers: the algorithm does not know in advance how many people are in the room—two or eight.

NeMo 3 Architecture: 100M Parameters and Resistance to Overlapping Speech

The NeMo 3 Diarization model contains approximately 100 million parameters—a compact size that allows it to run on local workstations with consumer GPUs or on enterprise edge servers. The pipeline consists of several coordinated modules:

• VAD (Voice Activity Detection): filters pauses, breathing, mouse clicks, and keyboard tapping;
• Speaker embedding encoder: a convolutional network with attention mechanisms that transforms short audio frames into dense vectors of acoustic features;
• Overlap Detection & Separation module: a specialized layer that detects the presence of two synchronized frequency patterns and duplicates the time segment for parallel matching with two different profiles;
• Online/Offline clusterer: spectral clustering with dynamic determination of the number of participants (supports up to 8 simultaneously active speakers).

Results in the Voice Arena Diarization Bench

The primary quality metric for speaker diarization is DER (Diarization Error Rate) — the aggregate error percentage, including false positives, missed speech, and speaker confusion errors.

Practical Pipeline for Integration with Whisper and LLMs

The most effective production stack is built as an end-to-end processing pipeline:

[Аудиозапись] ──► [NeMo 3 Diarization] ──► Таймкоды спикеров: [00:12 - 00:15: Спикер 1]
        │                                  [00:14 - 00:18: Спикер 2 (наложение)]
        ▼
[OpenAI Whisper / Conformer ASR] ────────► Текстовые сегменты с точными временными метками
        │
        ▼
[Модуль слияния (Alignment)] ────────────► Структурированный диалог по ролям
        │
        ▼
[LLM (Llama 3 / Claude / GPT)] ──────────► Выделение договоренностей, Action Items и протокол

Why Open Weights Are Critical for Business

Recordings of top management’s internal meetings, medical appointments with patients, and lawyers’ negotiations are protected by strict regulatory frameworks (152-FZ in Russia, GDPR in the EU, HIPAA in the United States). Transmitting such audio files to external cloud APIs creates unacceptable risks of data compromise. The open release of NeMo 3 Diarization allows companies to deploy a fully isolated transcription environment within their own perimeter, without the risk of sensitive negotiations being leaked.