Speaker Diarization on 95MB: NVIDIA Tracks 8 Speakers On-Device

Ask a speech-to-text model what was said in a meeting and it will hand you back a clean transcript. Ask it who said it and it goes quiet. That gap is speaker diarization, and until recently it was the most expensive part of any voice pipeline: the models capable of it were large, ran on servers, and mostly topped out at two or four speakers.

NVIDIA’s Nemotron 3 Diarization, released on 23 September 2026, changes that. It is a 100-million-parameter model that separates up to eight overlapping speakers, runs live or on recordings from the same checkpoint, and has already been converted by independent developers into builds small enough for a phone, a Mac, and a web browser. The largest build is 397MB. The smallest is 95MB.

Here is what speaker diarization actually does, how the new model performs, and what it takes to run it yourself.

What Speaker Diarization Actually Does (and Why Whisper Cannot)

Speech recognition and speaker diarization answer different questions. Speech recognition answers “what was said.” Speaker diarization answers “who spoke when.” Neither one does the other job, which is why pairing them is standard practice rather than a luxury. Nemotron 3 Diarization sits firmly on the second side of that line. It has no vocabulary and cannot write a word: its output is a grid of probabilities showing who is talking and when, so the text in any finished transcript always comes from a separate speech recognition model.

Consider a transcript where every sentence is transcribed correctly but nobody is named:

“I’ll send the report.” “Can you include the figures?” “Yes, by Friday.”

You can read it, but you cannot reliably tell who made a commitment, who raised an objection, or who interrupted. Search, summaries, action items, and long-term voice agent memory all degrade. Diarization adds a second layer of labels, so the same exchange becomes attributable to specific people.

This is also why OpenAI’s Whisper models do not support diarization. Whisper is a speech recognition model, and no amount of configuration will turn it into a speaker-attribution system. The common open-source workaround, WhisperX, bolts Pyannote diarization onto Faster-Whisper and aligns the two output streams so each word carries a speaker tag. That works, but it means running two separate models and an alignment stage.

Google’s recent Gemini 3.5 Transcribe illustrates the ceiling most commercial services hit. It attributes speech to up to three speakers in batch mode and none at all on the streaming endpoint. That is a sensible product decision for a cloud API. It is also the reason on-device diarization is worth paying attention to.

How diarization is scored

The field’s standard metric is the diarization error rate, or DER. It adds three kinds of mistakes and divides by the total reference speech time:

• Missed speech: a person was talking, and the system detected nothing.
• False alarm: the system marked a speaker as active when nobody was talking.
• Speaker confusion: the system found speech at the right moment but assigned it to the wrong person.

A DER of 10% means roughly 90% of speech time was attributed correctly. Overlapping speech is scored in every evaluation quoted below, which makes these numbers harsher than older published figures that quietly skipped interruptions.

Why Older Diarization Models Cap Out at Two to Four Speakers

There was a structural reason most diarization systems topped out early, and it is worth understanding because it explains the design of the model that displaced them. The dominant approach was a pipeline built from three separate stages: detect where speech occurs, extract a voice embedding for each detected segment, then cluster those embeddings and assign a label to each cluster. The third stage is where the ceiling came from. Clustering quality degrades as the number of distinct voices rises, because the algorithm is effectively guessing how many clusters to create, and a miscount in either direction is unrecoverable. Under-clustering merges two people into one label; over-clustering splits one person across several.

NVIDIA’s own streaming Sortformer, the previous generation, had a hard ceiling of four speakers with degraded performance beyond that. Several commercial services have sold diarization as a feature of a larger transcription product, which caps how much engineering effort goes into the speaker layer specifically.

End-to-end models changed the picture by removing the clustering step altogether. Rather than grouping segments after the fact, they predict speaker activity directly against a fixed set of channels, one per possible speaker, at ten-millisecond resolution. Because each channel is scored independently, overlap falls out of the architecture for free: if two people talk at once, two channels are active in the same time frame, which is something a clustering pipeline cannot represent at all since it must assign each instant to exactly one speaker. The trade is that the channel count becomes a hard design decision made before any audio is heard, which is why doubling from four to eight is a meaningful engineering decision rather than a parameter tweak.

Nemotron 3 Diarization allocates eight, which is roughly double its own predecessor and enough for a panel discussion, a board meeting, or a family dinner with arguing children.

Meet Nemotron 3 Diarization

The model was released by NVIDIA on 23 September 2026 under the OpenMDW 1.1 license, which permits commercial use. It is a 31-layer Transformer encoder with rotary positional embeddings, running on 100 million parameters, and it lives in the open under the same name on Hugging Face.

The technical design choices that matter to a non-specialist are straightforward:

• One checkpoint, two modes. The same weights process a recorded lecture at full accuracy or a live conversation in real time. You do not choose a different model for each.
• Eight speaker channels. Output is a grid of time steps by eight possible speakers, each cell holding the probability that speaker is active. A tenth person is outside the design limit, and NVIDIA is explicit that audio with more than eight speakers will see speech missed or misassigned.
• 10-millisecond resolution. The model reports activity ten times per second, and the output step is configurable in multiples of 10ms. That is fine-grained enough to catch a short interjection.
• Overlapping speech is native. Two active channels in the same frame means two people spoke at once. This is the thing clustering pipelines structurally cannot do.
• Labels are anonymous. The model reports “speaker_2,” not “Sarah.” Mapping anonymous channels to real identities is a separate step that applications handle using meeting metadata or voice verification.

The Benchmark Results, Measured Three Ways

Nemotron 3 Diarization has been evaluated against its predecessor across 901 condition-specific recordings spanning telephone speech, meetings, near-field and far-field microphones, and multilingual conditions. There is also a brand new independent leaderboard.

Against NVIDIA’s own previous model

At the 1.04-second latency setting, the new model lowers diarization error on all eight evaluation conditions against the previous streaming Sortformer. Relative reductions range from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1, with an unweighted mean of 41.0% across the eight conditions.

The gains widen as speaker count rises, which is the direct payoff of doubling the channel limit. On NOTSOFAR1 at the offline setting, the five-to-seven-speaker subset improved from 29.38% DER to 7.86%.

There is one honest regression. On the two-speaker CALLHOME subset at the offline setting, DER rose slightly from 5.68% to 5.98%. The full CALLHOME-Part2 set still improved from 10.32% to 9.10%.

Against the field, on a new leaderboard

VoiceArena, an independent evaluation platform, published the first results from its Diarization-Bench, covering 12 systems and 17 configurations across 139 English conversations totalling roughly 22 hours. Nemotron 3 Diarization ranks first with a 14.72% diarization error rate, against 19.3% for the next-best system, a relative reduction of about 24%.

It also ranked first with 100ms and 250ms boundary collars, and across both in-person and online recordings. The benchmark is strict: overlapping speech scored, system-generated voice activity detection rather than oracle labels, and no forgiveness collar at the boundaries.

Two caveats belong here. These are initial results, and NVIDIA notes they may change as VoiceArena completes its Version 1 evaluation. Separately, the leaderboard’s own site was returning a server error when this article was written, so the figures here are taken from NVIDIA’s published account of the results.

Throughput

At the offline setting with batch size 32 and compiled execution on an RTX PRO 5000, the model reaches 15,113 times real-time throughput, against 2,619 times for the previous model. These are batched numbers on a professional GPU, which is not the same as single-stream application latency, and NVIDIA is careful to say so.

The on-device numbers later in this article are the ones that matter for anyone not running a data center.

Latency: Choosing Between 30.4 Seconds and 0.32 Seconds

Streaming latency is where diarization models are judged on real-time applications. Nemotron 3 Diarization exposes four recommended operating points from a single set of weights.

The accuracy spread across that whole range is less than one percentage point, which is unusual. Most streaming systems trade significantly more accuracy for lower latency.

Two things to understand about these numbers. First, input-buffer latency is the audio the model must hear before it commits to a decision, calculated as chunk plus look-ahead multiplied by 80 milliseconds. It excludes computation, networking, and the transcription stage. A realistic end-to-end figure is higher.

Second, the model can technically run on an 80 millisecond buffer, but 0.32 seconds is the lowest setting NVIDIA recommends. Below that, accuracy degrades faster than the latency gain is worth.

For a voice agent that must respond before a human finishes a sentence, sub-second is the design target. For a meeting recorder running in the background, 30.4 seconds of buffer buys the best accuracy at almost no cost, since nobody is waiting.

What It Takes to Run This Yourself: Hardware Minimums

The single most consequential fact about this model is that it does not need a data centre. Officially it targets Linux with NVIDIA GPUs through the NeMo framework. In practice, developers have already rn it on hardware spanning the entire range.

• A laptop CPU. Apple Silicon builds run through MLX and CoreML on the Neural Engine, with measured single-stream throughput up to 904 times real time offline.
• A phone GPU. A LiteRT conversion runs the whole network on a Galaxy S26’s Adreno GPU, processing 0.72 seconds of audio every 0.72 seconds while audio arrives at microphone speed.
• A web browser. An ONNX Runtime Web port runs entirely client-side, at roughly 160 times real time using WebGPU on an M3 Max, and 25 times real time on plain multithreaded WASM.
• A Raspberry Pi class device. The q8 GGUF build is 107MB and runs through NVIDIA’s own NeMo-Speech.cpp runtime on CPU alone, and the same runtime exposes an OpenAI-compatible endpoint for tools that expect one.

The memory arithmetic is what makes this practical. A 100-million-parameter model stored in 32-bit floating point requires roughly 400MB, because each parameter occupies four bytes. Compressed to 8-bit integers, the same weights occupy about 100MB, since each now takes a single byte. That 300MB difference is the entire gulf between hardware that needs a server with a discrete GPU and hardware that fits in a pocket alongside everything else the device is doing, and it explains why the conversion work covered in the next section matters more to practical adoption than any architectural detail of the encoder itself.

The Quantization Story: Six Builds, 396MB Down to 95MB

Quantization is the process of storing model weights at reduced numerical precision. The weights are the learned parameters that carry all of the model’s knowledge, and because the overwhelming majority of them were originally trained as 32-bit floating point values, storing them at 8-bit or 16-bit immediately cuts the memory footprint. The usual trade-off is that lower precision occasionally confuses a calculation enough to change the output. For a model that has to fit on a phone or run inside a browser tab, however, quantization is not an incremental optimisation. It is the precondition that makes the deployment possible at all, because the full-precision version simply does not fit in the available budget.

Independent developers have built six versions of Nemotron 3 Diarization. The table below is the clearest picture currently available of what runs where.

Three findings here are worth drawing out.

Quantization costs very little accuracy here. The int8 ONNX build agrees with the reference model on 99.994% of per-frame decisions. The 95MB CoreML build scores 9.63% DER on the AMI test set against 9.36% for the full-precision 190MB build, a gap of 0.27 percentage points. For practical purposes that is free.

Lower precision can be faster, not just smaller. On the Galaxy S26, running the encoder in fp16 takes 115 milliseconds per step against 171 milliseconds in fp32, roughly 33% quicker. There is a catch, and the conversion’s author is candid about it: fp16 changes which frames the model keeps in its speaker cache, which shifts segment boundaries slightly. The recommendation is to use fp32 when exact segment boundaries matter, and fp16 when speed does. Half the file size either way.

Window size matters more than quantisation for speaker counting. One CoreML build found that the 95MB int8 variant matched full precision on all 16 test meetings when given a longer analysis window, but lost nearly 19 points of speaker-counting accuracy on a shorter window. Quantisation was not the deciding variable. How much audio the model considers at once was.

It Runs in a Web Browser and on a Galaxy S26

Two of these builds deserve particular attention because they remove the cloud from the equation entirely.

Diarization entirely inside a web page

The ONNX Runtime Web port processes audio without sending it anywhere. On an M3 Max, the full-precision graph runs at about 160 times real time on WebGPU, which is more than fast enough for live audio streaming. The 8-bit version on plain JavaScript with no GPU runs at roughly 25 times real time, which is still comfortably faster than playback.

The port’s author notes that the graph covers only the neural network, so the audio front end and the streaming speaker cache have to be wrapped around it in about 250 lines of JavaScript. That is a modest amount of work for something that turns any web page into a private transcription tool, with no API key, no usage meter, and no recording leaving the device.

A 100M-parameter model running on a phone GPU

The LiteRT conversion for the Galaxy S26 is the clearest demonstration that this is now a phone workload rather than a server workload. All 2,915 nodes of the network run on the GPU as a single partition with no CPU fallback, handling live microphone audio at real-time pace.

It works out to roughly 0.24 times real time, meaning the phone uses about a quarter of its processing capacity to keep up with live speech. Offline file mode is dramatically faster, processing a 97-second clip in about one second.

There is an engineering lesson buried in the conversion notes worth knowing if you try this yourself. Running back-to-back without pause heats the phone GPU enough that its clock throttles, roughly doubling per-step time. At genuine real-time pace, where the model has breathing room between steps, it held full speed throughout. The guidance is simple: use the offline mode for files, and streaming mode for live audio.

Combined, these two builds mean a note-taking app that labels every speaker in a conversation, runs entirely on the phone, needs no account, and costs nothing per hour. That is a meaningfully different proposition from the per-minute pricing that commercial transcription APIs charge.

Languages and What “Language-Agnostic” Really Means

Diarization works from voice characteristics rather than words, so it is not tied to a specific language the way a transcription model is. That is the same design instinct behind recent direct speech-to-retrieval research, which bypasses transcription entirely rather than trying to post-correct it. If you can hear a distinction between two voices, the model can potentially use it, regardless of what is being said.

The training data does shape performance. The model was trained on roughly 10,000 hours of real conversations plus a very large set of simulated multi-talker mixtures. Languages represented in the real data include English, Mandarin, Hindi, Kannada, Telugu, and Bengali, with the simulated mixtures built from licensed source material spanning 21 languages.

Two practical implications for multilingual use:

• Mixed-language conversations work. Because the model is not transcribing, it does not need to know which language a given speaker is using. A call switching between English and Kannada does not confuse it the way a language-locked transcription model might.
• It pairs well with translation. For someone who does not speak the language of a conversation, diarization plus a separate translation model can produce a speaker-attributed, translated transcript. Keeping the two stages separate means each can be improved or replaced independently.

The limitation is that language coverage in the training data is not perfectly even, and error rates will vary. Anyone deploying this for a specific low-resource language should test on their own audio rather than assuming parity with English.

Practical Uses: Voice Agents, Note-Taking, Translation, Robotics

The practical question is what a model that answers “who spoke when” is actually for. Several uses are already working.

Real-time voice agents

A voice agent that handles group calls needs to know whether a question was directed at it or at a colleague. With eight speaker channels and a 0.32 second operating point, the model can maintain a running picture of who holds the floor, which is the precondition for an agent that does not interrupt a human conversation. NVIDIA’s own streaming pipeline pairs diarization with a speech recognition model to keep a separate transcription stream per detected speaker, a pairing documented as a supported integration path.

Note-taking without a subscription

This is the most immediate consumer application. Combined with any small local transcription model, a phone can produce a speaker-attributed transcript of a meeting, interview, or lecture with no API calls and no monthly fee. The privacy dimension matters as much as the cost: a conversation that never leaves the device cannot leak.

Separating overlapping speakers in existing recordings

For archival work, diarization applied after the fact can clean up a recording far more cheaply than re-recording it. Interview transcripts, podcast production, and deposition audio all benefit from knowing which of two similar voices is asking the follow-up question.

Translation for people who do not speak the language

Pairing local diarization with a translation model produces a transcript where each speaker’s contribution is separated and rendered in a language the reader understands. Run locally, this is feasible on hardware a person already owns.

Robotics

A robot that needs to know which of several people is addressing it has the same problem as a voice agent, in a harder physical environment. A 95MB model running on a Neural Engine or a phone GPU is a plausible size for an onboard compute module, and speaker identity is often a stronger cue than voice direction for disambiguating who meant what in a noisy room.

Where This Goes Next: Smart Glasses and Hearing Access

The current hardware frontier is worth taking seriously, because it points at where the model is heading rather than where it sits.

A 100-million-parameter streaming diarizer already runs in real time on a phone GPU with no network connection, and the most aggressive conversions are down to 95MB running entirely on a Neural Engine. The trajectory across those builds is consistent: each reduction in precision has cost almost nothing in accuracy while roughly halving the footprint.

Smart glasses are a harder target than a phone, not an easier one. They have less thermal headroom, a smaller battery, and an expectation of all-day wear. What makes the case plausible is the direction of travel rather than the current numbers: the same quantisation techniques that took this model from 397MB to 95MB are exactly what a glasses manufacturer would need, and the trend across six independent builds suggests the floor keeps falling.

The application that seems most obvious is captioning for people who are deaf or hard of hearing. Imagine glasses that display a live transcript of the room with speaker labels, running entirely on the device, with nothing recorded and nothing uploaded. A 0.32-second operating point is short enough that captions could keep pace with natural conversation, and eight speaker channels cover the meetings and family gatherings where turn-taking is hardest to follow.

The same capability suits people who do not speak the language of their surroundings. Combined with on-device translation, it becomes a wearable interpreter that does not depend on connectivity, which is a meaningful difference in places where a network is not guaranteed.

None of this exists as a product yet. But the enabling condition is now in place, and it arrived quietly: not as a headline consumer gadget, but as a set of open weights that anyone can shrink to fit.

Limitations, Honestly Stated

The model card is unusually forthcoming, and the constraints are real.

• Eight speakers is a hard ceiling. Beyond that, speech is missed or assigned to the wrong channel. Press conferences, earnings calls, and large panels are the obvious cases where this bites. NVIDIA’s model card notes that the DIHARD III benchmark’s five-to-nine-speaker group includes recordings above the supported limit, so that aggregate is pessimistic by design.
• Labels are anonymous. The model says “speaker_2,” never “Sarah.” Any application claiming to name people needs an additional identification step, and that step carries its own error rate.
• Audio quality still dominates. Heavy noise, strong reverberation, and far-field capture all raise missed speech and confusion. Clean microphone placement remains the highest-leverage thing anyone can do.
• Very short utterances are hard. Back-channel sounds like “mm-hm” and one-word agreements give the model very little to work with, and misassignment there is common across the field.
• Quantisation trades something. The int8 and fp16 builds are near-lossless on measured benchmarks, but near-lossless is not lossless, and the fp16 speaker-cache shift on the phone build is a concrete example of a small numeric difference changing output.
• The leaderboard is young. The VoiceArena results are initial, and the benchmark’s own site was down when this was written. The underlying model card numbers are reproducible; the leaderboard ranking may yet move.

Frequently Asked Questions

Does Whisper support speaker diarization?

No. Whisper performs speech recognition only: it produces a transcript and does not attribute speech to speakers. Speaker labels require a separate diarization model. The standard open-source combination is WhisperX, which runs Faster-Whisper for transcription alongside pyannote for diarization and aligns the two so every word carries a speaker tag.

What is the best diarization model right now?

It depends on the constraint. Nemotron 3 Diarization leads the independent VoiceArena Diarization-Bench at 14.72% DER and is the strongest option for on-device or local deployment, because it is small, open, and already converted to phone, laptop, and browser builds. Commercial services such as pyannoteAI Precision-2 and Deepgram may score differently on their own benchmarks, and pyannote’s open pipeline has no hard speaker ceiling. If the question is specifically which model you can run yourself on hardware you already own, Nemotron 3 is the clear answer.

How does speaker diarization work in real time?

Audio arrives in small chunks. The model scores each chunk against eight speaker channels while carrying forward a memory of speakers it has already heard and a short queue of recent audio for context. Output resolution is ten milliseconds, and the whole system is configured at one of four latency points between 30.4 and 0.32 seconds. Overlapping speech shows up naturally as two channels active in the same frame.

Is Nemotron 3 Diarization free to use commercially?

Yes. It is released under the OpenMDW 1.1 licence, which permits commercial and non-commercial use. The cost is compute, and on-device builds make that close to zero for most individual and small business use.

Can diarization run offline on recorded audio?

Yes, and it is the more accurate mode. The offline operating point uses a 30.4 second buffer and scores 12.73% DER on DIHARD III against 13.55% at the fastest streaming setting. Because there is no deadline, offline processing of a file can be dramatically faster than real time, which is how the phone build handles a 97-second clip in about one second.

How many speakers can it track at once?

Up to eight, with overlapping speech supported. This is double the previous NVIDIA streaming Sortformer’s limit of four. Recordings with more than eight speakers will see missed or misassigned speech, and NVIDIA’s own model card flags this as a deployment consideration for press conferences and large panels.