End-of-Turn Detection: Benchmarks and Training Data

What the detector has to decide

A voice agent built as a cascade (speech recognition, a language model, speech synthesis) needs a separate part that says when the user is done. The simplest version waits for a fixed stretch of silence, and people pause mid-sentence, so a short timeout cuts them off and a long one slows every reply. The turn detection guide on fullduplex.ai walks through the trade-off, and the full-duplex vs cascade guide explains why a full-duplex model makes the same call inside the model.

Benchmarks frame the decision in two ways:

Pause by pause. LiveKit's eot-bench marks every silence of at least 100 ms in a user turn. The last pause is the true end of turn and every earlier one is a hold the agent should listen through.
Event by event. TurnBench defines an end of turn as the moment the floor leaves a speaker, and scores whether a system fires near that moment without firing during mid-turn pauses.

The benchmarks and their metrics

Benchmark	Test data	What it scores	Main metrics
TurnBench	30 h of hand-labeled two-person English conversation, six conversation types	End of turn and interruption	Recall at a false-positive ceiling of 0.15, signed latency
eot-bench	Real human-to-agent user turns from task-oriented calls, 14 languages	End of turn at each pause	False cutoffs at 300 and 600 ms budgets, latency at 5% and 10% cutoffs
otoSpeech EOT evaluation	20.96 h, 84 two-person English conversations, 11,568 pause decisions	End of turn vs. hold	AUC, AP, false cutoffs at 300 and 600 ms
MultiPartyTurnBench	240 synthetic, channel-separated conversations with 2 to 5 speakers	End of turn and interruption	Recall at fixed false-positive rate, signed latency
Full-Duplex-Bench v1	CANDOR, ICC, and TTS-generated samples	Pause handling, backchannels, smooth turn-taking, interruptions	Takeover rate, backchannel timing (JSD), latency
Full-Duplex-Bench v1.5	Four overlap scenarios played while the model talks	Interruption, backchannel, side talk, background speech	Behavior category, stop and response latency
Full-Duplex-Bench v2	Multi-turn tasks with an automated examiner	Turn-taking fluency over several turns	Fluency, instruction following
Full-Duplex-Bench v3	Real human audio with five disfluency types	Multi-step tool calls under disfluency	Pass@1, latency, turn-take rate
Talking Turns	Live user sessions, scored by a judge trained on Switchboard	When to speak, backchannel, interrupt, yield	Agreement with the judge's labels

Two kinds of benchmark sit in this table. TurnBench, eot-bench, the otoSpeech evaluation, and MultiPartyTurnBench score a detector's decisions against labeled timestamps. The Full-Duplex-Bench series (v1, v1.5, v2, v3) and Talking Turns play audio to a whole dialogue system and score what it does. If you are choosing a component for a cascade, the first group answers your question. If you are evaluating a full-duplex model, start with the second.

A few narrower benchmarks are worth knowing. SID-Bench is built from real dialogues and charges a time penalty for both false alarms and late responses to interruptions (Xia et al.). ECHO pairs the same Chinese overlap transcript with two different contexts, one where the system should yield and one where it should keep talking (paper). M3-DuplexBench covers English and Japanese (paper). The evaluation guide lists the rest.

Scores depend on the conversation in the test set

oto ran the same detectors on two kinds of conversation. eot-bench tests task-oriented turns from human-to-agent calls. oto's evaluation tests long, spontaneous conversation between two people. Five systems ran on both (oto, 2026):

Detector	Task-oriented, 300 ms	otoSpeech, 300 ms	Task-oriented, 600 ms	otoSpeech, 600 ms
LiveKit Turn Detector v1	9.9%	44.8%	4.5%	24.3%
UltraVAD	27.7%	51.6%	11.9%	27.8%
LiveKit Turn Detector v1-mini	27.8%	60.7%	12.1%	31.9%
Smart Turn v3.2	35.2%	63.0%	14.8%	33.1%
VAD timeout baseline	55.6%	80.5%	21.7%	39.9%

The numbers are false-cutoff rates (lower is better) at a fixed mean latency budget. At 300 ms, every learned detector lost between 23.9 and 34.9 percentage points when the conversation became spontaneous. The two test sets come from separate runs, so the comparison shows sensitivity to domain, and it should not be read as one leaderboard.

TurnBench found a related pattern across its six conversation types. End-of-turn recall stayed roughly stable, while interruption false positives rose in styles dense with backchannels (TurnBench). On its test set, VAP, fine-tuned on the benchmark's training split, reached 0.845 end-of-turn recall at a 368 ms median latency, Smart Turn v3 reached 0.752 at 1,017 ms, and Moshi, with turn ends read from its output, reached 0.233. People in the same corpus start speaking a median 151 ms before the current turn ends in smooth handovers, and no system matched that without too many false positives.

MultiPartyTurnBench tests two to five speakers. Learned systems reached 82.5% to 92.6% end-of-turn recall at false-positive rates of 4.3% to 11.8%. Interruption detection was much weaker, and five full-duplex agents scored near chance (46.9% to 54.7% balanced accuracy) on stopping only for real interruptions.

Open turn detectors and their training data

Detector	Input	Training data, as published
Smart Turn v3.2	Up to 8 s of user audio	270,946 clips labeled complete or incomplete, with flags for fillers and synthetic samples (data card)
LiveKit text turn detector	Transcript, up to 6 turns	Real call-center transcripts, synthetic dialogues with addresses, phone and card numbers, and varied STT output formats (card)
LiveKit Turn Detector v1	User audio	Not described in the release post
VAP	Two-channel audio	Switchboard in the original paper, self-supervised (Ekstedt and Skantze)
DualTurn Endpointing	One or two channels via a frozen Mimi encoder	Switchboard, otoSpeech, and private two-channel AI-agent calls (card)
ESPnet turn-taking	Two-speaker audio	Switchboard, labels derived from speech timing (Talking Turns)
Easy Turn	Audio and text	About 1,145 h of real and synthetic speech in four states: 580 h complete, 532 h incomplete, 10 h backchannel, 23 h wait (card)
TEN Turn Detection	Transcript	Not described on the card, which publishes a Chinese and English test set
Namo Turn Detector	Transcript	A named training set the card does not describe; 90.25% average accuracy on its own test set (card)
otoTurn	User audio (Smart Turn 3.2, fine-tuned)	oto's full-duplex conversation with human-reviewed end-of-turn and floor-hold labels

Reading down the table:

Text-only detectors cannot hear the pause. The LiveKit text model's own card lists prosody (pauses, intonation, emphasis) as something it cannot use, and its accuracy depends on the upstream speech recognizer.
Clip datasets and conversation datasets teach different things. Smart Turn learns from labeled clips of a user's turn. VAP and DualTurn learn from two-channel conversation, where the listener's channel is part of the signal.
Rare classes stay rare. In the Easy Turn training set, backchannels are 10 of about 1,145 hours. If your agent needs to tell a backchannel from a bid for the turn, check how many examples of each the training data holds.

What changing only the training data did

otoTurn isolates the effect of training data. oto took the public Smart Turn 3.2 model, fine-tuned it on two-channel conversation labeled for end of turn and floor holds, and left the architecture alone (oto, 2026). On the otoSpeech evaluation:

False cutoffs at 300 ms fell from 63.0% to 33.0%, a 47.6% relative reduction, or roughly 2,500 more pauses held across the 8,391 hold decisions.
AUC rose from 0.7238 to 0.8880, and AP from 0.4665 to 0.7226.
On the task-oriented eot-bench set, false cutoffs at 300 ms also fell, from 35.2% to 21.6%, though otoTurn did not become the strongest task-oriented model.

otoTurn ranked first of seven systems on the otoSpeech evaluation, which is English only.

How to pick a benchmark and a training set

Match the domain. A detector tuned on booking and support calls can look excellent on eot-bench and still cut people off in open conversation. Test on data that looks like your traffic.
Read the decision unit. Pause-level scoring (eot-bench, the otoSpeech evaluation) counts every mid-turn hesitation. Event-level scoring (TurnBench) counts matched boundaries within a time window.
Check language coverage. eot-bench covers 14 languages. TurnBench and the otoSpeech evaluation are English. M3-DuplexBench adds Japanese, and ECHO is Chinese.
Keep training and test speakers apart. TurnBench's 104-hour training set is speaker-disjoint from its benchmark. Ask the same of any vendor set you train on.
Read the license. The TurnBench evaluation data and its training set are both non-commercial research releases. Commercial training needs a separate agreement.

Where oto fits

The TurnBench training set. otoSpeech-full-duplex-turn-104h is 104.94 hours of two-person English conversation with 17 turn-taking labels per speaker, gated under a research license.
A detector trained on that kind of data. otoTurn shows the effect of fine-tuning on labeled spontaneous conversation.
Public work that uses it. The TurnBench baselines, DualTurn, a VAP model evaluated in eot-bench across 14 languages, and Pipecat's backchannel detector are among the projects on oto.earth/use-cases.
Licensing and custom labels. Licensed datasets run from $10 to $100 per hour of audio, and turn-taking annotation adds to the price. Custom collection with your label set is quoted per project. Write to agent@oto.earth.

Data for voice AI

Request samples or a quote

Tell us what you are building, and we will send samples and pricing for the data that fits.

Request samples See the datasets Commission a collection

Frequently asked questions

What is the best benchmark for end-of-turn detection?

For task-oriented voice agents, eot-bench; for spontaneous two-person conversation and interruptions, TurnBench. Scores shift between the two domains, so it helps to run both.

What metrics do end-of-turn benchmarks use?

Most report how often the detector cuts in during a mid-turn pause (false cutoffs or false-positive rate) against how long it waits after a real end (latency), or recall at a fixed false-positive ceiling.

What data is Smart Turn trained on?

Smart Turn v3.2 trains on pipecat-ai/smart-turn-data-v3.2-train, 270,946 audio clips labeled complete or incomplete, with flags for fillers and synthetic samples.

Is there an open turn-taking training dataset?

Yes. otoSpeech-full-duplex-turn-104h, the TurnBench training set, has 104.94 hours with 17 labels per speaker under a research license; the Easy Turn trainset and Smart Turn data are also open.

Why does my turn detector interrupt users who pause to think?

In oto's comparison, detectors that did well on task-oriented calls cut in far more often in spontaneous conversation, where people pause mid-thought. Fine-tuning on spontaneous conversation labeled for floor holds reduced false cutoffs by nearly half in oto's otoTurn test.

Do full-duplex models need an end-of-turn detector?

Usually not as a separate part, since they decide inside the model, but benchmarks such as TurnBench and Full-Duplex-Bench still score how well they time their turns.

For more benchmarks and turn detectors, see the fullduplex.ai catalog.

otoearth, Inc.