otoTurn: End-of-turn detection for spontaneous conversation

Everything below comes from one change: the training data. We took a public end-of-turn model, fine-tuned it on a dedicated full-duplex dataset with turn-taking labels, and left the architecture alone. That dataset is the product.

For an AI agent to hold a natural spoken conversation, it has to know when the other person has finished speaking. Existing end-of-turn systems already make this work well in customer support, booking, data collection, and other structured flows. That success matters. But it also reflects where most models are trained and measured: the user is trying to complete a request, and the agent is waiting for permission to act.

Ordinary conversation has a different shape. People discover what they mean while speaking. They revise sentences, add qualifications, search for words, and continue after pauses lasting a few hundred milliseconds or well over a second. A grammatically complete sentence may be the end of a turn, or merely the setup for what comes next.

We wanted to know whether systems validated on task-oriented voice-agent traffic behave the same way in spontaneous conversation. To test this, we built an evaluation from long-form human conversation, then fine-tuned SmartTurn 3.2 on our turn-taking-labeled full-duplex data. We call the resulting model otoTurn.

Task-oriented performance does not transfer cleanly

LiveKit’s public EOT Bench evaluates real human-to-agent turns from task-oriented conversations. It is a useful benchmark for the systems people deploy today. otoSpeech instead tests long-form, free-flowing human-to-human conversation.

Five current baselines were available in both English evaluations. Their behavior changed consistently when the conversation domain changed.

At 300 ms, the learned detectors lose between 23.9 and 34.9 percentage points on otoSpeech. At 600 ms, the loss is between 15.9 and 19.8 points. The effect is not specific to one provider, one architecture, or one operating threshold. The entire field becomes less reliable when the speaker is conversing rather than completing a task.

The important distinction is between detecting predictable endpoints in transactional speech and identifying the actual end of a turn amid the ambiguous floor transitions of spontaneous conversation.

An evaluation for the conversation we want

The otoSpeech test set contains 20.96 hours from 84 full-duplex, two-person conversations, with human-reviewed end-of-turn and floor-hold annotations. The recordings preserve the events that make end-of-turn detection difficult: floor holds, resumptions, interruptions, overlap, backchannels, and long regions of speech with several internal pauses.

The useful unit is not simply a completed utterance. It is each point at which a detector could reasonably but incorrectly declare an end of turn. We split long annotated regions at their internal pauses and turn each pause into a causal decision:

• End of turn: the speaker has finished the current turn.
• Hold: the speaker pauses but has not finished.

A long explanation can therefore contribute dozens of hold decisions instead of one easy endpoint. This makes word searches, self-corrections, restarts, and reflective pauses part of the evaluation rather than noise removed during preprocessing.

The final common test set contains 11,568 decisions: 3,177 true endpoints and 8,391 holds. Every system saw the same decision points. For each pause, the audio was cut 200 ms after the silence began. Each detector made its judgment from that causal prefix, without hearing the speaker’s later continuation.

What changed when the model learned from turn-taking labels

We started with the public SmartTurn 3.2 model and fine-tuned it on our full-duplex dataset with turn-taking labels. We refer to this fine-tuned model as otoTurn. Keeping the starting architecture fixed makes the central question simple: how much can spontaneous-conversation data change what an existing model considers a finished turn?

At the 300 ms operating point, fine-tuning cut false interruptions from 63.0% to 33.0%: a 30-point improvement, or a 47.6% relative reduction. At 600 ms, the rate fell from 33.1% to 18.0%. AUC rose from 0.7238 to 0.8880, while average precision rose from 0.4665 to 0.7226.

Across the 8,391 hold decisions in the fixed test set, the 300 ms improvement corresponds to roughly 2,500 pauses preserved. Word searches, self-corrections, restarts, and reflective pauses stopped looking so much like invitations to respond.

The same fine-tuning also improved SmartTurn on the task-oriented benchmark, reducing false cutoffs from 35.2% to 21.6% at 300 ms and from 14.8% to 11.6% at 600 ms. It did not become the strongest task-oriented model, which is expected: otoSpeech is built around spontaneous conversation, not transactional coverage. However, the strong positive transfer to a structured domain still demonstrates the broader value of the otoSpeech dataset.

otoTurn in the full evaluation

Fine-tuned on otoSpeech, otoTurn ranks first on every metric in the spontaneous-conversation evaluation.

Against the strongest off-the-shelf detector, otoTurn reduces false cutoffs by 11.85 percentage points at 300 ms, or 994 fewer interruptions across the hold decisions. At 600 ms, the reduction is 6.29 points, preserving 528 additional pauses.

The largest gains appeared in long narrative and instructional exchanges, where speakers frequently pause while assembling a larger point. The same direction remained visible across the other conversational settings in the test set, rather than coming from one favorable slice.

What the failures sound like

The aggregate result says that specialization helps. The following clips show what it helps with. All come from the fixed test set and all are cases where otoTurn avoided a false end-of-turn decision at the matched roughly 300 ms operating point.

1. A thought searches for its next phrase

Here, hesitation is part of the act of speaking. A conversational end-of-turn detector has to distinguish searching for words from deciding not to continue.

2. Grammar briefly looks finished

A transcript can reasonably place a full stop at the boundary. The missing information is conversational: whether this speaker is wrapping up or still building the point.

3. The conclusion arrives after the pause

Adaptive timing reduces unnecessary waiting

A static timeout has one setting for every silence. If it is short, the agent interrupts reflective speakers. If it is long, obvious endpoints feel slow. Choosing 300 ms or 600 ms merely chooses which failure is more tolerable.

otoTurn uses an adaptive policy instead. A pause that looks like a clear endpoint can be acted on early. An ambiguous pause receives more time. A pause that strongly resembles a floor hold receives more still.

This does not make the model itself compute faster. It removes unnecessary waiting from easy decisions and spends that time on hard ones.

In held-session cross-validation at approximately 590 ms mean latency, replacing one static decision time with adaptive timing reduced false cutoffs from 18.36% to 15.85%. Mean latency changed from 589.3 ms to 591.4 ms, about two milliseconds, while 121 additional pauses were preserved. That is a 13.7% relative reduction in false cutoffs for essentially the same average wait.

The subjective effect is larger than the average suggests. Easy turns stay fast. The extra latency is concentrated on exactly the pauses where an interruption would feel most careless.

Beyond end-of-turn detection

More data may improve end-of-turn detection, but natural conversation is difficult to describe through that signal alone. The broader problem is turn-taking.

An unfinished turn does not always mean the agent should remain silent, just as a finished turn does not always require an immediate response. Sometimes the floor is open without requiring anyone to take it.

We are extending this work to the annotations and research needed for that broader problem. Our aim is to contribute to voice AI primarily through data. Making these ambiguous moments observable, measurable, and useful to the field is one of the key problems we are working on.