Turn-Taking Labels: What to Annotate for Voice Agents

Why voice activity is not enough

If each speaker has a separate channel, voice activity per speaker comes almost for free. A 600 ms pause looks the same on the waveform whether the speaker is done or searching for a word, and a one-word overlap looks the same whether it is a backchannel or the start of a barge-in. A voice agent has to tell these apart at each pause, and again while it talks. The turn detection, interruption, and backchannel guides on fullduplex.ai cover these decisions one at a time.

The events worth labeling

The hardest pair is backchannel versus interruption. At its onset, a "yeah" that will end in two syllables sounds like a "yeah" that will become a full turn. TurnBench reports that interruption false positives were concentrated in conversation styles dense with backchannels (Jiang et al., 2026).

How published schemes define them

Benchmarks and models each draw these lines differently, so check the definitions before you mix datasets.

Where the schemes differ:

• TurnBench grounds its labels in conversation analysis. An end of turn is the point where the floor passes to the other speaker. A backchannel is a listener vocalization that does not claim the floor, regardless of how long it lasts (paper).
• Talking Turns derives labels from timing. It builds its labels from inter-pausal units in Switchboard and separates a floor-taking interruption from "butting in", where the interrupter has to wait (Arora et al., 2025).
• Full-Duplex-Bench uses a rule. A segment counts as a backchannel if it is shorter than one second and has fewer than two words (Lin et al., 2025). The rule is simple to reproduce and will miss long backchannels.
• "Wait" means something different in text models. In TEN and Easy Turn, wait is the user telling the agent to stop or pause, such as "shut up" in TEN's card, which is a different event from a hold.

How the 17 labels in otoSpeech-full-duplex-turn-104h are organized

otoSpeech-full-duplex-turn-104h is 420 two-person English conversations, about 104.94 hours, released as the TurnBench training set (dataset card). Each speaker channel has its own SubRip (.srt) file. Each entry carries a millisecond time interval, a bracketed label, and the transcript text, as in 00:00:12,340 --> 00:00:13,020 followed by [Acknowledgement Backchannel] Okay.

The card lists 17 labels. TurnBench's paper maps them to its canonical categories as follows:

The fine labels split three kinds of backchannel and four kinds of interruption (floor-taking or not, competitive or cooperative). TurnBench merges the subtypes for scoring because no public baseline predicts them, and keeps the fine labels in the released data. A model that only needs the canonical classes can collapse them in a few lines:

import re

# TurnBench's mapping from the 17 fine labels to canonical categories
CANONICAL = {
    "Normal Turn": "Turn", "Strong Floor Hold": "Turn", "Bounded Response": "Turn",
    "Filler": "Turn", "Overlap": "Turn",
    "Floor-taking Competitive Interruption": "Interruption",
    "Floor-taking Cooperative Interruption": "Interruption",
    "Non-floor Taking Competitive Interruption": "Non-floor-taking Interruption",
    "Non-floor Taking Cooperative Interruption": "Non-floor-taking Interruption",
    "Acknowledgement Backchannel": "Backchannel",
    "Continuer Backchannel": "Backchannel",
    "Reaction Backchannel": "Backchannel",
    "Laughter": "Laughter",
    "Awkward Silence": "Awkward Silence",
    "Non-Speech Noise": "NonContent", "Channel Bleed": "NonContent",
    "Speech, Non-Linguistic": "NonContent",
}

ENTRY = re.compile(
    r"(\d\d:\d\d:\d\d,\d{3}) --> (\d\d:\d\d:\d\d,\d{3})\n\[([^\]]+)\] ?(.*)"
)

def to_ms(t):
    h, m, rest = t.split(":")
    s, ms = rest.split(",")
    return ((int(h) * 60 + int(m)) * 60 + int(s)) * 1000 + int(ms)

def read_events(srt_text):
    for start, end, label, text in ENTRY.findall(srt_text):
        yield to_ms(start), to_ms(end), label, CANONICAL[label], text

sample = """1
00:00:12,340 --> 00:00:13,020
[Acknowledgement Backchannel] Okay.
"""
for event in read_events(sample):
    print(event)
# (12340, 13020, 'Acknowledgement Backchannel', 'Backchannel', 'Okay.')

The card also notes:

• Label strings are case-sensitive and stored verbatim inside the brackets. Transcript text can be empty for events with no words.
• Turn boundaries, interruption types, and backchannel categories can depend on context, and the annotations may contain subjective decisions or errors.
• The 420 sessions span six conversation types, from 61 Instructional to 81 Narrative/Storytelling, recorded on topics and scenarios rather than scripts.

Human labels and model labels

Human labels cost more and are slower. They are also where the definitions come from. TurnBench had three annotators label each segment of its 30-hour evaluation corpus. Agreement reached a Fleiss' kappa of 0.78 at the frame level, event onsets matched at a boundary F1 of 0.94 to 0.96 within 200 ms, and 85.8% of annotator events survived into the consensus gold. Retention was lower in the high-overlap styles (about 84% to 85%) than in Instructional, Narrative, and Task-Oriented (about 88% to 89%) (paper).

Labels from rules and models are cheaper to produce at scale. Published work uses several kinds:

• Timing rules. eot-bench labels the last silence in a user turn as the end and every earlier one as a hold. Full-Duplex-Bench applies a length rule to backchannels.
• Labels from another model. DuplexJev trained an end-of-turn task on otoSpeech-141h audio with labels from DualTurn, a model itself trained on otoSpeech (model card).
• Human and automatic tiers. SmoothConv is human-verified for evaluation, and DuplexConv adds 2,000 hours labeled by an automatic pipeline for training (paper).
• No labels at all. Voice Activity Projection learns by predicting future voice activity on both channels, with no turn labels during training (Ekstedt and Skantze, 2022). You still need labeled data to evaluate it.

A workable split: model labels for scale, human labels for the evaluation set and for the classes models confuse, such as backchannel against interruption onset.

What to ask a vendor

• Which labels, and where are the written definitions? Ask for the guideline, and ask how the labels map to TurnBench's canonical categories or to the scheme you already use.
• Labeled per speaker channel, or on mixed audio? Per-channel labels keep the timing of overlaps exact.
• Who made the labels? Human, model, or model then human review. If human, how many annotators per segment, and what agreement did they reach?
• What time resolution and boundary convention? Milliseconds, frames, or words, and whether a label marks the onset, the offset, or a span.
• How are ambiguous events handled? Dropped, marked as excluded, or forced into a class.
• How many examples of each class? Get counts for backchannels and each interruption type by conversation type, since a model cannot learn a class it rarely sees.
• Do the license terms cover the labels and models trained on them? Some research licenses restrict derived data and model artifacts.
• Can you audit a sample? Pick 20 interruptions and 20 backchannels at random, play each channel alone, and check the label.

Where oto fits

• A hand-labeled training set. otoSpeech-full-duplex-turn-104h, the TurnBench training split, has 17 labels on every speaker channel, 48 kHz audio, and a gated research license.
• A model trained on this kind of label. otoTurn was fine-tuned on oto conversation with human-reviewed end-of-turn and floor-hold labels and cut false interruptions by 47.6% at 300 ms.
• Projects built on these labels. TurnBench baselines, TurnMoshi, and VAP-OTO are listed on oto.earth/use-cases.
• Commercial licensing and custom labels. Licensed datasets run from $10 to $100 per hour of audio, and turn-taking annotation adds to the price. Annotation to your own label set is quoted per project. Write to agent@oto.earth.

Frequently asked questions

What labels do I need to train a turn-taking model?

At minimum end of turn and hold at each pause; add backchannel and floor-taking interruption if your agent must keep talking through "mm-hm" and stop for real barge-ins.

What is the difference between a backchannel and an interruption?

A backchannel shows attention and leaves the floor with the speaker; an interruption takes the floor mid-turn. At onset they can sound the same, which is why benchmarks score them separately.

What are the 17 labels in otoSpeech-full-duplex-turn-104h?

Normal Turn, Strong Floor Hold, Bounded Response, Filler, Overlap, three backchannel types, four interruption types, Laughter, Awkward Silence, Non-Speech Noise, Channel Bleed, and Speech, Non-Linguistic.

Can I use automatic labels instead of human annotation?

For training at scale, often yes, and several published models do. For the evaluation set and for backchannel versus interruption, human labels with measured agreement are safer.

How do TurnBench labels compare with Full-Duplex-Bench labels?

TurnBench uses 17 hand-annotated categories grounded in conversation analysis; Full-Duplex-Bench v1 classifies a model's output with rules, such as under one second and under two words for a backchannel.

Is there a dataset with interruption and backchannel labels?

Yes. otoSpeech-full-duplex-turn-104h labels both, with subtypes, on 104.94 hours of two-person English conversation under a research license.

For more labeled datasets and turn-taking benchmarks, see the fullduplex.ai catalog.

Related guides

• End-of-Turn Detection: Benchmarks and Training Data
• How to Check a Two-Channel Speech Dataset in 30 Minutes
• Free Conversation, Role-Play, or Scripted: What Each Teaches a Speech Model