Why two-person data is not enough
In a two-person conversation, when one person stops, the other is usually next. A model trained on pairs learns a simple rule: wait for the other side to finish, then speak. Most end-of-turn detection work is built on that setup, and the MP-Bench authors note that recent benchmarks mostly evaluate two-person interaction.
Add a third person and the rule fails in several ways:
- Anyone can take the next turn. A pause after someone speaks does not mean the robot should fill it.
- Most speech is not for the robot. Two people may be talking to each other in front of it.
- Several listeners give feedback at once. A "yeah" from one person and a laugh from another are backchannels, not bids for the floor.
- Conversations split. In a group of four, two pairs can talk at the same time, and the robot has to follow one without answering the other.
Zhu, Miyoshi, and Okafuji give a concrete case: a person asks a robot for help while someone standing nearby comments casually on the store layout. A robot that answers every utterance it detects interrupts the people around it or answers the wrong person (Zhu et al., 2025). 1X describes the same goal for its NEO humanoid, which it says recognizes when it is being addressed and responds only when appropriate (The Robot Report).
Addressee and floor control
Two ideas carry most of the multi-party problem.
- Addressee: who an utterance is meant for. People signal it with names, gaze, body orientation, and content. A question to the robot while looking at the robot is the easy case. A question to the room while looking at a colleague is the hard one.
- Floor: who holds the right to speak right now. A system has to decide when to enter, when to hold, and when to give way. It also has to tell a real takeover from overlapping speech that does not claim the floor, which is the multi-party version of handling an interruption.
Both decisions need labels that two-person data does not have. Addressee labels need at least three people. Floor labels need to mark, for each overlap, whether the floor changed hands.
What the benchmarks show
Three benchmarks released in 2026 test these decisions directly.
| Benchmark | What it tests | Data | Finding |
|---|---|---|---|
| MultiPartyTurnBench | End-of-turn and interruption detection with 2 to 5 speakers | 240 synthetic, channel-separated conversations in six interaction types | Five full-duplex agents scored 46.9% to 54.7% on selective interruption handling |
| MultiTalkBench | Long multi-party dialogue, including addressee selection | 104 real recordings, 60 English and 44 Chinese, averaging 32.6 minutes | Released with a model that beats Moshi, MiniCPM-o-4.5, and Qwen3-Omni on it |
| MP-Bench | Voice agents as participants: turn-taking awareness and response appropriateness | Multi-party conversations with comprehension questions | 12 voice agents tested; real-time agents stay at or below 22% on comprehension and near chance on turn-taking |
Sources: Mathur and Manocha, 2026; Wang et al., 2026 and the MultiTalkBench card; Shih et al., 2026.
MultiPartyTurnBench gives the sharpest picture. Learned end-of-turn detectors reached 82.5% to 92.6% recall at 4.3% to 11.8% false-positive rates, so end-of-turn detection holds up reasonably well even with five speakers. Interruption detection does not. The same five agents scored 86.1% to 95.8% balanced accuracy on entering a turn, then fell to near chance on deciding which overlaps to yield to. At the extremes, Gemini-3.8-Live yielded to 97.7% of backchannels, and MiniCPM-o-4.5 yielded to only 1.4% of genuine interruptions: one stops for almost every "mm-hm", and the other talks over people who are taking the floor.
Our reading of all three: current agents handle the end of a turn better than the overlaps inside it. They cannot yet tell a takeover from a backchannel or a side remark, and we think that is as much a training data problem as a model problem. The evaluation overview on fullduplex.ai covers how these metrics are computed.
Synthetic and real multi-party data
The MultiTalk authors write that open multi-party speech corpora remain small and are not designed for full-duplex modeling at the codec-frame level. Their answer was 57.6k hours of synthetic training dialogue with controllable participant count, overlap, backchannels, interruptions, and addressee shifts (Wang et al., 2026). MultiPartyTurnBench is synthetic too, built from scripted dialogue, synthesized voices, and timing statistics taken from AMI, and it checks its detectors against real two-person TurnBench audio and four-person AMI meetings (Mathur and Manocha, 2026).
Synthetic data gives coverage and exact labels. Its timing is generated, not observed, and MultiPartyTurnBench draws on a pool of 24 synthetic voices. Real recordings give the timing, prosody, and acoustics that a robot will meet in a room. We think a robot team needs both: synthetic dialogue to cover rare floor events at scale, and real group conversation to learn timing and to evaluate on. Robot-specific data is still tiny. The multi-party human-robot dataset Zhu and colleagues built, with a Sota robot and two people, runs to 82 minutes and 1,111 dialogues labeled with speaker and intended recipient.
Multi-party datasets you can use
| Dataset | People per session | Audio | Setting | License |
|---|---|---|---|---|
| AMI | 4 in the scenario meetings, about two-thirds of the data | Close-talking and far-field microphones, plus video | 100 h of meetings in three rooms | CC BY 4.0 |
| ICSI | Group meetings | Close-talking microphones, as separate files or one mix | About 70 h of meetings | CC BY 4.0 |
| CHiME-6 | 4 | Binaural microphones on each person plus six 4-microphone arrays, 16 kHz | 20 dinner parties in homes, about 50 h | CC BY-SA 4.0 |
| Basis Conversations 1500 | 2 to 4 | One 48 kHz channel per speaker | 2,645 speakers, median 33 minutes per conversation | Custom license, gated |
| MultiTalkPT / FT | Varies | Synthetic, with a speaker-to-channel map | English and Chinese, 57.6k h | CC BY-NC 4.0 |
| otoSpeech Face-to-Face | 2 to 5 | 16 channels, 48 kHz, 24-bit, in one room | Casual and deliberative sessions | On request |
| otoSpeech 3-Speaker | 3 | One 48 kHz mono file per speaker | Spontaneous conversation | On request |
Sources: AMI, ICSI, the CHiME-5 data page and OpenSLR 150, the Basis dataset card, the MultiTalkPT card, and oto's Face-to-Face and 3-Speaker pages.
The real recordings fall into two kinds, and a robot needs both:
- Shared-room recordings: AMI, ICSI, CHiME-6, and otoSpeech Face-to-Face put everyone in one room, so each close microphone also picks up the other voices. That bleed is what a robot's own microphones will hear, and it is the material for separation, diarization, and far-field speech. Face-to-Face lets you compare a lavalier, studio microphones, and a room microphone on the same speech.
- Separated recordings: Basis Conversations 1500 and otoSpeech 3-Speaker give each speaker a clean track. That is the cleanest target for floor and turn-taking labels, because you know who spoke at every moment, including during overlap. In the otoSpeech 3-Speaker sample, about 22% of speech time has more than one person talking.
What to ask for
- Group sizes: three, four, and five people, with enough sessions at each size to evaluate separately.
- Both channel types: a clean track per speaker for labels, and in-room microphones for realistic input.
- Floor and addressee labels: turn ends, holds, takeovers, backchannels, and who each utterance was for, marked by people.
- Split conversations: sessions where two pairs talk at once. otoSpeech Face-to-Face samples include this case.
- Real timing: unscripted or lightly prompted talk, with turns left where they happened.
- License and consent: AI training rights for every speaker, checked before you compare hours.
Where oto fits
oto builds conversation data with each speaker on a separate channel, in English, Japanese, and Korean.
- otoSpeech Face-to-Face: in-person English conversation with two to five people on 16 microphone channels at 48 kHz and 24-bit (details).
- otoSpeech 3-Speaker: spontaneous English conversation among three people with one mono file per speaker at 48 kHz. Transcripts and turn-taking labels are not included today and can be added to specification (details).
- otoSpeech Egocentric (designed): two to four people working on shared tasks, designed with labels for speaker and addressee and a first-person view (dataset page).
- Used in multi-party evaluation: MultiPartyTurnBench evaluates DualTurn and TurnBench models trained on otoSpeech (use cases).
- Collection to specification: larger groups, other rooms, other languages, and floor or addressee labels.
- Pricing: licensed datasets run from $10 to $100 per hour of audio; the license and labels such as turn-taking annotation add to the price, and custom collection is quoted per project. Contact agent@oto.earth.
For the first-person side of the problem, see What Conversation Data Does a Humanoid Robot Need?
Data for voice AI
Request samples or a quote
Tell us what you are building, and we will send samples and pricing for the data that fits.
Frequently asked questions
What is multi-party conversation data?
Recordings of three or more people talking together, ideally with a separate channel for each speaker so overlapping speech can be attributed.
Why can't a robot learn group conversation from two-person data?
In pairs, the other person is almost always next. With three or more, the robot has to decide whether it is being addressed at all, and two-person data has no examples of that choice.
What is addressee detection?
Working out who an utterance is meant for, from words, names, gaze, and body orientation. Robots in groups need it to avoid answering speech meant for someone else.
Are there benchmarks for multi-party turn-taking?
Yes. MultiPartyTurnBench, MultiTalkBench, and MP-Bench were released in 2026. MultiPartyTurnBench and MP-Bench both find current voice agents weak at multi-party floor control.
Which datasets have multi-speaker conversation with separate channels?
AMI, ICSI, and CHiME-6 have close microphones in a shared room. Basis Conversations 1500 and otoSpeech 3-Speaker have a clean track per speaker, and otoSpeech Face-to-Face has 16 in-room channels with two to five people.
For more datasets, benchmarks, and models, see the fullduplex.ai catalog.
Related guides
- What Conversation Data Does a Humanoid Robot Need?
- Egocentric Video and Speech Datasets Compared
- Turn-Taking Labels: What to Annotate for Voice Agents