How robot conversation differs from a phone call
A phone agent hears one person, close to a microphone, in a quiet channel. Its main question is when to speak, which is the job of end-of-turn detection. A robot has to answer two earlier questions first: is this speech for me, and what should my hands do while I answer?
These are the differences that change the data:
- Far-field, moving microphones: the robot's microphones sit on its head or torso, away from every speaker's mouth. When it turns or walks, the distance and direction to each person change from second to second.
- Ego-noise and echo: the robot hears its own motors, fans, gears, and footsteps, plus its own voice coming back from its speaker. Echo cancellation handles the voice. Motor noise changes with every movement.
- Several people at once: two people may talk to each other, to the robot, or over each other. Overlap is normal in groups, and diarization from a mixed track struggles most there.
- Addressee detection: most speech near a robot is not meant for it. 1X says NEO's speech system recognizes when it is being addressed and responds only when appropriate (The Robot Report).
- Speaking while acting: a robot that goes silent while it moves, or stops moving while it talks, is slow to work with. It also has to accept an interruption such as "stop" in the middle of a motion.
- Gaze and gesture: "put that one over there" means nothing without a pointing hand and a line of sight. Zhu, Miyoshi, and Okafuji list overlapping speech, identifying the intended addressee, and cues such as gaze direction and body orientation as what multi-party robot interaction requires (Zhu et al., 2025).
- Grounding in objects and tasks: the robot has to connect words to the objects in view and know whether the task it was given has been done.
What published robot systems do with speech
The control side is moving fast. Several systems now take natural language and act on it in real time.
- Figure Helix: a vision-language-action model with a slow vision-language model at 7 to 9 Hz and a fast visuomotor policy at 200 Hz. Figure trained it on about 500 hours of teleoperated behavior, with text instructions written after the fact by an auto-labeling model (Figure, Helix).
- Figure Project Go-Big: first-person human video gathered in Brookfield homes. After training only on human video, Helix navigated from commands such as "go to the fridge" (Figure, Project Go-Big). The announcement does not say whether that video includes speech.
- Gemini Robotics: Google DeepMind says it responds to everyday conversational language in different languages, watches for changes in its instructions, and adjusts, which it calls steerability (Google DeepMind).
- VITA-E: two copies of a vision-language-action model run side by side so a Fourier GR2 humanoid can see, listen, talk, and act at once. In 30 trials it stopped its own speech when interrupted and obeyed emergency stops every time, and switched tasks in 93.3% of trials. Its average voice response latency across ten tests was 2.26 seconds (Liu et al., 2025).
Read together, these systems hear language as a command from one user. Helix learns from text instructions, and VITA-E's training data is demonstration video, instructions, and actions, reformatted and augmented to teach control tokens such as a halt. None of these descriptions mention training on group conversation heard from the robot's position. The next step for robot listening is data where instructions arrive as they do in a real room, spoken over other talk and sometimes meant for someone else.
What datasets exist today
Public data splits into two families. Robot learning sets have the first-person view and the hands. Conversation sets have the voices.
| Dataset | What it has | What a robot still needs |
|---|---|---|
| Ego4D | 3,670 h of first-person daily life, audio on portions, a "Talking to me" benchmark | Each speaker on a separate channel |
| Ego-Exo4D | 1,286 h of skilled activity from first- and third-person views, 7-channel audio | Group conversation |
| EgoVerse | 1,362 h of manipulation demonstrations across 1,965 tasks | Any speech; the paper does not mention audio |
| EgoCom | 38.5 h of three-person conversation from glasses, word-level transcripts | A physical task; per-speaker channels |
| AMI | 100 h of meetings, close-talking and far-field microphones, video, gaze and gesture labels | First-person view; work with objects |
| CHiME-6 | 20 dinner parties, four people, binaural microphones on each person plus six arrays | First-person view; task labels |
| Seamless Interaction | Over 4,000 h of two-person interaction, one audio file and one video per participant | Three or more people; a first-person view |
Sources: Ego4D and Ego-Exo4D papers, the EgoCom README, the AMI overview, the CHiME-5 data page (CHiME-6 reuses the same recordings), and the Seamless Interaction paper.
EgoCom comes closest to a robot's situation: three people talk while two or three of them wear recording glasses. It has no shared physical task, and each wearer's stereo audio carries everyone's voice. Ego4D's "Talking to me" task asks whether a visible person is speaking to the camera wearer, which is the addressee question a robot faces. A side-by-side comparison of the first-person sets is in Egocentric Video and Speech Datasets Compared.
What a robot conversation dataset should contain
The requirement fits in one sentence: what the robot would see, what each person said, and who said it to whom, all on one timeline. In practice that means checking for the following.
| Requirement | Why the robot needs it | What to check |
|---|---|---|
| Microphones where the robot's ears would be | That is the input the model gets at run time | Are there head- or ear-level channels, synced with the rest? |
| A close microphone for every person | It is the answer key for who said what during overlap | One channel per speaker? How much bleed? |
| Three or more people | Addressee and side talk only appear in groups | Which group sizes, and how many sessions of each? |
| Addressee labels | Training "is this for me" needs a target | Is the addressee marked per utterance, by people? |
| First-person video | Gaze, pointing, and objects in view | Head-mounted camera? How tight is the sync? |
| A physical task with objects | "That one" must resolve to a thing | Are objects, actions, and results labeled? |
| Talk during action, and action without talk | The robot must work while listening and answer with a nod or a move | Are there sessions where the worker stays silent? |
| Consent for faces and voices | Commercial training needs explicit rights | Can faces be used unblurred? Is AI training in the consent form? |
Turn-taking still matters inside all of this. A robot in a group needs the same backchannel and interruption behavior as a voice agent, and the training data guide on fullduplex.ai explains why separate channels matter for it.
What human data cannot cover
Recordings of people do not contain a robot's motor noise, its speaker echo, or its joint control. Zhu and colleagues also note that the applicability of human-only multi-party corpora to robot interaction remains limited, and their own robot dataset, a Sota robot with two people in a mall-like setting, runs to 82 minutes (Zhu et al., 2025).
That points to a split we recommend. Human conversation data teaches how groups share the floor, who is talking to whom, and how talk lines up with hands and gaze. Your own platform supplies its noise and echo, recorded separately and mixed in during training, plus a small set of on-robot sessions for adaptation.
Where oto fits
oto builds conversation data with each speaker on a separate channel, in English, Japanese, and Korean. What it offers for humanoid work:
- otoSpeech Egocentric (designed): a first-person conversation dataset designed for humanoids. Two to four people fold towels, tidy everyday objects, and play Jenga across 18 conditions, with a close microphone for everyone, left- and right-ear microphones on the main worker, head-mounted cameras, and labels for speaker, addressee, gesture, hand action, objects, and result. See the design and the dataset page.
- otoSpeech Face-to-Face: in-person English conversation with two to five people on 16 microphone channels at 48 kHz and 24-bit, with a lavalier, studio microphones, and a room microphone on the same speech (details).
- Multi-speaker data: English conversation among three to five people with a separate recording for each speaker (multi-speaker datasets).
- Video: two-person English conversation with one camera per speaker and one speaker per audio channel (video dataset).
- Collection to specification: your tasks, group sizes, languages, camera positions, and labels.
- Pricing: licensed datasets run from $10 to $100 per hour of audio; the license and labels such as turn-taking annotation add to the price, and custom collection is quoted per project. Write to agent@oto.earth.
Data for voice AI
Request samples or a quote
Tell us what you are building, and we will send samples and pricing for the data that fits.
Frequently asked questions
What training data do humanoid robots need for conversation?
Multi-person conversation with first-person video, a separate audio channel for each speaker, microphones near the robot's ears, and labels for who addressed whom, recorded while people work on physical tasks.
Can I train a robot's voice interface on phone call data?
Only for part of it. Phone data teaches two-person turn-taking on clean close audio, but it has no far-field sound, no groups, no addressee choice, and no link to objects or actions.
Is there a public egocentric dataset with conversation and physical tasks?
Not one that has both with per-speaker audio. EgoCom has three-person conversation without a task, and robot learning sets such as EgoVerse have tasks without speech.
How do robots know when someone is talking to them?
They learn addressee detection from data that labels who each utterance was meant for, using voice, gaze, and body orientation. Ego4D's "Talking to me" task and multi-party robot datasets provide such labels.
Does human conversation data transfer to robots?
Partly. It teaches group turn-taking and grounding, but it lacks the robot's motor noise and echo, so teams add platform noise and a small amount of on-robot data.
Where can I get egocentric conversation data for a humanoid?
oto offers otoSpeech Egocentric, a designed first-person conversation dataset, along with Face-to-Face, multi-speaker, and video conversation data, and collection to specification at agent@oto.earth.
For more datasets, benchmarks, and models, see the fullduplex.ai catalog.
Related guides
- Egocentric Video and Speech Datasets Compared
- Multi-Party Conversation Data for Robots That Join a Group
- End-of-Turn Detection: Benchmarks and Training Data