otoSpeech Egocentric: full-duplex conversation from a first-person view

Why humanoids need this data

Speech is likely to be a main interface for humanoids. Brett Adcock, Figure’s CEO, has predicted it will be how people operate them, with language models turning spoken requests into robot commands.[1] Google DeepMind describes Gemini Robotics as adjusting when instructions change, and calls that steerability.[2]

That asks a lot of a robot’s listening. It has to stop when someone tells it to, even mid-motion. VITA-E, a research system tested on a physical humanoid, listens, talks, and acts at once, and its authors report high success on emergency stops and speech interruptions.[3] It also has to follow instructions that depend on the scene, such as “put that one over there,” and tell a request from a side comment when several people talk around it.

Training for that takes three things on one timeline: what the system would see, what everyone said, and who said it to whom. Egocentric video covers the first. A separate channel for each speaker covers the second, even when people talk over each other. Labels cover the third.

What existing data covers

The egocentric sets we looked at for robot learning center on hands and movement. Figure’s Project Go-Big collects first-person video in Brookfield homes to train its Helix model, and its announcement does not say whether that video includes speech.[4] EgoVerse gathers egocentric human demonstrations with manipulation annotations, and its abstract does not mention audio.[5]

Conversation has been studied from this viewpoint too. EgoCom recorded natural conversation from each person’s point of view, with 38.5 hours and 34 speakers, and its description covers conversation with no physical tasks.[6] otoSpeech Egocentric is designed to put conversation and shared physical work in one recording.

A person stands in for the robot here, so the data does not cover a robot’s own motor noise, speaker echo, or joint control.

The tasks

• Towel folding: six towels in three colors are folded and sorted as requested. The person folding hears requests, changes, and questions while their hands are busy.
• Tidying up: two cups, two books, and two towels go from a table to a shelf, a basket, or a tray. The person carrying them walks a short way, so their distance and direction to each speaker change.
• Jenga: everyone takes turns on a standard 54-block tower. Players advise, ask questions, and chat while one person’s hands are on the blocks.

18 recording conditions

Each of the three tasks runs with two, three, and four people. In one setting the main worker talks freely. In the other they work without words and respond by nodding, pointing, and acting, while the others keep talking. Comparing the two shows what a system can do through action alone.

What each session includes

• Audio: a close microphone for every person, plus microphones at the main worker’s left and right ears
• First-person video: head-mounted cameras on the main worker and one other participant
• Fixed video: a wide view of the room and a close view of the hands and task
• Labels: who spoke, who they addressed, gestures, hand actions and movement, the objects involved, and the result

fullduplex.ai, a site oto runs, explains how full-duplex voice AI is built and what training data it needs.