Egocentric video dataset with full-duplex conversation.
otoSpeech Egocentric captures people talking while they work together, from the point of view of the person doing the work. Two to four people fold towels, tidy everyday objects, and play Jenga. Everyone has their own close microphone on a separate channel.
| Task-oriented | Language | Speakers | Media | License | Actions |
|---|---|---|---|---|---|
| English | 2-4 | On request | |||
English conversation with first-person video, as two to four people fold towels, tidy everyday objects, or play Jenga.
|
Tasks
Towel folding
Six towels in three colors are folded and sorted as requested. The person folding hears requests, changes, and questions while their hands are busy.
Tidying up
Two cups, two books, and two towels go from a table to a shelf, a basket, or a tray. The person carrying them walks a short way, so their distance and direction to each speaker change.
Jenga
Everyone takes turns on a standard 54-block tower. Players advise, ask questions, and chat while one person’s hands are on the blocks.
18 recording conditions
Each of the three tasks runs with two, three, and four people. In one setting the main worker talks freely. In the other they work without words and respond by nodding, pointing, and acting, while the others keep talking.
What each session includes
- Audio
- A close microphone for every person, plus microphones at the main worker’s left and right ears: 4 channels with two people, 5 with three, 6 with four.
- First-person video
- Head-mounted cameras on the main worker and one other participant.
- Fixed video
- A wide view of the room and a close view of the hands and task.
- Labels
- Who spoke, who they addressed, gestures, hand actions and movement, the objects involved, and the result.
Why record egocentric video in full-duplex
When several people talk around someone who is working, a model has to know who is speaking, who they are speaking to, and what was in view at that moment. A mixed track makes overlapping speech hard to attribute. A separate channel for each speaker keeps each voice on its own track, lined up with the video.
Voice AI that works beside people needs this, for example on home robots and smart glasses.
Listen to related samples
| Casual | Language | Speakers | Media | License | Actions |
|---|---|---|---|---|---|
| English | 2 | On request | |||
English conversation with a camera view of each speaker, synced to two-channel audio. Faces are mosaicked in the sample to protect the speakers' privacy. The licensed data has no mosaic.
| |||||
| English | 2-5 | On request | |||
English conversation recorded in person, with two to five people. Sessions are casual or deliberative.
|
FAQ
What is a full-duplex egocentric dataset?
Conversation recorded with first-person video and a separate audio channel for each speaker. The video shows a speaker’s point of view, and the channels show who said what, even when people talk at the same time.
What tasks does otoSpeech Egocentric cover?
Towel folding, tidying everyday objects, and Jenga. Each task runs with two, three, and four people, and with the main worker either talking freely or working without words.
How is it different from otoSpeech Video?
otoSpeech Video has a camera view of each of two speakers. otoSpeech Egocentric shows the scene from a speaker’s point of view, with two to four people doing a task together.
Can I ask for other tasks or languages?
Yes, please ask. Tell us the tasks, languages, and number of speakers you need, and we will tell you what we can do.
Read more: About otoSpeech Egocentric.
More otoSpeech data: English, Japanese, Korean, video, and multi-speaker conversation, or the full catalog.