Egocentric Video and Speech Datasets Compared

What the columns mean

First-person video: a camera worn on the head or glasses, so the frame shows what the wearer sees. This is the closest stand-in for a robot's head camera.
Per-speaker close audio: a microphone near each person's mouth, recorded on its own channel. A head-worn microphone hears everyone mixed together, which is the realistic input. The close channels tell you who said what, even when people overlap.
People: how many people take part in one recording. Addressee questions ("was that for me?") only appear with three or more.
Conversation or task: whether people are talking to each other, doing a physical task, or both.
Transcripts: whether speech is written out with timing and speaker labels. Spoken narration of one's own actions is a different thing from a dialogue transcript.
License: the terms you train under. Check this before you compare hours.

The comparison

Dataset	First-person video	Per-speaker close audio	People	Conversation or task	Transcripts	License
EgoCom	Yes, glasses on 2 or 3 of the 3 people	No, stereo audio on each wearer	3	Conversation	Human, word-level, with speakers	Modified MIT (per README)
Ego4D	Yes	No	Varies; the social subset is in groups	Daily-life activity	On the audio-visual diarization subset, English only	Ego4D license agreement
Ego-Exo4D	Yes, plus third-person cameras	No, a 7-channel array on the glasses	Activity-centered	Skilled tasks: sports, music, cooking, repair	Narration and expert commentary, not dialogue	Ego-Exo4D license agreement
EgoVerse	Yes	Not described in the paper	2,087 demonstrators in total	Manipulation, 1,965 tasks	Language annotations of actions	Not stated in the paper
EPIC-KITCHENS-100	Yes, head-mounted	No	One camera wearer	Kitchen activity	Spoken action narrations, not dialogue	CC BY-NC 4.0
Seamless Interaction	No, one video per participant	Yes, one denoised 48 kHz file per participant	2	Conversation	Time-aligned	CC BY-NC 4.0
MM-F2F	No, online conversation videos	No, audio comes from the source video	2 per clip	Conversation	Word-level, with turn-taking and backchannel labels	Annotations and video links; videos stay with their uploaders
otoSpeech Egocentric (designed)	Yes, on the main worker and one other person, plus wide and close fixed views	Yes, a close microphone for everyone, plus left- and right-ear microphones on the main worker	2 to 4	Conversation during shared tasks	Labels for speaker, addressee, gesture, hand action, objects, and result	On request

Sizes, from the same sources: EgoCom 38.5 hours with 34 speakers; Ego4D 3,670 hours from 931 camera wearers; Ego-Exo4D 1,286 hours in the paper; EgoVerse 1,362 hours in its current release; EPIC-KITCHENS-100 100 hours from 45 kitchens; Seamless Interaction over 4,000 hours from over 4,000 participants; MM-F2F over 210 hours of video. otoSpeech Egocentric is specified as 3 tasks run in 18 conditions. Ego-Exo4D's documentation now lists 1,422 hours, and EPIC-KITCHENS offers commercial licenses by email (EPIC-KITCHENS).

Primary pages for the rows above: EgoCom README, Ego4D Social benchmark, Ego-Exo4D documentation, EgoVerse, Seamless Interaction dataset card, MM-F2F repository, and the otoSpeech Egocentric design.

Reading the table

The sets fall into three groups, and none covers all six columns.

Robot learning sets (EgoVerse, EPIC-KITCHENS-100, and Ego-Exo4D) are strong on hands and tasks. Their language is a description of actions: EgoVerse's dense annotations, EPIC-KITCHENS' narrations recorded through a pause-and-talk tool, and Ego-Exo4D's narrations and expert commentary. These sets suit manipulation policies and activity recognition. They do not show people talking to the person who is working.

Among first-person conversation sets, EgoCom is the clearest example: three people talk while two or three of them wear recording glasses, with 240,000 human-transcribed words and speaker labels. Its paper used the data for turn-taking prediction. There is no shared physical task, and the README notes that each microphone hears every voice, with the wearer loudest. Ego4D adds a social benchmark built for this viewpoint, with "Looking at me" and "Talking to me" tasks that ask whether a visible person is looking or speaking to the camera wearer. That is the closest public resource for addressee detection from a first-person view.

Seamless Interaction and MM-F2F are face-to-face conversation sets. They show how two people behave in conversation, from voice and face to timing. Seamless Interaction keeps each participant's audio and video separate, under a non-commercial license. MM-F2F labels turn-taking and backchannels on web video, and it releases annotations and links instead of the video itself. Both are dyadic and filmed from outside, so neither has a wearer's view.

Why egocentric data needs both kinds of microphone

A head-worn microphone records what a robot or a pair of glasses would hear: everyone at once, at different distances, with the wearer's own voice loudest. That is the input your model gets in use. A close microphone on each person records the answer: which words came from whom, including during overlap.

A dataset with only head-worn audio forces you to estimate speaker labels from a mixture, and the estimate is weakest exactly where people talk over each other. A dataset with only close microphones has clean labels and an unrealistic input. You want both on one clock, lined up with the video. That pairing is also what makes turn-taking and interruption labels reliable in a group.

Video adds its own requirement. VideoFDB, a benchmark for full-duplex agents that see and speak, found that current agents use video to answer explicit visual questions but not for continuous grounding during conversation (Mazumdar et al., 2026). Closing that gap needs training data in which gaze, pointing, and speech carry meaning together.

How to choose

Match the dataset to the behavior you are training:

A manipulation policy from human video: EgoVerse, EPIC-KITCHENS-100, or Ego-Exo4D.
Addressee detection from a first-person view: Ego4D's social benchmark and EgoCom.
Face and voice timing for an avatar or a robot face: Seamless Interaction, MM-F2F, and otoSpeech Video.
Spoken requests during shared physical work in a group: no public set covers this today. oto designed otoSpeech Egocentric for it, and custom collection can fit your tasks.

Then check the license. Two sets in the table are CC BY-NC 4.0, two require a signed agreement, and MM-F2F points to videos owned by their uploaders. For commercial training, read each license and consent scope before you count hours.

What to check in a sample

Speech exists and is usable: play a few minutes. Some first-person sets have audio on only part of the footage.
Sync: clap or tap events should line up across every camera and microphone within a frame.
Channel separation: in an overlap, play each close channel alone. You should hear one main voice.
Labels match the claim: if the spec lists addressee labels, open the files and check that each utterance has one.
Faces: confirm whether faces may be used without blurring for your purpose.

Where oto fits

oto builds conversation data with each speaker on a separate channel, in English, Japanese, and Korean.

otoSpeech Egocentric (designed): two to four people fold towels, tidy everyday objects, and play Jenga across 18 conditions, with the main worker either talking or working without words. The design has 4 to 6 audio channels depending on group size, head-mounted and fixed cameras, and labels for speaker, addressee, gesture, hand action, objects, and result (dataset page).
otoSpeech Video: two-person English conversation with one camera per speaker, two-channel audio at 48 kHz, and a transcript for each speaker (video dataset).
Multi-speaker data: Face-to-Face with two to five people on 16 channels, and three-speaker conversation with one file per speaker (multi-speaker datasets).
Collection to specification: other tasks, camera positions, languages, and group sizes.
Pricing: licensed datasets run from $10 to $100 per hour of audio; the license and labels add to the price, and custom collection is quoted per project. Contact agent@oto.earth.

For what a humanoid needs beyond the data itself, see What Conversation Data Does a Humanoid Robot Need?

Data for voice AI

Request samples or a quote

Tell us what you are building, and we will send samples and pricing for the data that fits.

Request samples See the datasets Commission a collection

Frequently asked questions

Which egocentric datasets include conversation?

EgoCom is built around three-person conversation, and Ego4D includes audio-visual diarization and social interaction subsets. Most other first-person sets focus on actions and have narration instead of dialogue.

Does Ego4D have audio?

Yes, for portions of the footage. Its audio-visual diarization and social benchmarks use it, but the audio comes from the camera wearer's device, so voices are mixed.

Is there an egocentric dataset with a separate microphone for each speaker?

Among public sets we checked, no. otoSpeech Egocentric is designed with a close microphone for every person plus ear-level microphones on the main worker.

Can I use EPIC-KITCHENS or Seamless Interaction commercially?

Both are CC BY-NC 4.0. EPIC-KITCHENS offers commercial licenses by email, and the Seamless Interaction license says commercial use needs explicit permission.

What is the difference between egocentric and exocentric video?

Egocentric video comes from a camera worn by a person, showing their view. Exocentric video comes from cameras placed around the scene. Ego-Exo4D records both at once.

For more datasets, benchmarks, and models, see the fullduplex.ai catalog.

otoearth, Inc.