otoSpeech Video: multimodal conversation with a camera on each speaker

Watch a sample

This 30-second clip comes from a conversation about manga, anime, and movies. The player shows both cameras and both audio tracks, and you can switch either speaker off to hear one side alone.

• Manga, anime, and movies

AI is getting eyes and a body

Voice AI started with audio alone. GPT-4o, in 2024, took text, audio, images, and video into one network and answered by voice in about 320 milliseconds on average.[1] In 2025, Google DeepMind put Gemini on robots, including work with Apptronik’s Apollo humanoid.[2] Later that year, 1X opened orders for NEO, a humanoid it said would ship to homes in 2026.[3]

A humanoid in someone’s home will be read the way people read each other. In conversation, people keep adjusting their words, voice, and gestures, and they mirror smiles and share gaze as they go.[4] Timing counts too. Emo, a robot face from Columbia, learned from videos of human faces to predict a smile about 840 milliseconds ahead and smile along with the person. Its authors say delayed mimicry looks insincere.[5]

Learning this takes people who are on camera and on a microphone at the same time. Meta built the Seamless Interaction Dataset, with over 4,000 hours of two-person interaction, to model it for virtual agents.[4] otoSpeech Video is remote, so each person faces their own camera. That is a different view from a robot’s, but it lines up each face with that person’s own clean audio, which we think is the useful part for learning expression timing.

Full-duplex is moving to video

Several teams now build AI that holds a conversation with video in the loop. Tavus splits the job into three models: Raven-1 reads a user’s voice, video, and screen, Sparrow-2 decides when to listen, wait, or speak, and Phoenix-4.5 renders the face, expressions, and upper body in real time.[6] ByteDance’s SeedRealtime, released in August 2026, runs perception and speech in parallel on continuous audio and video.[7] MiniCPM-o 4.5 aligns what it sees, hears, and says on one timeline.[8] Wan-Streamer generates video as well as audio from a single model, with about 550 milliseconds of total latency.[9]

The hard part is still behavior. VideoFDB, a 2026 benchmark led by NVIDIA researchers, tested vision-speech models and speech-to-avatar pipelines on 237 clips from real video calls. No model it evaluated came close to the human reference, and the largest gaps were in pause handling and nonverbal backchannels. Many models described the user’s appearance instead of conversing, and adding video often left their output almost unchanged. Speech-to-avatar pipelines fell short on nonverbal cues, because the avatar follows its own speech and cannot react during the user’s turn.[10]

Closing that gap takes conversation where each person’s face and voice line up in time, including what they do while the other person talks. otoSpeech Video records exactly that.

Why the audio stays separate

An audio-visual model has to know which voice belongs to which face. With one mixed track, that link has to be estimated, and the estimate is weakest when both people talk at once. In otoSpeech Video each voice has its own channel and each face has its own video, so the link comes from the recording.

The cameras also capture listening. A nod or a smile while the other person talks leaves nothing on that person’s audio channel, but it is on their video, lined up with what the other speaker is saying.

What is included

• One camera video per speaker
• Stereo audio with one speaker on each channel
• A transcript for each speaker

The transcripts have no manual review, and the dataset has no turn-taking labels. If your work needs them, we can add them to your specification.

Faces and consent

Faces are mosaicked in the public sample to protect the speakers’ privacy. The licensed data has no mosaic, and the speakers agreed to share their unmosaicked video with licensees.

fullduplex.ai, a site oto runs, describes VideoFDB, a full-duplex benchmark for video calls.