otoSpeech 1,125h
Massive. Diverse. Affordable.
The world's most accessible 1,125-hour full-duplex corpus.
BuyFrom $10 per hour
oto builds full-duplex speech datasets for voice AI and humanoids.
Joint research with Sesame otoSpeech is the training set for TurnBenchProjects using otoSpeech
Sesame AITurnBench: Baseline turn-taking models (VAP, Mimi, WavLM) trained on otoSpeech.
NVIDIAMind the Gap: 500 hours of synthetic conversation built from otoSpeech to train multi-talker speech recognition.
PipecatPipecatBackchannel detector: A streaming model that tells backchannels from interruptions, trained and evaluated in part on otoSpeech.Voice Reader: An experimental model for tone of voice and sarcasm, with otoSpeech among its training and evaluation data.
LiveKitLiveKiteot-bench: Evaluates a VAP model trained on otoSpeech across 14 languages.
AmazonTurnNat and SPEARBench: Fine-tunes the weights of DualTurn, a model trained on otoSpeech, then reuses them as an evaluator.
Carnegie Mellon UniversityTurnBench: Baseline turn-taking models (VAP, Mimi, WavLM) trained on otoSpeech.Mind the Gap: 500 hours of synthetic conversation built from otoSpeech to train multi-talker speech recognition.TurnMoshi: A turn-taking model trained and distilled on about 100 hours of otoSpeech.
National Taiwan UniversityTurnBench: Baseline turn-taking models (VAP, Mimi, WavLM) trained on otoSpeech.
Academia SinicaTurnBench: Baseline turn-taking models (VAP, Mimi, WavLM) trained on otoSpeech.
Brno University of TechnologyTurnBench: Baseline turn-taking models (VAP, Mimi, WavLM) trained on otoSpeech.Mind the Gap: 500 hours of synthetic conversation built from otoSpeech to train multi-talker speech recognition.TurnMoshi: A turn-taking model trained and distilled on about 100 hours of otoSpeech.
Anyreach AIDualTurn: A turn-taking model trained on otoSpeech, published with its weights and derived data.DualTurn Endpointing: A small end-of-turn model trained on otoSpeech.
Indian Institute of ScienceTurnMoshi: A turn-taking model trained and distilled on about 100 hours of otoSpeech.
BIGAIDuplexAct-Bench: otoSpeech as evaluation data for natural backchannels.
Peking UniversityDuplexAct-Bench: otoSpeech as evaluation data for natural backchannels.
Shanghai Jiao Tong UniversityDuplexAct-Bench: otoSpeech as evaluation data for natural backchannels.
Johns Hopkins UniversityTurnNat and SPEARBench: Fine-tunes the weights of DualTurn, a model trained on otoSpeech, then reuses them as an evaluator.
University of MarylandMultiPartyTurnBench: Evaluates DualTurn and TurnBench models, both trained on otoSpeech, on multi-party conversation.
JoveoJoveoBarnOwl: An interruption model that lists otoSpeech in its training data.
Adventists.aiAdventists.aiDuplexJev: End-of-speech models fine-tuned on otoSpeech audio with labels derived from DualTurn.
OpenRealtimeOpenRealtimeOpenRealtime: An open realtime voice project that integrates DualTurn, a model trained on otoSpeech, and evaluates it offline.
Induction LabsFrontierML: An end-of-turn benchmark with published trials that train on otoSpeech and on models derived from it.
SoniqoSoniqoVAP-OTO: A VAP model trained on otoSpeech, converted to Core ML and ONNX and added as an optional model in its speech SDKs.
Lhotse communityLhotse communityLhotse: A recipe for downloading and preparing otoSpeech in a speech data toolkit.
Swiss AISwiss AIMultimodal data: Scripts that download otoSpeech and convert it for training.
Logos show the organizations of each project's authors or publishers. They do not mean product adoption or endorsement. See all use cases
otoSpeech datasets
All datasets are full-duplex and recorded with speaker consent.
| Casual | Language | Speakers | Media | License | Listen |
|---|---|---|---|---|---|
| English | 2 | Commercial | |||
English casual conversation with a separate channel for each speaker.
| |||||
| English | 2 | CC BY 4.0 | |||
Raw English conversation recorded in varied real-world conditions. Microphones and rooms differ across sessions.
| |||||
| English | 2 | CC BY 4.0 | |||
Conversations selected from the 280h release by human review, with noise reduction and speech enhancement. It also adds conversations recorded after that release.
| |||||
| English | 2 | oto Speech License | |||
420 conversations annotated by hand for turn-taking. This is the training set for TurnBench, the open benchmark from Sesame.
| |||||
| English | 2 | On request | |||
English conversation with a camera view of each speaker, synced to two-channel audio. Faces are mosaicked in the sample to protect the speakers' privacy. The licensed data has no mosaic.
| |||||
| English | 2-5 | On request | |||
English conversation recorded in person, with two to five people. Sessions are casual or deliberative.
| |||||
| English | 3 | On request | |||
English conversation among three people, with a separate recording for each speaker.
| |||||
| Korean | 2 | On request | |||
Korean free conversation on a given topic, with one speaker on each channel.
| |||||
| Japanese | 2 | On request | |||
Japanese free conversation on a given topic, with one speaker on each channel.
| |||||
| Task-oriented | |||||
| English | 2 | CC BY 4.0 | |||
58 sessions across seven collaborative tasks. What each speaker saw and did is recorded on the same timeline as the audio.
| |||||
| English | 2 | CC BY 4.0 | |||
47 sessions across eleven collaborative tasks, recorded after the 20h release. None of its sessions repeat here.
| |||||
| English | 2-4 | On request | |||
English conversation with first-person video, as two to four people fold towels, tidy everyday objects, or play Jenga.
| |||||
| Role-play | |||||
| Japanese | 2 | On request | |||
Japanese two-speaker role-play in set scenarios, from shop visits to business talks. The speakers use their own words.
| |||||
| Scripted | |||||
| Japanese | 2 | On request | |||
Japanese two-speaker conversation read from scripts, with each speaker on their own channel. Every recording comes with its script and a timed transcript.
|
Looking for a specific kind of data? See English, Japanese, Korean, video, multi-speaker, and egocentric conversation.
What we offer
Datasets
Full-duplex conversation in English, Japanese, and Korean, licensed from our catalog or recorded to your specification.
Ask for a datasetJoint research
We work with speech labs from data design to publication, as we did with Sesame on TurnBench.
Ask about joint researchBenchmark improvement
We find where your model falls short on benchmarks such as TurnBench and supply the data to improve it.
Ask about benchmarks
Research
FAQ
What is full-duplex speech data?
Audio of a conversation recorded with a separate channel for each speaker, at the source. Both people can talk at once, and a single mixed track would flatten the overlap, backchannels, and interruptions that separate channels keep. fullduplex.ai, a site oto runs, explains how full-duplex voice AI is built and measured.
How much full-duplex conversation data has oto released?
Five otoSpeech datasets on Hugging Face: 280 hours for pretraining, 141 hours of cleaned US-native speech, the 104-hour hand-annotated training set for Sesame’s TurnBench, and task-oriented sets of 20 and 17 hours with interaction logs. The 141-hour set draws partly on the 280-hour set, so the hours do not add up to one total.
How is full-duplex data different from diarized audio?
Diarization estimates who spoke when from a single mixed recording, and it struggles most during overlapping speech. With channel-separated recording, each speaker’s audio is on its own track from the start, so nothing needs estimating.
Can I use otoSpeech commercially?
It depends on the dataset. The 280-hour, 141-hour, and task-oriented releases are CC BY 4.0, and downloading them requires an access request on Hugging Face. The TurnBench training set is under the oto Speech License for non-commercial research. For commercial licensing, write to agent@oto.earth.
Does oto collect custom speech datasets?
Yes. We record and annotate to specification: languages, task designs, turn-taking labels, multi-party, in-person, and video-grounded setups. Tell us what your model needs.
Backed by
- Berkeley SkyDeckIPP cohort
- NVIDIAInception Program
- DelightXAccelerator
- StoryFoundation