otoSpeech Task: Task-oriented full-duplex dataset

The future of voice AI

Production voice AI today comes in two shapes, and they are optimized against opposite things.

Cascaded systems chain speech recognition, a language model, and speech synthesis. Because the middle of the pipeline is text, the task is inspectable: you can check the slot, validate the field, retry the tool call, and log what went wrong. This is what runs in booking, support, and data capture. The cost is conversational. A pipeline built on clean turns treats overlap, backchannels, and barge-in as edge cases rather than as the way talking works.

Full-duplex speech-to-speech models take the opposite trade. They listen and speak at the same time, so overlap and backchannels are native and the interaction feels much closer to talking to a person. The cost is that there is no text stage where the task can be checked, and task reliability trails.

Neither family intends to stay where it is. Cascaded systems keep buying back naturalness, and full-duplex models keep buying back task competence, because both are aimed at the same place: a system that talks like a person and still gets the order number right. Our expectation is that the two lines meet.

Training and evaluating anything in that corner takes data that is both things at once: conversation that behaves like conversation, attached to a task whose outcome can be checked. Most speech data gives you one or the other.

What is new about otoSpeech Task

otoSpeech Task is full-duplex, 48 kHz, task-oriented, released under CC BY 4.0, and shipped with synchronized interaction logs. No existing dataset we have been able to find combines all five.

Each property on its own has been done. The HCRC Map Task recorded two speakers on separate channels with a deliberate information gap in 1991.[5] The Spot the Difference corpus added video to a similar setup in 2018.[6] SpokenWOZ collected 249 hours of task-oriented human-to-human phone calls in 2023, two-track, one track per speaker.[7] Large channel-separated full-duplex conversational corpora exist as well, including our own.

The combination is what is missing: wideband channel-separated audio, the visual material each speaker was looking at, the task state as timestamped machine-readable events with per-event visibility, and reference answers, across several task types, under a permissive license. SpokenWOZ is the nearest neighbor, and it is 8 kHz telephone audio with no visual channel and no interface log. If prior work covers all of it and we have missed it, we would like to know.

So we recorded pairs of people working on tasks, and kept the task.

Each session has two speakers on separate channels, a goal they have to reach together, and an interface that records what each of them saw and did. Where the task has a defined answer, the answer ships with the recording. A conversational repair is therefore attached to the action that provoked it and the action that resolved it.

The seven tasks

The tasks differ along three properties, and the combination is what decides the kind of conversation each one produces.

• Visual grounding. Whether the task makes speakers turn what they can see into language: shape, color, position, appearance. Tasks marked no still run on a screen, but the material on it is text to be read rather than an image to be described.
• Reference answer. Whether the outcome can be checked against something fixed that ships with the data.
• Information asymmetry. Whether the two speakers have unequal access to the task material or the interface state. This is about access, not about differences of opinion or background knowledge.

Where there is a reference answer, the result can be scored and the conversation that produced it can be read against that score. Where there is none, the conversation is the object of study rather than the outcome.

Spot the Difference

Each speaker sees a version of the same scene with a set of curated changes, and neither can see the other’s screen. Speech is the only bridge between two visual worlds. Speakers name objects, describe where they are, and confirm that they are both looking at the same part of the picture before they can decide whether anything differs.

Play a Spot the Difference session

Ten scene pairs are included. Each pair has a ground-truth list of its differences, with normalized positions and readable labels, so a marked difference can be checked against the intended one. The interaction log records markers added and removed, scene changes, and the moment a pair reveals the comparison.

The design follows the DiapixUK paradigm published by Rachel Baker and Valerie Hazan in 2011, which used picture pairs to elicit spontaneous interactive speech and worked to balance difficulty across pairs and contributions across speakers.[4] The images here are our own.

Photo Talk

Both speakers see the same photograph and talk about it with no goal to reach. They describe what is there, guess at what is happening, and connect it to their own experience.

Play a Photo Talk session

It is the control condition for the rest of the set: same recording setup, same pairs of people, same shared visual material, but nothing hidden and nothing to complete. Sessions are marked partial for that reason, because the task has no terminal state to reach. The log records slide exposure, navigation, and zoom, so it stays possible to tell which image a stretch of talk belongs to.

Half the 164 photographs are ours and ship with the release. The other half come from YFCC100M and are indexed rather than redistributed. resources/photo-talk/<version>/yfcc-index.json lists all 82 by Flickr id, original URL, licence, licence URL, and attribution, so each one can be fetched from the source under its own terms. The same fields appear on the slide in the interaction log, which is how a session ties a stretch of talk to the picture it was about.

Describe-and-Draw Portrait

One speaker sees a face; the other draws it without seeing it. There is no correct drawing, so the interest is in how facial features, proportions, and relative positions get turned into language, and how the drawer’s questions reshape the description.

Play a Describe-and-Draw Portrait session

This is the only task with saved artifacts. Twenty-six drawings ship with the release, each tied to the round and the speaker who made it, so a finished drawing can be read against the talk that produced it.

Tangram Direction

The Director sees a target arrangement the Matcher cannot. The pieces are abstract, so they have no obvious names. The pair has to invent descriptions, then repair them when they fail.

Play a Tangram Direction session

This is the task Herbert H. Clark and Deanna Wilkes-Gibbs used in 1986 to show that referring is collaborative: a speaker proposes an expression, the listener accepts, refuses, or amends it, and the two converge on something that works for them rather than something correct in the abstract.[3] The target arrangement is known, and every piece selection, move, rotation, and flip is logged with a timestamp, so an instruction can be traced to the action it caused and a repair to the correction it produced.

Consensus Ranking

A pair ranks ten items by importance and then compares their order against a fixed reference ranking. Both speakers see the same material. Producing one order still requires explaining reasons, meeting objections, and deciding whose judgment wins.

Play a Consensus Ranking session

Ten scenarios are used across the sessions, not one. Survival on the Moon is the variant with published provenance: the task and an expert criterion supplied by NASA appear in the appendix to Hall and Watson’s 1970 study, where the criterion consolidated judgments from five staff at NASA’s Manned Spacecraft Center.[1] A later NASA education module published the exercise with rankings and reasons from two NASA scientists.[2] A Lost at Sea variant is also included. The remaining eight are curated facilitation scenarios, each shipping with its own fixed reference ranking and a stated rationale per item.

Correct here means the fixed key that ships with the data, not a settled physical fact. The distance between a pair’s ranking and that key can be measured separately from how much they said to get there.

Hiring Decision

Two speakers review fictional candidate profiles against a brief and pick one. The candidates were built so that strengths and concerns are genuinely balanced, and there is nothing to check the answer against.

Play a Hiring Decision session

It is the counterpart to Consensus Ranking: the same symmetric, deliberative shape, with the reference ranking removed. What the sessions show is how two people share subjective reasons and still arrive at a single decision.

Voice-to-Form

A synthetic document is separated from the form it has to be typed into. One speaker reads; the other types what they hear. The documents contain product codes, names, email addresses, postal addresses, prices, and drug names, all synthetic, and the completed form has field-level ground truth. Of the seven, this is the task aimed most directly at precise transfer of character strings, spelling out, and correction.

Play a Voice-to-Form session

This release contains one session of it: 24 minutes and 55 seconds, marked partial, with 104 logged events. That is enough to show the design and the data format. It is not enough to characterize how people spell things out to each other, and we would not use it to make a claim about that. We are collecting more.

One timeline

Every event carries a timestamp in seconds from the start of the recording, on the same clock as the audio. Events are typed by whether they are a lifecycle step, an exposure, an interaction, a state change, or an outcome. Each one names the speaker and channel that produced it, and who could see it.

That last field does real work. An event marked visible to the actor alone, rather than shared, is what lets you reconstruct what each speaker knew at a given second, and that is what makes an asymmetric task interpretable at all.

It also opens up questions a transcript cannot answer. Which piece moved after that instruction? Which field changed after the speaker repeated themselves? Which description was the one that got both people looking at the same object?

You can page through all 58 sessions in the browser, with the audio, the task state as it changes, and the event stream side by side.

What is in the release

• 20.005954 hours across 58 sessions and seven tasks
• 48 kHz channel-separated stereo FLAC, one channel per speaker, distributed as WebDataset shards
• A session JSON per recording: speakers, task, completion status, rounds, redaction spans, and the full event stream
• 11,025 events, 6,115 of them primary, timestamped against the recording
• Task stimuli under resources/, versioned, so task state can be reproduced
• Reference answers where the task has one
• 26 participant drawings from Describe-and-Draw Portrait
• JSON Schemas for sessions, events, and artifacts
• export-manifest.json with release totals and per-task coverage
• Language: English
• License: CC BY 4.0

Coverage is not spread evenly across the seven tasks.

This release has no transcripts. It is audio, interaction, task state, and reference answers. If you need text, you will need to run your own recognition over the channels, and the channel separation makes that considerably easier than it would be on a mixed recording.

Access is by request. The Hugging Face repository is gated, and approval is manual.

Limits

The recordings come from designed task settings. They are spontaneous and interactive, and they are not live customer support traffic or unconstrained daily conversation. Anyone evaluating transfer to a production domain should treat that gap as real.

The distribution across tasks is uneven, as the table above shows. Spot the Difference and Photo Talk carry most of the audio, and Voice-to-Form is one session. Per-task conclusions should be weighted accordingly.

Completion status is not a quality signal. Open-ended tasks are marked partial because they have no terminal state, not because something went wrong.

Speaker profile fields are opt-in and may be empty, name is always empty by construction, and speaker_id values are anonymized stable identifiers. We ran automated PII detection, and each session marks the spans that were redacted.

Everything here is English, and it is one recording setup. This is a 20-hour release, not a corpus that will settle a question on its own.