Real vs. Synthetic Dialogue for Speech-to-Speech Training

What counts as synthetic dialogue

The word covers several different things, and they fail in different ways:

• TTS rendering of text dialogue. An LLM or a text corpus supplies the words, a TTS model voices each side, and a script places the turns on a timeline. OmniFlatten, SyncLLM, PersonaPlex, and most synthetic datasets work this way.
• Multi-stream TTS trained on real conversation. Moshi's TTS generates both speakers at once and was itself trained on real two-channel recordings, so some real timing carries into its output.
• Scripted human performance. People read or act out an LLM-written script. HumDial's evaluation data was made this way.
• Simulated mixtures of real speech. Real single-speaker segments are mixed into conversations by a simulator, as in the Mind the Gap study.
• Mono audio split into channels. Diarization assigns a mixed recording to two streams. Moshi and J-Moshi used this for an earlier training stage, and its overlaps are whatever the separation produced.

The data ceiling essay on fullduplex.ai and the training data guide explain why the field ended up here.

What published systems did

Several synthetic datasets are now public or announced:

What synthetic data did well

Reported gains:

• Content and instruction following. SyncLLM, trained mostly on synthetic dialogue, scored higher than dGSLM on meaningfulness while keeping turn-taking naturalness at parity. In J-Moshi, adding 602 h of synthetic dialogue raised meaningfulness from 2.19 to 2.30 on a five-point scale.
• Roles and voices. PersonaPlex reports that adding synthetic data greatly improved voice cloning and role adherence over the Moshi baseline. Its synthetic dialogs use synthetic voices, which the authors chose for data privacy.
• Rare and exact events. Full-Duplex-Bench v1 generated its interruption tests with GPT-4o and ChatTTS because public datasets have few user interruptions (Lin et al., 2025). MultiPartyTurnBench uses 240 synthetic conversations so every turn end and interruption has an exact timestamp (paper).
• Simulation from real recordings. Mind the Gap simulated conversations from real speech, including 500 h built from otoSpeech. Synthetic-only training came close to real-data baselines for multi-talker ASR and diarization, and adding simulated data to real recordings beat real-only training on both tasks (Polok et al., 2026).

What synthetic data did not provide

The same papers report shortfalls, mostly in timing:

• Overlap. SyncLLM's synthetic stage treats dialogue as having no overlap: during one speaker's turn, the other is silent. The model learns turn-taking cues in a final stage on Fisher.
• Naturalness and overlap in Japanese. After J-Moshi added its 602 synthetic hours, naturalness did not move (2.67 before, 2.66 after). Overlap in its generated dialogue fell from 5.0 to 4.2 seconds per minute, against 8.1 in real Japanese dialogue.
• Backchannels. PersonaPlex's released checkpoint added 1,217 h of Fisher to improve natural backchanneling, expressions, and emotional responses, and the authors report a significant rise in backchannel frequency and better pause handling.
• Few full-duplex events. In DuplexDrama, 3.8% of all turns carry an interruption, backchannel, or incomplete-turn behavior.
• Simulated timing. OmniFlatten's simulator places turns in three situations: an immediate reply, a user interruption that stops the assistant, and silence while the assistant waits. The ICASSP 2026 HumDial organizers argue that artificially overlapped audio tracks miss hesitations and cooperative barge-ins, so they had professional actors perform LLM-written scripts instead (HumDial).
• Model-to-model timing. When two PersonaPlex instances talked to each other, the floor changed hands after a median 400 to 560 ms, against 137 ms between people in Switchboard, so self-play data inherits that late timing (Coupled but Late).

How teams mix them

In the recipes above:

• A real seed for the generator. Moshi trained its multi-stream TTS on 170 h of real two-channel conversation, then generated more than 20,000 h of instruction data from it.
• Real data where timing is learned. SyncLLM, Moshi, and PersonaPlex all put real two-channel conversation in the stage meant to teach turn-taking or naturalness, alongside or after the synthetic data.
• Real data for evaluation. MultiTalk trains on synthetic dialogue and evaluates on MultiTalkBench, built from real recordings. DuplexAct draws 2,531 of its 4,691 test items from real recordings (paper).

The real share varies widely. From the published hours, real dialogue was about 1% of SyncLLM's dialogue data, about 35% of PersonaPlex's hours once Fisher was added (1,217 of 3,467), and about 36% of J-Moshi-ext's fine-tuning data. Real data was the smaller share in all three. J-Moshi also matched language on both sides: its real hours were Japanese, and its synthetic hours came from Japanese text.

Before you add synthetic data

• Measure timing against real conversation. Compare overlap, pauses, gaps, and backchannel rate per minute in generated dialogue with a real reference in the same language, as J-Moshi did.
• Count the events. Ask what share of turns contain an interruption or backchannel, and whether a rule or a model placed them.
• Check the channel layout. The Korean preview ships 24 kHz mono files, with speaker turns given as timestamps in the annotation.
• Trace every license. Synthetic data inherits terms from the text source, the TTS model, and any voice-conditioning data. MultiTalk is non-commercial, and Behavior-SD is CC BY 4.0.
• Keep a real test set. Score turn-taking on real two-channel conversation, such as TurnBench, whatever the training mix.

Where oto fits

• The real side of the mix. oto builds two-channel conversation in English, Japanese, and Korean, with one speaker per channel. otoSpeech-full-duplex-280h and otoSpeech-full-duplex-processed-141h are open under CC BY 4.0, with access on request.
• A source for simulation. Mind the Gap used otoSpeech as real material for its synthetic multi-talker data, as listed on oto.earth/use-cases.
• Labeled real conversation for evaluation. otoSpeech-full-duplex-turn-104h carries 17 turn-taking labels per speaker.
• Licensing. Licensed datasets run from $10 to $100 per hour of audio, and the license and labels add to the price. Custom collection is quoted per project. Write to agent@oto.earth.

Frequently asked questions

Can you train a full-duplex model on synthetic data only?

OmniFlatten reached full-duplex behavior on 2,000 hours of synthetic dialogue, but Moshi, SyncLLM, PersonaPlex, and J-Moshi all added real two-channel conversation.

How much real conversation data do I need alongside synthetic data?

Published recipes range from about 1% real (SyncLLM, 2,000 of 214,000 hours) to about 35% (PersonaPlex's released model, 1,217 hours of Fisher).

Does synthetic dialogue teach backchannels?

Only where the generator puts them. PersonaPlex added Fisher to its released model to improve backchanneling and reports a significant rise in backchannel frequency.

What is multi-stream TTS?

A TTS model that generates both speakers of a conversation together; Moshi's was trained on 170 hours of real two-channel conversation and used to create over 20,000 hours of training dialogue.

Are there public synthetic full-duplex datasets?

Yes: Behavior-SD (2,164 hours, CC BY 4.0), MultiTalk (57.6k hours, non-commercial), a Korean synthetic preview, and a DuplexDrama subset announced for release.

Is synthetic data useful for multi-talker ASR and diarization?

Yes. Mind the Gap found synthetic-only training came close to real-data baselines, and combining simulated with real recordings beat real-only training.

For more datasets, real and synthetic, see the fullduplex.ai catalog.

Related guides

• How Much Full-Duplex Data Do You Need? What Published Systems Used
• Free Conversation, Role-Play, or Scripted: What Each Teaches a Speech Model
• End-of-Turn Detection: Benchmarks and Training Data