Dialogue-to-Duplex

Paper · Demo · GitHub · Quick Start · Research Results

Dialogue-to-Duplex (D2) turns pretrained speech models into full-duplex spoken language models that listen and speak at the same time. A distilled causal audio encoder and dialogue fine-tuning teach the model when to respond, continue listening, and yield to interruptions, while keeping its native speech generator. D2 achieves strong performance on VoiceBench and Full-Duplex-Bench.

Checkpoints

All nine main checkpoint variants from Tables 1 and 2 of the paper are available. Each release contains the benchmark model at update 10,000 and its matching streaming encoder.

Model	Chunk durations (ms)	Hugging Face revisions
Qwen3-Omni-D2	80, 160, 320, 640, 1040	qwen-80ms, qwen-160ms, qwen-320ms, qwen-640ms, qwen-1040ms
LLaMA-Omni 2-D2	100, 200, 400, 800	llama-100ms, llama-200ms, llama-400ms, llama-800ms

main remains the Qwen 80-ms default. Each named revision contains one variant, so downloading a revision does not download the other models. Use --revision to select a chunk duration, or pass a commit SHA to pin the exact release. The interval is the audio processing chunk duration; response latency also depends on generation and computation.

For Qwen, after the installation below:

d2-qwen infer --revision qwen-320ms --input question.wav --output response.wav --tail-seconds 20

For LLaMA-Omni 2, install its dependencies in a separate Python 3.12 environment:

python3.12 -m venv .venv-llama
source .venv-llama/bin/activate
python -m pip install --upgrade pip
python -m pip install -c llama-omni2-D2/configs/requirements.txt -e '.[llama]'
d2-llama infer --model penguinfish1688/dialogue-to-duplex --revision llama-100ms --input question.wav --output response.wav --tail-seconds 20

Run these installation commands from the cloned code repository. Use the same --model and --revision options with app for the browser interface or download to fetch weights in advance.

The default 80-ms checkpoint builds on Qwen3-Omni-30B-A3B-Instruct. It processes incoming audio and its own speech on an 80 ms clock and generates both text and speech. The interaction interval is the audio processing step; response latency also depends on generation and computation.

Quick Start

Use Linux, Python 3.12, and an NVIDIA GPU with driver 580 or newer. The pinned PyTorch build uses CUDA 13; see NVIDIA's driver requirements. We recommend an RTX PRO 6000 Blackwell (96 GB) and 96 GB of system RAM. Reserve at least 100 GB of free disk space: approximately 73 GB for model weights, plus the Python environment, downloads, and compiler caches.

Install git, ffmpeg, libsndfile, a C++ compiler, and Python 3.12 with development headers and venv support. On Ubuntu 24.04:

sudo apt-get update
sudo apt-get install -y git ffmpeg libsndfile1 build-essential python3.12-dev python3.12-venv

Then install D2 in its own environment:

git clone https://github.com/penguinfish1688/D2-dialogue-to-duplex.git
cd D2-dialogue-to-duplex
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -c qwen3-omni-D2/configs/requirements.txt -e '.[qwen]'

Browser Demo

d2-qwen app

The D2 loader downloads this checkpoint and its Qwen backbone automatically; no login is required. If your home directory has limited space, set HF_HOME to a larger disk before launching. The first startup compiles kernels and can take several minutes. Once Uvicorn running appears, open http://localhost:8000, use headphones, click Start conversation, and allow microphone access. Wait for Listening, then speak.

In a new terminal, run source .venv/bin/activate from the repository directory before starting the app again.

For a remote GPU, run this on your computer before opening the same URL:

ssh -L 8000:localhost:8000 user@gpu-host

Audio File Inference

d2-qwen infer --input question.wav --output response.wav --tail-seconds 20

The command saves speech to response.wav and a transcript and event trace to response.json. Both the app and file inference use BF16, a fixed KV cache, and CUDA graphs for prefill and serial decoding. The default context supports about 54 seconds; increase --kv-budget for longer conversations.

See the GitHub documentation for fine-tuning, runtime options, and evaluation commands. To measure real-time performance on your GPU, run the app performance check.

Research Results

The following figures and table summarize D2 research results across models and interaction intervals. The nine D2 variants in the tables are available through the revisions listed above.

VoiceBench

The Qwen3-Omni 80 ms model achieves an overall VoiceBench score of 70.07. The horizontal axis shows interaction intervals.

Full-Duplex-Bench

Model	D2 interval (ms)	Quality / 5 ↑	Turn taking (%) ↑	Turn latency (s) ↓	Interruption (%) ↑	Interruption latency (s) ↓	Pause success (%) ↑
Qwen3-Omni-D2	80	4.56	99.2	0.521	99.5	0.403	21.3
Qwen3-Omni-D2	160	4.67	100	0.539	99.5	0.432	19.0
Qwen3-Omni-D2	320	4.61	100	0.710	99.5	0.571	25.5
Qwen3-Omni-D2	640	4.62	99.2	1.01	99.5	0.928	55.1
Qwen3-Omni-D2	1040	4.84	100	1.38	99.5	1.30	61.6
LLaMA-Omni2-D2	100	3.33	100	0.746	99.5	0.621	74.5
LLaMA-Omni2-D2	200	3.62	99.2	0.825	100	0.710	78.2
LLaMA-Omni2-D2	400	3.69	100	1.02	99.0	0.894	69.4
LLaMA-Omni2-D2	800	4.03	98.3	1.45	99.0	1.32	70.8
BayLing-Duplex	—	3.55	85.7	5.57	100	5.19	72.2
Freeze-Omni	—	3.82	34.5	0.288	98.5	0.349	69.4
MiniCPM-o 4.5	—	4.45	75.6	1.92	94.5	1.89	91.2
PersonaPlex	—	4.43	99.2	0.346	95.0	0.294	31.9
Moshi	—	3.40	100	0.615	89.5	1.23	0.0
Nemotron VoiceChat	—	3.97	91.6	0.584	99.0	0.638	60.2

Quality uses GPT-4o ratings on a 0–5 scale. Turn taking and interruption measure successful responses to user events; pause handling measures whether the model waits through a user's pause. Latencies are event-aligned acoustic gaps plus the interaction interval, excluding computation time.

See the evaluation guide to run the benchmarks.

Base Models

Qwen variants use Qwen3-Omni-30B-A3B-Instruct. LLaMA-Omni 2 variants use ICTNLP/LLaMA-Omni2-7B-Bilingual, Whisper-large-v3, and ICTNLP/cosy2_decoder. Follow the respective upstream licenses and terms of use.