pipecat-ai/backchannel-detector

Pipecat backchannel detector

A streaming audio model for voice agents. While the bot is talking, it decides whether the user is backchannelling ("mm-hmm", "right"), so the bot keeps talking, or taking the floor ("wait", "no"), so the bot stops. It runs ahead of the transcript.

734k parameters (causal conv encoder → GRU) and 40 ms chunks, using about 0.3 ms of CPU per chunk. Code: pipecat-ai/pipecat-vap.

Files

File
backchannel-fp32.onnx	streaming model (default)
backchannel-int8.onnx	int8 quantized
model.safetensors	PyTorch weights for bcdet.model.net.BackchannelNet
model.json	config, chunk size, state tensor names

Inputs and outputs

One call per 40 ms chunk. Outputs come per 20 ms frame (2 per chunk).

Input	Shape
chunk	[1, 640]	16 kHz mono float32 in [-1, 1], user mic after echo cancellation
bot_state	[1, 2, 8]	per frame: bot speaking (0/1); seconds since bot onset, since the bot's last ≥200 ms pause, and since bot stop (each clipped to 5 s, ÷5); filler playing (0/1); 3 zeros
*_cache, gru_state	see model.json	start at zeros; feed each *_out back

Output	Shape
intent	[1, 2, 5]	softmax over continuer, assessment, negative, floor_take, non_speech
vad	[1, 2]	user voice activity (auxiliary)
vap	[1, 2, 8]	user then bot activity at 0–0.2, 0.2–0.6, 0.6–1.2 and 1.2–2.0 s ahead (auxiliary, after VAP)

p(backchannel) = continuer + assessment. p(floor) = negative + floor_take.

Results

Held-out conversations, replayed as a Pipecat agent sees them (200 ms VAD lag included). Floor threshold 0.7; a backchannel is called only once the user stops (threshold 0.7, 1.2 s backstop).

	Open Yap (60 conv.)	otoSpeech (40 conv.)
interruptions caught within 700 ms	82.9 %	81.2 %
interruptions lost as backchannels	2.6 %	4.1 %
backchannels the bot talks through	86.5 %	75.2 %

Median time to catch an interruption: 280 / 300 ms after the user starts speaking.

Limitations

• English only.
• Trained and tested on human-to-human conversation, not on people talking to a bot.
• The training echo is the partner's channel mixed back in, not real echo cancellation residue. Band-limiting is simulated; real phone codecs are untested.

Training data

Data	Use	Licence
Open Yap 1K v1.0, full corpus (1,516 conv.)	training, validation	Open Yap 1K Data Use Agreement v1
Open Yap 1K sample	background speech for the synthetic negatives	CC BY 4.0
otoSpeech-full-duplex-280h (1,081 conv.)	training, validation	CC BY 4.0
Synthetic hard negatives (600 windows)	training	made for this project

No training data or labels are included here.

Models used to build the training data

Model	Role
Silero VAD 6.2.3 (MIT)	speech segments on every channel
Deepgram Nova-3	word timings (shipped with Open Yap; made per channel for otoSpeech)
Deepgram Aura-2	12 TTS voices for the synthetic hard negatives

Attribution

• "Open Yap 1K sample" by The Agentic Data Company, https://huggingface.co/datasets/TheAgenticDataCompany/open-yap-1k, CC BY 4.0. Changes: resampled to 16 kHz, segmented with Silero VAD and relabelled; 12 s windows used as background speech under TTS clips.
• "otoSpeech-full-duplex-280h" by otoearth (oto), https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-280h, CC BY 4.0. Changes: resampled to 16 kHz, transcribed per channel with Deepgram Nova-3, segmented with Silero VAD and labelled from the words.

@misc{openyap1k,
  title     = {Open Yap 1K: Channel-Separated English Natural Two-Speaker Conversations},
  author    = {The Agentic Data Company},
  year      = {2026},
  version   = {1.0},
  publisher = {The Agentic Data Company},
  url       = {https://theagenticdatacompany.com/open-yap-1k},
}

@misc{otoSpeech-full-duplex-280h,
  title        = {otoSpeech-full-duplex-280h: Full-Duplex Conversational Speech Dataset},
  author       = {otoearth},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-280h}},
  note         = {License: CC BY 4.0}
}

@misc{silero-vad,
  author       = {Silero Team},
  title        = {Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier},
  year         = {2024},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/snakers4/silero-vad}},
  note         = {Version 6.2.3}
}

@inproceedings{ekstedt2022vap,
  title     = {Voice Activity Projection: Self-supervised Learning of Turn-taking Events},
  author    = {Ekstedt, Erik and Skantze, Gabriel},
  booktitle = {Interspeech},
  year      = {2022},
  url       = {https://arxiv.org/abs/2205.09812}
}