FD-VAD

FD-VAD is a compact, ASR-free and streaming semantic endpoint detector for full-duplex spoken interaction. It feeds short sliding windows of audio directly to a small language model and emits a per-chunk Continue / Stop decision, deciding while the user is still speaking whether the turn is semantically complete — using meaning, not just silence.

Model

• Audio encoder: frozen WavLM-base-plus (16 kHz, 50 Hz frame rate).
• Modality adapter: temporal downsampling by 4 (concatenate consecutive frames) + 2-layer MLP into the LLM embedding space.
• LLM backbone: Qwen2.5-0.5B-Instruct adapted with LoRA (rank 16). Encoder and LLM base weights are frozen; only the adapter and LoRA parameters are trained (~10 M params, ~1.7% of the total).
• Streaming: sliding window of 2.56 s (256 frames), stride 320 ms; each window is encoded independently (causal — no future audio leaks into a decision), so per-window compute is fixed regardless of utterance length (~46 ms/window on an A100).
• Objective: last-chunk cross-entropy over a Continue (0) / Stop (1) target.

Training data

English subset of pipecat-ai/smart-turn-data-v3.1 (BSD-2), a 16 kHz conversational endpointing corpus with native complete / incomplete labels. One epoch, AdamW, effective batch 64, bf16.

Results (English smart-turn-v3.1 test)

Metric	Value
Chunk-level F1 (Stop=positive)	0.83
Chunk false-interruption rate	0.6%
Per-utterance endpoint accuracy	0.965 (complete 0.968 / incomplete 0.962)

Statistically indistinguishable from a strong non-streaming turn classifier on the endpoint decision, while deciding incrementally. Generalizes from synthetic to unseen human speech with a ~1.1% chunk-accuracy drop, and transfers zero-shot to external end-of-turn benchmarks.

Files (self-contained, fully offline — ~1.5 GB)

• fd_vad.pt — trained adapter + LoRA weights (119 MB).
• base/wavlm-base-plus/ — the frozen WavLM encoder weights (MIT), bundled.
• base/qwen2.5-0.5b/ — the frozen Qwen2.5-0.5B-Instruct backbone + tokenizer (Apache-2.0), bundled (bf16).
• config.yaml — architecture / window configuration (points at the bundled base/ weights).
• fd_vad/ — the inference code (model, encoder wrapper, adapter, windowing).
• infer.py — ready-to-run end-of-turn inference on an audio file.
• requirements.txt — dependencies.

This repo is fully self-contained: everything needed to run end-of-turn inference — encoder, LLM backbone, and the trained adapter/LoRA — is included. No weights are downloaded at runtime (infer.py sets HF_HUB_OFFLINE=1 when the bundled base/ weights are present). The bundled base models are redistributed under their original permissive licenses (WavLM: MIT; Qwen2.5: Apache-2.0).

Usage

pip install -r requirements.txt
python infer.py path/to/audio.wav                 # prints the committed EOT time, or "incomplete"
python infer.py audio.wav --tau 0.9 --k 1         # more conservative endpoint
python infer.py audio.wav --show-stream           # per-320ms P(Stop) trace

Or in Python:

import torch, sys; sys.path.insert(0, ".")
from fd_vad.config import load_config
from fd_vad.models.fd_vad import FDVAD

cfg = load_config("config.yaml"); cfg.encoder.kind = "wavlm"
model = FDVAD(cfg).eval()                          # pulls WavLM + Qwen from the Hub
model.load_trainable_state_dict(torch.load("fd_vad.pt", map_location="cpu")["model"])
# slide a 2.56 s window (stride 320 ms) over 16 kHz mono audio;
# out = model(window[B,T], lengths[B]); out.probs2[:, 1] is P(Stop) per chunk. See infer.py.

Intended use & limitations

Intended as a portable turn-taking signal for full-duplex voice agents. English only (WavLM-base-plus + English training data). It is a semantic endpoint detector, not a general VAD; for continuous multi-turn audio, pair it with a lightweight acoustic VAD gate that decides when an endpoint is plausible while FD-VAD decides whether the preceding speech is complete.

License

MIT.