telenow/telenow-pvad-v1

pVAD v1 — speaker-conditioned frame VAD for telephony barge-in

A 171k-parameter streaming model that labels every 10 ms of an 8 kHz caller audio stream as silence, the enrolled caller, or someone else (a bystander, a TV, the agent's own echo). It is conditioned on a 192-d speaker embedding of the caller taken from the first seconds of the call, so a voice-agent's barge-in logic can ignore other voices in the room without waiting for a transcript.

Trained by Telenow AI on its own Indian PSTN call recordings (Hindi, English and other Indian languages) with synthetic bystander / TV / echo mixtures. No audio is released with this model.

Results (held-out recordings, synthetic mixtures, 1.5 s windows)

window	cosine speaker gate (baseline)	pVAD v1
caller alone → accepted as caller	22 %	96 %
caller + bystander overlapping → accepted	11 %	81 %
bystander only → accepted (false accept)	0 % (+19 % unsure)	6 %
window EER	0.22	0.091

Frame accuracy 0.87, speech-frame AUC 0.97, target recall 0.84 at 5 % false accept. One 10 ms frame costs ~0.08 ms single-threaded in onnxruntime.

Files

file	what
pvad_stream_v1.onnx	streaming graph — one 10 ms frame per call, carries its own state (use this in production)
pvad_full_v1.onnx	whole-sequence graph feats [1,T,40], emb [1,192], h0 [2,1,96] → logits [1,T,3] (offline scoring)
pvad_v1.pt	PyTorch state_dict of the model below
pvad_norm.json	feature mean / sd (already baked into both ONNX graphs; here for retraining)
config.json	the full I/O and feature contract
example_stream.py	reference streaming inference with onnxruntime
train_log.txt	the training run's validation curve

Model

fbank(40) → Linear(40→64) + ReLU
          → causal Conv1d(64, k=5)           + ReLU
          → causal Conv1d(64, k=5, dil=2)    + ReLU
          → FiLM(emb 192 → scale, shift)     (x · (1+γ) + β)
          → GRU(64→96, 2 layers)
          → Linear(96→3)   [silence, caller, other]

Normalisation (feats − mu) / sd with the stored global statistics is the first op inside both graphs.

Input features

Kaldi-style log-mel filterbank, 8 kHz, 40 bins, 25 ms Hamming window, 10 ms hop, no dither, DC removal, 0.97 pre-emphasis, 256-point FFT, energies floored at float epsilon then ln, no mean subtraction, on int16-scale samples (i.e. waveform × 32768). Equivalent to:

torchaudio.compliance.kaldi.fbank(x_int16_scale[None], num_mel_bins=40,
    frame_length=25, frame_shift=10, dither=0.0, energy_floor=0.0,
    sample_frequency=8000, window_type="hamming", use_energy=False, snip_edges=True)

Conditioning (enrollment) embedding

A 192-d, L2-normalised WeSpeaker ECAPA-TDNN512-LM embedding (voxceleb_ECAPA512_LM.onnx, CC-BY-4.0) of 1.5–3 s of the caller's clean speech: audio upsampled 8 → 16 kHz, 80-bin Kaldi fbank (25/10 ms, Hamming, no dither), per-utterance mean subtraction, feats [1,T,80] → embs [1,192]. The model was trained with enrollments of 1.5–3 s (first turn) and 3 s (later in the call), 50/50. Any other speaker encoder requires retraining.

Streaming contract (pvad_stream_v1.onnx)

tensor	shape	meaning
feats	[1,1,40]	the new 10 ms frame
hctx	[1,4,64]	last 4 post-ReLU input-hidden frames (zero-init)
cctx	[1,8,64]	last 8 conv-1 outputs (zero-init)
emb	[1,192]	enrollment embedding
h0	[2,1,96]	GRU state (zero-init)
→ logits	[1,1,3]	silence / caller / other
→ hn	[2,1,96]	next GRU state
→ h1	[1,1,64]	push into hctx (drop the oldest)
→ c1	[1,1,64]	push into cctx (drop the oldest)

The two rings reproduce the training graph's causal zero padding exactly: streaming output equals the whole-sequence graph to ~2e-6 from the first frame.

Intended use and limits

• Built for barge-in gating in a full-duplex voice agent on telephony audio: "is this speech the caller's?" before STT has returned. It says who is speaking, not whom they address.
• Evaluated on synthetic bystanders (other real callers, room-filtered and mixed at 3–22 dB below the caller, TV at 12–25 dB below, agent echo); real-room recordings were not part of the evaluation.
• Trained on 8 kHz Indian-language telephony; wideband or non-telephony audio is out of distribution.
• Needs an enrollment; until the caller has spoken ~1.5 s there is nothing to condition on — pass through until then.
• Two people legitimately sharing a call must be handled by policy on top (e.g. admit a non-caller voice after N speech runs inside a window).