espnet/frontia_vad_v1_online: voice activity detection, online

One of two ESPnet voice activity detectors trained together:

	model	reads ahead	for
this one (online)	causal TCN	nothing: every frame is decided from the audio up to it	live streams and the microphone

Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop) gets the probability that someone is speaking. The model was trained as one-speaker diarization (diar1, num_spk 1): every speaker of a recording merged into one "speech" label.

Use

from espnet2.bin.diar_inference import VoiceActivityDetector

vad = VoiceActivityDetector.for_mode("online", "espnet/frontia_vad_v1_online")
for seg in vad.detect_file("recording.wav"):
    print(f"{seg.start:.2f}\t{seg.end:.2f}")

From a terminal, espnet detect recording.wav (offline model) or espnet detect --live (online model, from the microphone); from an agent, the MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained loads it too, as a num_spk=1 diarization model.

Decision rule

A frame is speech when its probability is above 0.7, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 0.

Latency: frame i covers samples [128 i, 128 i + 512), so its probability is ready 32 ms after the frame starts. The median filter then needs the frames after it: a median of 11 frames settles a decision 5 frames (40 ms) later. With median 1 the decision is final as soon as the probability is.

Results

Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.

set	speech h	DetER (tuned)	FA	Miss	DetER (set's own best)	its threshold, median
aishell4_eval	11.5	8.29	3.86	4.43	8.00	0.6, 11
alimeeting_dev	3.9	7.52	5.23	2.29	7.49	0.6, 11
alimeeting_eval	9.9	7.21	4.85	2.36	7.21	0.7, 11
ami_dev	75.1	10.17	4.78	5.39	9.79	0.6, 11
ami_eval	90.1	10.12	2.95	7.18	8.77	0.5, 11
ami_mh_dev	7.6	11.08	7.72	3.36	11.08	0.7, 11
ami_mh_eval	7.3	11.44	6.38	5.07	11.44	0.7, 11
ava_speech_eval	19.9	50.29	44.27	6.02	33.03	0.9, 11
chime6_dev	117.9	9.64	1.94	7.71	4.65	0.3, 11
chime6_eval	122.7	14.78	5.41	9.37	14.76	0.6, 11
dev	329.2	7.93	2.72	5.21	6.58	0.4, 11
dipco_dev	98.1	10.51	4.16	6.35	8.52	0.4, 11
dipco_eval	92.0	10.06	5.38	4.69	9.41	0.6, 11
earnings21_eval	32.8	14.87	10.51	4.36	14.87	0.7, 11
icsi_dev	12.5	11.96	5.42	6.55	11.96	0.7, 11
icsi_eval	11.9	14.71	3.92	10.79	13.60	0.5, 11
notsofar_dev	123.7	4.52	1.94	2.59	4.37	0.6, 11
notsofar_eval	398.5	4.48	2.87	1.62	4.48	0.7, 11
ten_vad_eval	0.1	22.97	21.46	1.51	16.92	0.9, 11
voxconverse_dev	18.9	4.94	3.89	1.05	4.94	0.7, 11
voxconverse_eval	39.0	8.26	7.32	0.94	8.08	0.8, 11

Hour-weighted DetER: 8.71. Macro average over the 21 sets: 12.18.

Training

Recipe: egs2/frontia_vad_v1/diar1 in ESPnet. The config is in this repository. Weights: the checkpoint of epoch 11.

training data	licence
AMI (every microphone)	CC BY 4.0
CHiME-6, OpenSLR SLR150 (every array and binaural channel)	CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0)
NOTSOFAR-1 240825.1	CC BY 4.0
ICSI	CC BY 4.0
noise: MUSAN noise/ and music/; CHiME-3 background	CC / public domain; CC BY 4.0

The other sets in the table above are used for evaluation only.