espnet/frontia_vad_v1_offline: voice activity detection, offline

One of two ESPnet voice activity detectors trained together:

	model	reads ahead	for
this one (offline)	dual-mode TCN, run at its largest look-ahead	up to 124 frames (0.99 s) of future audio	whole recordings, cleaning a corpus

Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop) gets the probability that someone is speaking. The model was trained as one-speaker diarization (diar1, num_spk 1): every speaker of a recording merged into one "speech" label.

Use

from espnet2.bin.diar_inference import VoiceActivityDetector

vad = VoiceActivityDetector.for_mode("offline", "espnet/frontia_vad_v1_offline")
for seg in vad.detect_file("recording.wav"):
    print(f"{seg.start:.2f}\t{seg.end:.2f}")

From a terminal, espnet detect recording.wav (offline model) or espnet detect --live (online model, from the microphone); from an agent, the MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained loads it too, as a num_spk=1 diarization model.

Decision rule

A frame is speech when its probability is above 0.8, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 124 (the model's max_look_ahead).

Latency: a frame's probability reads 124 frames (0.99 s) of future audio, and the median filter 5 frames (40 ms) more, so this model is for whole recordings rather than live streams.

Results

Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.

set	speech h	DetER (tuned)	FA	Miss	DetER (set's own best)	its threshold, median
aishell4_eval	11.5	7.51	2.11	5.40	5.70	0.5, 11
alimeeting_dev	3.9	6.26	4.00	2.26	6.21	0.7, 11
alimeeting_eval	9.9	5.77	3.37	2.40	5.77	0.8, 11
ami_dev	75.1	8.48	3.39	5.09	7.75	0.6, 11
ami_eval	90.1	8.63	1.70	6.93	6.73	0.6, 11
ami_mh_dev	7.6	8.89	5.38	3.51	8.89	0.8, 11
ami_mh_eval	7.3	9.75	4.10	5.65	9.33	0.7, 11
ava_speech_eval	19.9	32.39	22.59	9.80	28.47	0.9, 11
chime6_dev	117.9	9.75	1.58	8.18	4.25	0.3, 11
chime6_eval	122.7	13.92	3.57	10.35	12.54	0.6, 11
dev	329.2	7.12	1.77	5.35	5.27	0.5, 11
dipco_dev	98.1	11.67	2.64	9.04	7.67	0.5, 11
dipco_eval	92.0	9.96	3.56	6.41	8.53	0.6, 11
earnings21_eval	32.8	9.50	6.94	2.56	9.50	0.8, 11
icsi_dev	12.5	11.03	2.83	8.20	10.35	0.7, 11
icsi_eval	11.9	15.16	2.25	12.91	12.67	0.5, 11
notsofar_dev	123.7	3.39	0.87	2.52	2.89	0.7, 11
notsofar_eval	398.5	3.24	1.58	1.65	3.15	0.7, 11
ten_vad_eval	0.1	19.52	17.53	1.98	16.97	0.9, 11
voxconverse_dev	18.9	4.54	2.58	1.96	4.48	0.7, 11
voxconverse_eval	39.0	7.42	5.72	1.70	7.42	0.8, 11

Hour-weighted DetER: 7.61. Macro average over the 21 sets: 10.19.

Training

Recipe: egs2/frontia_vad_v1/diar1 in ESPnet. The config is in this repository. Weights: the average of the two checkpoints with the best validation accuracy, epochs 11 and 17.

training data	licence
AMI (every microphone)	CC BY 4.0
CHiME-6, OpenSLR SLR150 (every array and binaural channel)	CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0)
NOTSOFAR-1 240825.1	CC BY 4.0
ICSI	CC BY 4.0
noise: MUSAN noise/ and music/; CHiME-3 background	CC / public domain; CC BY 4.0

The other sets in the table above are used for evaluation only.