vrc-we-voice-ja

Japanese keyword-spotting model for vrc-WiredExpressions: wav2vec2 CTC over IPA phonemes, 20 ms frames, 16 transformer layers, int8 ONNX (254 MB).

Files

File	Content
model.onnx	Input input_values: float32 [batch, samples], 16 kHz mono, zero mean, unit variance. Output logits: [batch, frames, 29]. int8 dynamic quantization of the MatMuls.
enc.onnx	model.onnx after its convolutional feature encoder: input /wav2vec2/Transpose_output_0: float32 [batch, frames, 512], the output of fx.onnx in vrc-we-voice-en; output logits. enc.onnx(fx.onnx(x)) equals model.onnx(x) exactly.
vocab.json	Token ids; <pad> (0) is the CTC blank.
config.json	Wav2Vec2Config of the pruned network.
voice.json	Language, units, firing threshold (-0.65).

fx.onnx and enc.onnx are model.onnx cut in two at the end of the convolutional feature encoder, which fine-tuning left frozen, so every vrc-we-voice model shares it: a client listening in several languages runs it once per audio window instead of once per language. model.onnx stays for clients that load the whole network.

Training

Fine-tuned from facebook/wav2vec2-xlsr-53-espeak-cv-ft (Apache 2.0), pruned to its lower 16 of 24 layers, with CTC on IPA phonemes.

Dataset	Licence	Utterances
Common Voice 19.0	CC0 1.0	22,686
FLEURS, train split	CC BY 4.0	1,619

Phoneme labels: transcripts read by OpenJTalk (jpreprocess) with NAIST-jdic (BSD-3-Clause).

Noise: MUSAN (CC BY 4.0).

Evaluation

Local keyword-spotting tests, not a standard benchmark. Phrases are registered as text and spotted in continuous audio. Base: facebook/wav2vec2-xlsr-53-espeak-cv-ft. Both models run int8 at their own threshold (this model -0.65, the base -0.75).

Hits % / false fires per hour	wav2vec2-xlsr-53-espeak-cv-ft (base)	vrc-we-voice-ja
Sentences	65.7 / 0.25	88.9 / 0.25
Sentences, noisy	58.6 / 0.44	85.9 / 0.44
Crowded	65.7 / 0.25	88.9 / 0.25
Non-speech	— / 0.92	— / 0.03

• Sentences: one phrase per FLEURS test sentence; noisy adds noise at 10 dB SNR.
• Words: isolated Multilingual Spoken Words test clips, from speakers held out of training.
• Crowded: every sentence and word phrase registered at once.
• Non-speech: music (MUSAN) and vocal sounds (VocalSound), every phrase registered; the worst of five operating points.

Addendum: training log

Shared settings: batches of 40 s of audio, 2 accumulated per step; AdamW, 10% warm-up, linear decay; all layers and the head trained; utterances 0.5 to 15 s; MUSAN noise in half the utterances at 5 to 20 dB SNR, and 5% noise-only samples with an empty target.

A continuation was kept when its hit rate, averaged over five false-fire budgets (0.05 to 1 per phrase-hour) and the speech tests, rose by at least one point in total, no test fell by more than one, and non-speech false fires stayed at or under 0.1 per phrase-hour.

Stage	Outcome
Pruned to 16 layers, new phoneme head; 10,000 steps, 5e-5 (head 5e-4)	kept: this model (above its 5,000-step checkpoint)
Continuation, 6,000 steps, 1.5e-5	rejected: net −0.8

Licence

Apache 2.0, as the base models. Dataset licences are listed above.