frankharrison/wavlm-base-plus-web

WavLM Base+ for the web, convolutions at full precision

An ONNX build of microsoft/wavlm-base-plus for transformers.js, quantised so that the convolutional feature encoder stays fp32 and only the transformer's matrix multiplications are 8-bit. It loads with dtype: 'q8':

import { AutoModel, AutoProcessor } from '@huggingface/transformers'
const processor = await AutoProcessor.from_pretrained('frankharrison/wavlm-base-plus-web')
const model = await AutoModel.from_pretrained('frankharrison/wavlm-base-plus-web', { dtype: 'q8', device: 'wasm' })

Why not the usual q8 file

The commonly used model_quantized.onnx dynamically quantises every convolution too, including the first, whose input is the raw waveform. Each input is then effectively 8-bit audio scaled to its own loudest sample, which erases quiet sounds next to loud ones — in speech and singing, the consonants. On short clips (0.3–0.45 s) of sung syllables, its embeddings were a median 0.953 cosine from fp32 (worst 0.385), and moving a clip by 1–2 ms changed 6.6% of nearest-template classifications (fp32: 0.2%).

This build is a median 0.99 cosine from fp32 and classified the same test set as fp32 did. It is 116 MB against 91 MB, and in onnxruntime-web's WASM backend it runs at about the same speed: 8-bit weights save download size there, not time.

What changed from the original

• Converted to ONNX by Xenova/wavlm-base-plus (onnx/model.onnx, revision 27b8ee283bf6c3a5ae77d7e90d8e40a6925288e2, sha256 07789ad4d9e01cba75fce7fd923e7a5e9a0aaf0f43df25ba6ee6d3d6d7adaa1c). The configs are copied unchanged from that revision.
• Re-quantised here with onnxruntime 1.19.2: quantize_dynamic(model.onnx, ..., op_types_to_quantize=['MatMul'], weight_type=QuantType.QUInt8, per_channel=False). Convolutions, including the positional convolution, are untouched. Output sha256 a976307be5c3e46ccc2f7e6925c46edfc6cd9c5e9e630c0f79bfae223849dd85.

Not affiliated with or endorsed by Microsoft or the WavLM authors.

Attribution and licence

WavLM is by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng and Furu Wei (Microsoft) — WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing — and released through microsoft/UniSpeech under the Creative Commons Attribution-ShareAlike 3.0 licence (text). This adaptation is distributed under the same licence.