|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
library_name: fairseq |
|
pipeline_tag: automatic-speech-recognition |
|
inference: false |
|
--- |
|
|
|
|
|
<br> |
|
<br> |
|
|
|
# ARMHuBERT Model Card |
|
|
|
This repo contains the models from our paper [**Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation**](https://arxiv.org/abs/2305.11685), INTERSPEECH 2023. |
|
|
|
|
|
## Model details |
|
|
|
**Model type:** |
|
ARMHuBERT is an open-source speech SSL model distilled from HuBERT-Base, by attention map reusing and masking distillation. |
|
We also provide the model checkpoints of MaskHuBERT (without attention map reusing) and ARMwavLM (wavLM-Base teacher). |
|
|
|
- Attention Map Reusing: Reuse previous layer's attention map to remove key & query parameters in Transformer. |
|
- Masking Distillation: Masking distillation treating masked frames and unmasked frames separately. |
|
|
|
**License:** |
|
Apache 2.0 License |
|
|
|
**Where to send questions or comments about the model:** |
|
https://github.com/sungnyun/ARMHuBERT/issues |
|
|
|
|
|
## Training dataset |
|
Pretraining data: [LibriSpeech](https://www.openslr.org/12) |
|
- ``[ModelName]-100h.ckpt``: train-clean-100 |
|
- ``[ModelName]-960h.ckpt``: train-clean-100 + train-clean-360 + train-other-500 |
|
|
|
|
|
<br> |
|
|
|
More detials are in our github, https://github.com/sungnyun/ARMHuBERT. |
