| --- |
| tags: |
| - espnet |
| - audio |
| - automatic-speech-recognition |
| - speech-translation |
| language: multilingual |
| datasets: |
| - owsm_v3.1 |
| license: cc-by-4.0 |
| metrics: |
| - wer |
| - bleu |
| --- |
| |
| ## OWLS: Open Whisper-style Large-scale neural model Suite |
|
|
| [Paper](https://arxiv.org/abs/2502.10373) |
|
|
| OWLS is a suite of Whisper-style models, designed to help researchers understand the scaling properties of speech models. |
| OWLS models range from 0.25B to 18B parameters, and are trained on up to 360K hours of data. |
|
|
| OWLS models are developed using [ESPnet](https://github.com/espnet/espnet), and support multilingual Speech Recognition and Translation. |
|
|
| It is part of the [OWSM](https://www.wavlab.org/activities/2024/owsm/) project, which aims to develop fully open speech foundation models using publicly available data and open-source toolkits. |
|
|
|
|
| The model in this repo has 4.66B parameters in total and is trained on 180k hours of public speech data. |
| Specifically, it supports the following speech-to-text tasks: |
| - Speech recognition |
| - Any-to-any-language speech translation |
| - Utterance-level alignment |
| - Long-form transcription |
| - Language identification |
|
|
| ## Use this model |
|
|
| You can use this model in your projects with the following code: |
|
|
| ```python |
| # make sure espnet is installed: pip install espnet |
| from espnet2.bin.s2t_inference import Speech2Text |
| |
| model = Speech2Text.from_pretrained( |
| "espnet/owls_4B_180K" |
| ) |
| |
| speech, rate = soundfile.read("speech.wav") |
| speech = librosa.resample(speech, orig_sr=rate, target_sr=16000) # make sure 16k sampling rate |
| text, *_ = model(speech)[0] |
| ``` |
|
|
| ## OWLS models |
| | Model Name | Checkpoint | Training Artifacts | |
| | ------------------ | ------- | --------------------------------------------------------------------------------------- | |
| | OWLS 0.25B 180K | https://huggingface.co/espnet/owls_025B_180K | TBA | |
| | OWLS 0.50B 180K | https://huggingface.co/espnet/owls_05B_180K | https://huggingface.co/espnet/owls_05B_180K_intermediates/tree/main | |
| | OWLS 1B 11K | TBA | TBA | |
| | OWLS 1B 22K | TBA | TBA | |
| | OWLS 1B 45K | TBA | TBA | |
| | OWLS 1B 90K | TBA | TBA | |
| | OWLS 1B 180K | https://huggingface.co/espnet/owls_1B_180K | TBA | |
| | OWLS 2B 180K | https://huggingface.co/espnet/owls_2B_180K | TBA | |
| | OWLS 4B 180K | https://huggingface.co/espnet/owls_4B_180K | https://huggingface.co/espnet/owls_4B_180K_intermediates | |
| | OWLS 9B 180K | https://huggingface.co/espnet/owls_9B_180K | https://huggingface.co/espnet/owls_9B_180K_intermediates | |
| | OWLS 18B 180K | https://huggingface.co/espnet/owls_18B_180K | TBA | |
| | OWLS 18B 360K | https://huggingface.co/espnet/owls_18B_360K | TBA | |
| |
| |
| ## Citations |
| |
| ``` |
| @article{chen2025owls, |
| title={OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models}, |
| author={Chen, William and Tian, Jinchuan and Peng, Yifan and Yan, Brian and Yang, Chao-Han Huck and Watanabe, Shinji}, |
| journal={arXiv preprint arXiv:2502.10373}, |
| year={2025} |
| } |
| |
| ``` |
