|
--- |
|
license: cc-by-nc-4.0 |
|
--- |
|
# lina-speech (beta) |
|
|
|
Exploring "linear attention" for text-to-speech. |
|
|
|
It predicts audio codec "à la" [MusicGen](https://arxiv.org/abs/2306.05284) : delayed residual vector quantizers so that we do not need multiple models. |
|
|
|
Featuring [RWKV](https://github.com/BlinkDL/RWKV-LM), [Mamba](https://github.com/state-spaces/mamba), [Gated Linear Attention](https://github.com/sustcsonglin/flash-linear-attention). |
|
|
|
Compared to other LM TTS model : |
|
- Can be easily pretrained and finetuned on midrange GPUs. |
|
- Tiny memory footprint. |
|
- Trained on long context (up to 2000 tokens : ~27s). |
|
|
|
### Models |
|
|
|
| Model | #Params | Dataset | Checkpoint | Steps | Note | |
|
| :---: | :---: |:---: |:---: |:---: |:---: | |
|
| GLA | 60M, 130M | Librilight-medium | [Download](https://nubo.ircam.fr/index.php/s/wjNYLb54m7L8xf9) | 300k | GPU inference only | |
|
| Mamba| 60M | Librilight-medium |[Download](https://nubo.ircam.fr/index.php/s/wjNYLb54m7L8xf9)| 300k | GPU inference only | |
|
| RWKV v6 | 60M | LibriTTS |[Download](https://nubo.ircam.fr/index.php/s/wjNYLb54m7L8xf9) | 150k | GPU inference only | |
|
|
|
### Installation |
|
Following the linear complexity LM you choose, follow respective instructions first: |
|
- For Mamba check the [official repo](https://github.com/state-spaces/mamba). |
|
- For GLA/RWKV inference check [flash-linear-attention](https://github.com/sustcsonglin/flash-linear-attention). |
|
- For RWKV training check [RWKV-LM](https://github.com/BlinkDL/RWKV-LM) |
|
|
|
### Acknowledgment |
|
|
|
- The RWKV authors and the community around for carrying high-level truly opensource research. |
|
- @SmerkyG for making my life easy at testing cutting edge language model. |
|
- @lucidrains for its huge codebase. |
|
- @sustcsonglin who made [GLA and FLA](https://github.com/sustcsonglin/flash-linear-attention). |
|
- @harrisonvanderbyl fixing RWKV inference. |
|
|
|
### Cite |
|
```bib |
|
@software{lemerle2024linaspeech, |
|
title = {LinaSpeech: Exploring "linear attention" for text-to-speech.}, |
|
author = {Lemerle, Théodor}, |
|
url = {https://github.com/theodorblackbird/lina-speech}, |
|
month = april, |
|
year = {2024} |
|
} |
|
``` |
|
### IRCAM |
|
|
|
This work takes place at IRCAM, and is part of the following project : |
|
[ANR Exovoices](https://anr.fr/Projet-ANR-21-CE23-0040) |
|
|
|
<img align="left" width="200" height="200" src="logo_ircam.jpeg"> |
