Duplicate from BSC-LT/matcha-tts-cat-multispeaker

Co-authored-by: Alex Peiró Lilja <AlexK-PL@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,209 @@
+---
+language:
+- ca
+licence:
+- apache-2.0
+tags:
+- matcha-tts
+- acoustic modelling
+- speech
+- multispeaker
+pipeline_tag: text-to-speech
+datasets:
+- projecte-aina/festcat_trimmed_denoised
+- projecte-aina/openslr-slr69-ca-trimmed-denoised
+---
+
+# Matcha-TTS Catalan Multispeaker
+
+## Table of Contents
+<details>
+<summary>Click to expand</summary>
+
+- [Model description](#model-description)
+- [Intended uses and limitations](#intended-uses-and-limitations)
+- [How to use](#how-to-use)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Citation](#citation)
+- [Additional information](#additional-information)
+
+</details>
+
+## Model Description
+
+**Matcha-TTS** is an encoder-decoder architecture designed for fast acoustic modelling in TTS. 
+The encoder part is based on a text encoder and a phoneme duration prediction that together predict averaged acoustic features.
+And the decoder has essentially a U-Net backbone inspired by [Grad-TTS](https://arxiv.org/pdf/2105.06337.pdf), which is based on the Transformer architecture. 
+In the latter, by replacing 2D CNNs by 1D CNNs, a large reduction in memory consumption and fast synthesis is achieved.
+
+**Matcha-TTS** is a non-autorregressive model trained with optimal-transport conditional flow matching (OT-CFM). 
+This yields an ODE-based decoder capable of generating high output quality in fewer synthesis steps than models trained using score matching.
+
+## Intended Uses and Limitations
+
+This model is intended to serve as an acoustic feature generator for multispeaker text-to-speech systems for the Catalan language. 
+It has been finetuned using a Catalan phonemizer, therefore if the model is used for other languages it may will not produce intelligible samples after mapping 
+its output into a speech waveform. 
+
+The quality of the samples can vary depending on the speaker. 
+This may be due to the sensitivity of the model in learning specific frequencies and also due to the quality of samples for each speaker.
+
+## How to Get Started with the Model
+
+### Installation
+
+This model has been trained using the espeak-ng open source text-to-speech software. 
+The espeak-ng containing the Catalan phonemizer can be found [here](https://github.com/projecte-aina/espeak-ng)
+
+Create a virtual environment:
+```bash
+python -m venv /path/to/venv
+```
+```bash
+source /path/to/venv/bin/activate
+```
+
+For training and inferencing with Catalan Matcha-TTS you need to compile the provided espeak-ng with the Catalan phonemizer:
+```bash
+git clone https://github.com/projecte-aina/espeak-ng.git
+
+export PYTHON=/path/to/env/<env_name>/bin/python
+cd /path/to/espeak-ng
+./autogen.sh
+./configure --prefix=/path/to/espeak-ng
+make
+make install
+
+pip cache purge
+pip install mecab-python3
+pip install unidic-lite
+```
+Clone the repository:
+
+```bash
+git clone -b dev-cat https://github.com/langtech-bsc/Matcha-TTS.git
+cd Matcha-TTS
+
+```
+Install the package from source:
+```bash
+pip install -e .
+
+```
+
+
+### For Inference
+
+#### PyTorch
+
+Speech end-to-end inference can be done together with **Catalan Matcha-TTS**. 
+Both models (Catalan Matcha-TTS and Vocos) are loaded remotely from the HF hub.  
+
+First, export the following environment variables to include the installed espeak-ng version:
+
+```bash
+export PYTHON=/path/to/your/venv/bin/python
+export ESPEAK_DATA_PATH=/path/to/espeak-ng/espeak-ng-data
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/espeak-ng/lib
+export PATH="/path/to/espeak-ng/bin:$PATH"
+
+```
+Then you can run the inference script:
+```bash
+cd Matcha-TTS
+python3 matcha_vocos_inference.py --output_path=/output/path --text_input="Bon dia Manel, avui anem a la muntanya."
+
+```
+You can also modify the length scale (speech rate) and the temperature of the generated sample:
+```bash
+python3 matcha_vocos_inference.py --output_path=/output/path --text_input="Bon dia Manel, avui anem a la muntanya." --length_scale=0.8 --temperature=0.7
+
+```
+
+#### ONNX
+
+We also release a ONNX version of the model
+
+### For Training
+
+The entire checkpoint is also released to continue training or finetuning. 
+See the [repo instructions](https://github.com/langtech-bsc/Matcha-TTS/tree/dev-cat)
+
+
+## Training Details
+
+### Training data
+
+The model was trained on 2 **Catalan** speech datasets
+
+| Dataset             | Language | Hours   | Num. Speakers   |
+|---------------------|----------|---------|-----------------|
+| [Festcat](https://huggingface.co/datasets/projecte-aina/festcat_trimmed_denoised)             | ca       | 22      | 11              |
+| [OpenSLR69](https://huggingface.co/datasets/projecte-aina/openslr-slr69-ca-trimmed-denoised)           | ca       | 5       | 36              |
+
+### Training procedure
+
+***Catalan Matcha-TTS*** was finetuned from the English multispeaker checkpoint, 
+which was trained with the [VCTK dataset](https://huggingface.co/datasets/vctk) and provided by the model authors.
+
+The embedding layer was initialized with the number of catalan speakers (47) and the original hyperparameters were kept.
+
+### Training Hyperparameters
+
+* batch size: 32 (x2 GPUs)
+* learning rate: 1e-4
+* number of speakers: 47
+* n_fft: 1024
+* n_feats: 80
+* sample_rate: 22050
+* hop_length: 256
+* win_length: 1024
+* f_min: 0
+* f_max: 8000
+* data_statistics:
+  * mel_mean: -6578195
+  * mel_std: 2.538758
+* number of samples: 13340
+
+## Evaluation
+
+Validation values obtained from tensorboard from epoch 2399*: 
+
+* val_dur_loss_epoch: 0.38
+* val_prior_loss_epoch: 0.97
+* val_diff_loss_epoch: 2.195
+
+(Note that the finetuning started from epoch 1864, as previous ones were trained with VCTK dataset)
+
+## Citation
+
+If this code contributes to your research, please cite the work:
+
+```
+@misc{mehta2024matchatts,
+      title={Matcha-TTS: A fast TTS architecture with conditional flow matching}, 
+      author={Shivam Mehta and Ruibo Tu and Jonas Beskow and Éva Székely and Gustav Eje Henter},
+      year={2024},
+      eprint={2309.03199},
+      archivePrefix={arXiv},
+      primaryClass={eess.AS}
+}
+```
+
+## Additional Information
+
+### Author
+The Language Technologies Unit from Barcelona Supercomputing Center.
+
+### Contact
+For further information, please send an email to <langtech@bsc.es>.
+
+### Copyright
+Copyright(c) 2023 by Language Technologies Unit, Barcelona Supercomputing Center.
+
+### License
+[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+### Funding
+This work has been promoted and financed by the Generalitat de Catalunya through the [Aina project](https://projecteaina.cat/).

checkpoint_epoch=2399.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:352de98d658b7096f2b270da79e398217045a566c33e0496be3df21efd217b55
+size 250638851

config.yaml

@@ -0,0 +1,43 @@
+cfm:
+  name: CFM
+  sigma_min: 0.0001
+  solver: euler
+data_statistics:
+  mel_mean: -6.578195
+  mel_std: 2.538758
+decoder:
+  act_fn: snakebeta
+  attention_head_dim: 64
+  channels:
+  - 256
+  - 256
+  dropout: 0.05
+  n_blocks: 1
+  num_heads: 2
+  num_mid_blocks: 2
+encoder:
+  duration_predictor_params:
+    filter_channels_dp: 256
+    kernel_size: 3
+    p_dropout: 0.1
+  encoder_params:
+    filter_channels: 768
+    filter_channels_dp: 256
+    kernel_size: 3
+    n_channels: 192
+    n_feats: 80
+    n_heads: 2
+    n_layers: 6
+    n_spks: 47
+    p_dropout: 0.1
+    prenet: true
+    spk_emb_dim: 64
+  encoder_type: RoPE Encoder
+n_feats: 80
+n_spks: 47
+n_vocab: 178
+optimizer: null
+out_size: null
+prior_loss: true
+scheduler: null
+spk_emb_dim: 64

matcha_multispeaker_cat_opset_15_10_steps_2399.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87faec2df4837126ca324a72b16b53cf378a230dc0dc86f1781431388e714a94
+size 86049399

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44b9640678f0d3be86a09484bbcf2cd55c9c4d2a92fc0eb3fb193ada6b5d01aa
+size 83535314