VITS2 Text-to-Speech on Natasha Dataset

Model Details

Model Description

This model is an implementation of VITS2, a single-stage text-to-speech system, trained on the Natasha dataset for the Russian language. VITS2 improves upon the previous VITS model by addressing issues such as unnaturalness, computational efficiency, and dependence on phoneme conversion. The model leverages adversarial learning and architecture design for enhanced quality and efficiency.

Developed by: Jungil Kong, Jihoon Park, Beomjeong Kim, Jeongmin Kim, Dohee Kong, Sangjin Kim
Shared by: LangSwap.app
Model type: Text-to-Speech
Language(s) (NLP): Russian
License: MIT
Finetuned from model: No

Model Sources

Repository: VITS2 PyTorch Implementation
Paper: VITS2 paper

Usage

This model was dedicated to be used with this repository. https://github.com/shigabeev/vits2-inference

Sample usage:

git clone git@github.com:shigabeev/vits2-inference.git
cd vits2-inference
pip install -r requirements.txt
python infer_onnx.py --model natasha.onnx --text "Привет! Я Наташа!"

Direct Use

The model can be used to convert text into speech directly. Given a text input in Russian, it will produce a corresponding audio output.

Downstream Use

Potential downstream applications include voice assistants, audiobook generation, voiceovers for animations or videos, and any other application where text-to-speech conversion in Russian is required.

Out-of-Scope Use

The model is specifically trained for the Russian language and might not produce satisfactory results for other languages.

Bias, Risks, and Limitations

The performance and bias of the model can be influenced by the Natasha dataset it was trained on. If the dataset lacks diversity in terms of dialects, accents, or styles, the generated speech might also reflect these limitations.

Recommendations

Users should evaluate the model's performance in their specific application context and be aware of potential biases or limitations.

How to Get Started with the Model

To use the model, users can follow the guidelines and scripts provided in the VITS2 PyTorch Implementation repository.

Training Details

Training Data

The model was trained on the Natasha dataset, which is a collection of Russian speech recordings.

Training Procedure

Preprocessing

Text and audio preprocessing steps, as mentioned in the repository README, were followed.

Training Hyperparameters

Training regime: This can be filled with details such as learning rate, batch size, optimizer used, etc.

Summary

The VITS2 model demonstrates improved performance over previous TTS models, offering more natural and efficient speech synthesis.

Environmental Impact

You can fill in the details regarding the environmental impact, based on the compute resources used for training.

Technical Specifications

Model Architecture and Objective

The VITS2 architecture comprises of various improvements over the original VITS, including but not limited to speaker-conditioned text encoder, mel spectrogram posterior encoder, and transformer blocks in the normalizing flow.

Compute Infrastructure

Hardware

Single Nvidia RTX 4090

Software

Python >= 3.11
PyTorch version 2.0.0

APA:

Kong, J., Park, J., Kim, B., Kim, J., Kong, D., & Kim, S. (Year). VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design. [Journal/Conference Name], [pages].

Model Card Contact

https://t.me/voicestuff

https://t.me/frappuccino

https://github.com/shigabeev