themohal/saraiki-speecht5

Saraiki SpeechT5 TTS 🇵🇰

A fine-tuned SpeechT5 model for Saraiki Text-to-Speech (TTS).

Base Model: microsoft/speecht5_tts
Language: Saraiki (سرائیکی)
Task: Text-to-Speech
Vocoder: microsoft/speecht5_hifigan

Installation

Install the required packages:

pip install -U transformers torch soundfile datasets

Inference

import torch
import soundfile as sf
from datasets import load_dataset
from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan

model_id = "themohal/saraiki-speecht5"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = SpeechT5Processor.from_pretrained(model_id)
model = SpeechT5ForTextToSpeech.from_pretrained(model_id).to(device)
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").to(device)

embeddings = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
speaker_embedding = torch.tensor(
    embeddings[0]["xvector"]
).unsqueeze(0).to(device)

text = "میڈا ناں فرجاد اے"
inputs = processor(text=text, return_tensors="pt")

with torch.no_grad():
    speech = model.generate_speech(
        inputs["input_ids"].to(device),
        speaker_embeddings=speaker_embedding,
        vocoder=vocoder,
    )

sf.write(
    "saraiki_output.wav",
    speech.cpu().numpy(),
    16000
)

Example

Input: میڈا ناں فرجاد اے

Output: Saraiki speech audio.

SpeechT5 requires a 512-dimensional speaker embedding. The example uses a CMU Arctic x-vector for demonstration. For best results, use a speaker embedding matching your target voice.

Training

The model was fine-tuned on Kaggle using the following notebooks:

• Saraiki SpeechT5 Kaggle Fine-tuning
• Saraiki SpeechT5 Continue Training

Limitations

This is an experimental low-resource Saraiki TTS model. Pronunciation and speech quality may vary depending on the input text and speaker embedding.

Citation

@article{ao2022speecht5,
  title={SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing},
  author={Ao, Junyi and Wang, Rui and Zhou, Long and others},
  journal={arXiv preprint arXiv:2110.07205},
  year={2022}
}

Author

Muhammad Farjad Ali Raza

Hugging Face