themohal/saraiki-vits-tts

Saraiki VITS TTS

facebook/mms-tts-urd-script_arabic fine-tuned for Saraiki (ISO 639-3 skr) on ~7.5 h of single-speaker speech, with the full VITS GAN objective: mel, KL, duration, adversarial and feature-matching losses. The HiFi-GAN discriminator was initialised from the official MMS release (facebook/mms-tts -> full_models/urd-script_arabic/D_100000.pth).

The character vocabulary was extended from 59 to 92 entries to cover the Saraiki letters absent from Urdu - including the implosives ݨ ݙ ڳ ٻ ڄ - and the text embedding resized accordingly.

The bundled tokenizer is required; the base Urdu tokenizer will not work.

from transformers import VitsModel, VitsTokenizer
import torch, soundfile as sf
m = VitsModel.from_pretrained('themohal/saraiki-vits-tts')
t = VitsTokenizer.from_pretrained('themohal/saraiki-vits-tts')
x = t('تہاݙے کیا حال ہِن'', return_tensors='pt')
with torch.no_grad(): w = m(**x).waveform[0].numpy()
sf.write('out.wav', w, 16000)

🚀 Load and Run a Specific Training Checkpoint

The following example loads the base Saraiki VITS model and explicitly replaces its weights with a specific training checkpoint from the Hugging Face repository.

import torch
import soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import VitsModel, VitsTokenizer

# Hugging Face repository
model_name = "themohal/saraiki-vits-tts"

# 1. Load tokenizer and base VITS model
tokenizer = VitsTokenizer.from_pretrained(model_name)
model = VitsModel.from_pretrained(model_name)

# 2. Download the specific training checkpoint
print("Downloading checkpoint_epoch_100.pt...")

checkpoint_path = hf_hub_download(
    repo_id=model_name,
    filename="checkpoints/checkpoint_epoch_100.pt"
)

print(f"Checkpoint downloaded to: {checkpoint_path}")

# 3. Load checkpoint weights
print("Loading checkpoint weights...")

raw_checkpoint = torch.load(
    checkpoint_path,
    map_location="cpu"
)

# Extract model state dictionary
if "model" in raw_checkpoint:
    state_dict = raw_checkpoint["model"]
elif "generator" in raw_checkpoint:
    state_dict = raw_checkpoint["generator"]
else:
    state_dict = raw_checkpoint

### Load checkpoint into VITS model
model.load_state_dict(
    state_dict,
    strict=False
)

model.eval()

# 4. Generate Saraiki speech
text = "تہاݙے کیا حال ہِن"

inputs = tokenizer(
    text=text,
    return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**inputs)

# 5. Convert waveform to NumPy
waveform = outputs.waveform.cpu().numpy().squeeze()

# 6. Save generated audio
output_file = "epoch_100_output.wav"

sf.write(
    output_file,
    waveform,
    model.config.sampling_rate
)

print(f"Finished. Saved generated audio to {output_file}")

📊 Evaluation Baseline (10-Epoch Checkpoint)

Automated Speech Recognition (ASR) alignment evaluations were run over a held-out test set to trace the Character Error Rate (CER) gap between real speech recordings and the synthesized outputs.

Performance Metrics

• Total Steps Trained: 3,825 steps (~10 Epochs)
• Real Recordings Mean CER: 0.120 (Ground-truth verification metric)
• Synthesized Audio Mean CER: 0.515
• Synthesized-to-Real Gap: +0.395
• Status: WEAK — The model output is audible but phonetic details remain unstable under heavy character sequences.

Hard-Sample Error Patterns

During early iterations, the generator experiences dropping or hallucinating phonemes when encountering structural Saraiki phrases:

• Reference: جݙاں چور ہاسے، تݙاں کُجھ ہور ہاسے
• Synthesized: چرینان چوہ رہافسے، ݙانہ اچ اہو رہافسے (Individual CER: 0.576)
• Reference: تُوں اپݨی نبیڑ ٻِنہاں کوں ناں چھیڑ
• Synthesized: اوں نیدلے ݙینہ ہم اوں دانچ (Individual CER: 0.647)

🛑 Limitations & Future Work

This is a proof-of-concept base checkpoint trained within limited GPU hours. Contributors can optimize performance via the following paths:

• Extended Steps: 10 epochs are insufficient for stable GAN text-to-wave alignment. Training should be pushed past 50,000–100,000 steps to allow feature matching (fm) and adversarial (adv) losses to converge cleanly.
• Character Distribution Balancing: Implosives like ٻ or ڳ suffer from high error rates due to lower frequency in natural text layouts. Targeted text corpus upsampling or custom weighted loss masks are recommended.
• Lexicon Expansion: The single-speaker ~10-hour dataset limits vocabulary depth. The model requires broader coverage to accurately capture compound formal words found in traditional literature.
• Multi-Speaker Architectures: Moving from single-speaker configurations toward multi-speaker variants will enhance dialectal adaptation across regional Southern and Northern variant accents.

Training Pipeline

The end-to-end training notebook, dataset pipelines, and extended vocabulary parameters are available here:

• 👉 Kaggle Notebook: Saraiki VITS GAN TTS
• 👉 Kaggle Notebook: Saraiki VITS GAN TTS CONTINOUS TRAINING

📜 Data and License

Derived from the Saraiki 10 Hours TTS Dataset (MirasAI, Mozilla Data Collective), licensed under CC-BY-NC-SA-4.0. Given that the core MMS checkpoints use CC-BY-NC-4.0, this fine-tuned variation is strictly for non-commercial use with attribution required. No speaker identity theft, voice cloning, or commercial imitation is permitted.

Developed by: Muhammad Farjad Ali Raza