Swar

Swar (स्वर, "voice") is a small, experimental text-to-speech model for Nepali and English, including sentences that mix the two, tuned toward a calm customer-service manner of speaking.

38.5 M parameters, 16 kHz output. The architecture is derived from Kokoro-82M (StyleTTS2 family), built from its layer classes: non-autoregressive, so a sentence is generated in one pass.
Reads Devanagari and Latin letters directly: no phonemiser, no language switch. Numbers, currency ("Rs. 250", "रु ५००"), times and phone numbers are written out in words in the language of their sentence.
Default voice: bijaya (voices/bijaya.pt). Any 3-10 s clip of one speaker gives another voice.
Fast: about 15x real time on a desktop CPU with all cores, about 6x on a single thread, and about 30x on a consumer GPU (measured on a Ryzen 9 5900X and an RTX 3090).

Try it in the Space: Nepali Call Center Experimental - Powered by Swar.

Quick start

pip install torch torchaudio transformers safetensors huggingface_hub soundfile

import sys, soundfile as sf
from huggingface_hub import snapshot_download

folder = snapshot_download("elitexp/Swar")
sys.path.insert(0, folder)
from swar import Swar

tts = Swar.from_pretrained(folder)                       # GPU if available, else CPU
wav, sr = tts.synthesize("नमस्ते, ग्राहक सेवा केन्द्रमा यहाँलाई स्वागत छ।")   # voice "bijaya"
sf.write("hello.wav", wav, sr)

# another voice from a reference clip, a little slower, with clear pauses at punctuation
wav, sr = tts.synthesize("Your balance is Rs. 250. धन्यवाद।",
                         voice="my_reference.wav", speed=0.95, pause_floor_ms=300)

From the command line, inside the downloaded folder:

python -m swar.inference "नमस्ते, यहाँलाई स्वागत छ।" -o hello.wav

Options

argument	meaning
`voice`	a name from `voices/` (default `bijaya`), a `.pt` of style vectors, or an audio clip (wav / flac / mp3 / ogg)
`speed`	0.5-2.0, scales the predicted durations (1.0 = as predicted)
`pause_floor_ms`	every comma / full stop / question mark inside a sentence lasts at least this long (0 = as predicted)
`loanwords`	in a mostly-Nepali sentence, common English words are said the Nepali way ("recharge" -> रिचार्ज, "balance" -> ब्यालेन्स; list in `swar/loanwords.py`). Default on
`sentence_pause_ms`	silence between two sentences (default 350); the start and end of each sentence are trimmed, so the first sound comes within about 0.1 s
`return_info`	also return timing and the exact letters the model read

What is inside

part	params	role
PL-BERT (ALBERT, letters) + projection	3.5 M	contextual letter features
prosody predictor	9.7 M	duration per letter, pitch and energy at 80 Hz
prosodic style encoder	3.5 M	manner of speaking from a reference clip
text encoder	3.2 M	letter features for the decoder
decoder (iSTFTNet with harmonic source)	15.1 M	waveform at 16 kHz
acoustic style encoder	3.5 M	timbre from a reference clip

A voice is two 128-d vectors: acoustic (timbre, read by the decoder) and prosodic (manner, read by the predictor). voices/bijaya.pt stores them as {"s_ac", "s_pr"}.

Files: swar.safetensors (inference weights only), config.json, swar/ (model, text front-end, mel, inference; Kokoro's layer code is vendored under swar/kokoro/ with its Apache-2.0 licence), voices/.

Training

Trained from scratch in five stages, after the StyleTTS2 / Kokoro recipe adapted to letters at 16 kHz:

PL-BERT on Nepali and English text (masked letters).
Letter aligner (CTC) on both languages, for durations.
Acoustic model: text encoder, decoder and both style encoders, adversarial training with multi-period and multi-resolution discriminators (300 k steps).
Prosody: duration, pitch and energy predictors and the prosodic style encoder (100 k steps).
Register: a short fine-tune (6 k steps) of the prosody half on about 8 hours of customer-service style Nepali recordings, with general data replayed so the rest of the model does not drift.

Data (acoustic and prosody stages): about 800 hours of Nepali speech from public corpora (including OpenSLR 43 and OpenSLR 143) and other licensed and in-house recordings, part of it synthetic, and about 570 hours of English speech synthesised with Kokoro-82M (af_heart).

Evaluation

Copy-synthesis intelligibility (stage 3, durations and pitch taken from the real clip, recognised by the same ASR as the real recordings): Nepali character error rate through a G.711 telephone channel 0.199 against 0.220 for the real recordings (285 clips); English word error rate 0.097 against 0.098 (200 clips).
Customer-service register (stage 5): median pitch stays within 15 Hz of the reference voice. The pause rate (about 0.13 pauses per second, about 0.20 with pause_floor_ms=300) is still below the 0.30-0.45 target band: pauses are only lengthened where the text has punctuation.
Full text-to-speech intelligibility has not yet been measured with the same protocol.

Limitations

Experimental. Rare words, names and some English spellings can be mispronounced; Nepali rhythm is less polished than English.
16 kHz, telephone-oriented; not a studio-quality voice.
Voices from a reference clip copy that clip's recording conditions too; use clean audio.

Responsible use

Swar can speak in a voice taken from a short recording. Only use voices you have permission to use, label synthetic speech as synthetic where people could mistake it for a person, and do not use the model to impersonate anyone or to deceive.

Acknowledgements

Swar would not exist without Kokoro-82M by hexgrad. Its architecture is Swar's blueprint: the text encoder, the prosody predictor (duration, pitch and energy), the iSTFTNet decoder with its harmonic source and the split acoustic / prosodic style design all come from Kokoro, and Kokoro's own layer code is vendored in swar/kokoro/. Swar re-sizes that design for 16 kHz, reads letters instead of phonemes, adds trainable style encoders and is trained from scratch; Kokoro (voice af_heart) also generated Swar's English training speech. Kokoro in turn builds on StyleTTS2 (Li et al.), whose style encoder and PL-BERT recipe Swar also uses.

License

The Swar weights (swar.safetensors, voices/) and Swar's own code are released under CC BY-NC 4.0: free for research and other non-commercial use, with attribution. The vendored Kokoro code in swar/kokoro/ stays under its original Apache-2.0 licence (swar/kokoro/LICENSE); the StyleTTS2 style encoder code is MIT. Training data includes OpenSLR corpora under their own licences.

Downloads last month: 23

elitexp
/

Swar