indicF5-best (research)

Swamiji's Telugu voice from the F5 family: the IndicF5 pick since 2026-11-05. It was first pushed as indicf5-swamiji-te-87h-nav-2stage (swamiji-tts experiment 07/06, stage B):

1. ai4bharat/IndicF5
2. 10 epochs on Navaratri
3. 1 epoch on 83 h of his Telugu talks (stage A)
4. 5 more epochs on Navaratri

It needs a reference clip of his voice and that clip's transcript. The one used in every evaluation is in this repo: reference/p3R1wZLty6A_029.wav, 10 s, from his Navaratri talks.

Use (float32, 16 steps)

pip install f5-tts==1.1.22 soundfile

import soundfile as sf
import torch
from huggingface_hub import hf_hub_download
from importlib.resources import files
from omegaconf import OmegaConf
from hydra.utils import get_class
from safetensors.torch import load_file
from pypinyin import Style, lazy_pinyin
import rjieba
import f5_tts.infer.utils_infer as ui
from f5_tts.model import CFM
from f5_tts.model.utils import get_tokenizer

REPO = "sw-voice/indicF5-best"
REF_TEXT = "నీకు నమస్కారం. మాకిప్పుడు వేదాలను అనుగ్రహించు. కానీ ముందుగా మాకు అన్నం పెట్టి మా ఆకలి తీర్చు అని కోరారు."
TEXT = "భయం అనేది భేదం చూడటం వల్ల వస్తుంది. మనం భగవంతుణ్ణి తలచుకుంటూ, సాధన చేస్తూ ఉంటే అది తగ్గిపోతుంది."
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"


def telugu_safe_tokens(text_list, polyphone=True):
    """F5's convert_char_to_pinyin sends every 3-byte segment (Telugu too) to pypinyin, which garbles it; keep pinyin for Chinese only."""
    is_zh = lambda c: "㄀" <= c <= "鿿"
    out = []
    for text in text_list:
        cl = []
        for seg in rjieba.cut(text.translate(str.maketrans({";": ",", "“": '"', "”": '"', "‘": "'", "’": "'"}))):
            n = len(seg.encode("utf-8"))
            if n == len(seg):
                if cl and n > 1 and cl[-1] not in " :'\"":
                    cl.append(" ")
                cl.extend(seg)
            else:
                for c in seg:
                    if is_zh(c):
                        cl.append(" "); cl.extend(lazy_pinyin(c, style=Style.TONE3, tone_sandhi=True))
                    else:
                        cl.append(c)
        out.append(cl)
    return out


ui.convert_char_to_pinyin = telugu_safe_tokens
if device != "cuda":  # F5 speaks the text pieces in parallel threads, which crashes Apple's MPS: one piece at a time instead
    from concurrent.futures import ThreadPoolExecutor
    ui.ThreadPoolExecutor = lambda *a, **k: ThreadPoolExecutor(max_workers=1)


def load_indicf5_best():
    """F5TTS_Base with this repo's vocab, weights kept in float32 (float16 turns IndicF5-family weights into noise; F5's own
    loader casts to float16 on CUDA, so the model is built here instead)."""
    vocab = hf_hub_download(REPO, "vocab.txt")
    mc = OmegaConf.load(str(files("f5_tts").joinpath("configs/F5TTS_Base.yaml"))).model
    vmap, vsize = get_tokenizer(vocab, "custom")
    ms = mc.mel_spec
    model = CFM(transformer=get_class(f"f5_tts.model.{mc.backbone}")(**mc.arch, text_num_embeds=vsize, mel_dim=ms.n_mel_channels),
                mel_spec_kwargs=dict(n_fft=ms.n_fft, hop_length=ms.hop_length, win_length=ms.win_length, n_mel_channels=ms.n_mel_channels,
                                     target_sample_rate=ms.target_sample_rate, mel_spec_type=ms.mel_spec_type),
                odeint_kwargs=dict(method="euler"), vocab_char_map=vmap)
    sd = {k.removeprefix("ema_model."): v for k, v in load_file(hf_hub_download(REPO, "model.safetensors")).items()}
    missing, unexpected = model.load_state_dict(sd, strict=False)
    assert not unexpected and all(k.startswith("mel_spec.") for k in missing), (missing[:5], unexpected[:5])
    return model.float().to(device).eval()


model = load_indicf5_best()
vocoder = ui.load_vocoder("vocos", device=device)
ref_audio, ref_text = ui.preprocess_ref_audio_text(hf_hub_download(REPO, "reference/p3R1wZLty6A_029.wav"), REF_TEXT)
wav, sr, _ = ui.infer_process(ref_audio, ref_text, " ".join(TEXT.split()), model, vocoder, "vocos",
                              nfe_step=16,              # 32 = full original quality; 16 rated the same by Gemini, half the time
                              cfg_strength=2, sway_sampling_coef=-1,
                              speed=0.92,               # Swamiji's pace with this reference clip (1.0 speaks ~16% fast; 0.86 = his slowest, Navaratri)
                              cross_fade_duration=0.15, device=device)
sf.write("indicf5_best.wav", wav, sr)

Notes

• Keep float32. float16 produces noise with IndicF5-family weights.
• The tokenizer patch is required for Telugu. Without it, F5 garbles the text.
• Long text works. F5 splits it into pieces of about 12 s and joins them with a 0.15 s cross-fade.
• On a Mac, F5 would otherwise generate those pieces in parallel threads, which crashes MPS; the patch makes it generate them one at a time.
• Pace. F5 copies its pace from the reference clip. With this clip, speed=0.92 gives 0.109 s per letter, his Mahabharatam pace. 0.89 matches his 83 h average (0.113), and 0.86 his slower Navaratri pace (0.117), which sounded too slow. At 1.0 it speaks about 16% fast.
• Steps. 32 steps (nfe_step=32) is full original quality. 16 steps is half the time, and the Gemini judge rated it the same (prosody 4.08 vs 4.00, clean audio equal).

Speed

Timed with the code above (float32, 10 s reference; at speed 0.86, so the speech was 7% longer than at 0.92) on a 75 s Telugu answer. Gen time per minute = time to generate one minute of speech.

Machine	Steps	Gen time per minute	Speed vs real time
MacBook Air, Apple M5, 16 GB (MPS)	16	about 2 min	1.97× slower
MacBook Air, Apple M5, 16 GB (MPS)	32	about 5 min	4.83× slower
NVIDIA A100	32	about 26 s	2.3× faster

An RTX 3090 handles one call at a time in float32 at 16 steps, with 1.7 s to first audio. An RTX 4090 handles two at a time.

On the Mac, 32 steps took 2.4× as long as 16 steps rather than 2×. It ran straight after the 16-step test, and the fanless Air slows down as it heats up, so expect 4–5 minutes per minute of speech at full quality.

Training

• Base: sw-voice/indicf5-swamiji-te-navaratri-10ep, which is ai4bharat/IndicF5 plus 10 epochs on 402 Navaratri clips.
• Stage A data: sw-voice/swamiji-telugu-reuse-train without its shlokas: Mahabharatam, Bhagavatam and Datta Maata themes (82.97 h), RE-USE-cleaned.
  • Re-cut at their own VAD clip boundaries into 15,364 pieces of up to 25 s (single clips up to 30 s), 81.62 h. The cut is needed because F5's loader skips anything over 30 s.
  • Plus the 402 Navaratri clips, for 15,766 samples and 83.41 h.
• Recipe: upstream SWivid/F5-TTS 1.1.22 finetune_cli, F5TTS_Base, IndicF5 vocab (tokenizer custom), bf16, LR 1e-5, batch 9,600 frames, grad clip 1.0.
  • Stage A: 1 epoch (3,302 updates, 8.6 min on one H100).
  • Stage B: the 402 Navaratri clips, 5 epochs (355 updates, 1.7 min), warm-up 70, starting from stage A's online weights in a fresh EMA.
• Weights: the online weights in float32, under ema_model.* keys.

Evaluation (10 Telugu whole answers, seeds 0/1, float32, 32 steps, speed 1)

	indicF5-best	Stage A only	Base (Navaratri 10ep)	best-vox (VoxCPM2)
Voice similarity, WavLM-ECAPA (% of real vs real)	96.6	95.9	95.7	96.6
CER / WER, MAI-Transcribe-2 (%)	1.84 / 8.4	1.27 / 7.5	1.24 / 6.7	1.15 / 7.6
Pitch spread (semitones; real 4.55)	3.91	3.89	4.03	4.17
Pauses per min, short / sentence / long (real 7.1 / 14.0 / 0.6)	8.4 / 10.6 / 0.4	9.2 / 10.0 / 0.2	11.4 / 10.7 / 0.0	9.8 / 10.6 / 3.0
Gemini prosody gap vs base (both orders)	−0.03 (tie)	−0.29
Gemini prosody gap vs best-vox (both orders)	+0.76 (6 answers to 1)	+0.53	+0.80

Telugu Q&A (the 10 answers of sw-voice/swamiji-telugu-spiritual-qa, at speed 0.86): CER 1.25%, WER 5.6%. The Space now plays speed 0.92.

Code: swamiji-tts research/exp-iterations/07-f5-tts/experiments/06-indicf5-te87h-two-stage/.