nepali-tts-pipelines

These are the code and recipes from the Ampixa Nepali TTS work that ran on a Thunder Compute RTX A6000, 2026-11-01 to 2026-11-03. It holds the data-selection, preprocessing, tokenisation, fine-tuning, generation, scoring and bench-comparison code for:

• OmniVoice (k2-fsa) zero-shot and fine-tuned (ft_v1, ft_v2 data, ft_v3);
• Swar / Swar-X (F5-TTS based);
• Audio8 / Audio8-F;
• the head-to-head (h2h) harness on the Nepali Voice Cloning Bench.

Not included: audio, codec tokens, weights and data rows. The only data file is a text-free ID manifest (manifests/). The training data includes YouTube-derived audio under an Ampixa–channels MOU (internal only) and voidash/nepali-speech-compact-v1 (research-only, no redistribution), so it cannot be published here. The bulk archive, with audio, checkpoints and tokens, is the gated dataset voidash/tts-a6000-archive-20261103. Its checksums and full provenance are in the private voidash/tts-a6000-archive-provenance-20261103.

Licences

• Code in this repository (ours): licence TBD by the owner. Internal research code, shared under gated access for review. No licence is granted until the owner sets one.
• OmniVoice: code k2-fsa/OmniVoice @ 08be0b4 is Apache-2.0. Weights k2-fsa/OmniVoice @ c5fdb5ccb189668d56333f77ba2629f4cd7535f4 are CC-BY-NC and are not included. Its audio_tokenizer (Higgs Audio 2) is under the Boson Higgs Audio 2 Community License.
• Swar: code is MIT. Weights Aarjan/swar @ 9e5752a7244f4ed4dc7e3fb2d1d6c2c1045516fd are CC BY-SA 4.0 and are not included.
• Nepali Voice Cloning Bench (Naamche Labs) code: MIT. It is not vendored here; fetch it at the pinned revision (see audio8f/h2h/README_L40_PORT.md).
• FLEURS (google/fleurs, ne_np): CC BY 4.0.

Layout

The tree mirrors the original $HOME (/home/ubuntu) on the A6000, and many scripts use ~/omnivoice/...-style paths. To restore, copy each top-level folder into $HOME, or run as a user whose home holds this layout. A few JSON configs record absolute /home/ubuntu/... paths, as provenance only.

Stage	Path	README
OmniVoice zero-shot setup, dev generator, sweeps	omnivoice/code/, omnivoice/sweep/	omnivoice/README.md
ft_v1 data build (manifest, fetch, tokenise)	omnivoice/code/ft_*.py	omnivoice/data/ft_v1/README.md
ft_v2 data build (ASR/SNR screening, GPU tokenise)	omnivoice/code/ft_v2/	omnivoice/data/ft_v2/README.md
ft_v3 data build (cross-utterance pairs, ResNet34 embeddings)	omnivoice/code/ft_v3/	omnivoice/data/ft_v3/README.md
ft_v1 / ft_v3 training + eval hook + gate	omnivoice/exp/ft_v1/, omnivoice/exp/ft_v3/	their README.md
fix_v2 patch module + dev generator	omnivoice/fix_v2/ (omnivoice_patches.py, fx_gen.py, configs/E*.json)	omnivoice/fix_v2/README.md
Frozen configs v1 / v2	omnivoice/final_configs/	omnivoice/wrapup_v1/README.md
wrapup_v1 (decision rules, selection rule, pipeline, bench_synth_v2.py)	omnivoice/wrapup_v1/	omnivoice/wrapup_v1/README.md
bench_proxy_v1 (bench-scorer wrapper, parity, fleurs_dev_v3 builder build_v3.py)	omnivoice/bench_proxy_v1/code/	omnivoice/bench_proxy_v1/README.md, audio8f/dev/fleurs_dev_v3/README.md
Other OmniVoice studies (denoise rules, prompt enhancement, pruning, SV rerank, pronunciation, RL r1)	omnivoice/{denoise_rule_v1,denoise_rule_v2,promptenh_v1,prune_v1,rerank_sv_v1,dev_pron_v1,rl_r1,infer_v2,neptts_fleurs_diag_v1}/	their README.md
Swar-X data map, text prep, mel extraction, training	swarx/code/ (build_map.py, prep_text.py, pilot_mel.py, extract_mel_full.py, swarx_train.py, swar_dev_gen.py), swarx/data_map/src/code/, swarx/ft_v1/	swarx/README.md
Swar environment	swar/env/ (build_venv.sh, download.py, freezes)	inside the scripts
Audio8-F NAR head	audio8f/src_f/, audio8f/src/	audio8f/README.md
Dev tricks harness (dev_gen, dev_score, dev_summary, naturalness)	audio8f/tricks/code/	audio8f/tricks/README.md
Head-to-head bench harness	audio8f/h2h/code/	audio8f/h2h/README.md, audio8f/h2h/README_L40_PORT.md
Eval port (L40 → A6000 parity, venv build)	audio8f/eval_port/	audio8f/README_EVAL_PORT.md
Environments	env/freeze_*.txt, env/python_versions.txt, env/gpu_a6000.csv	below
ID manifest	manifests/fleurs_dev_v3_item_ids.jsonl (390 items, 5 draws; FLEURS file/sentence IDs only, no text)	below

Setup from scratch (for example, on the L40)

1. Python 3.11 (audio8f/.venv used 3.12) and uv. One venv per stack, pinned by the freezes in env/:
   • env/freeze_omnivoice_venv.txt: OmniVoice (torch 2.8.0+cu128, transformers 5.3.0, -e the OmniVoice repo). Build it with git clone https://github.com/k2-fsa/OmniVoice ~/omnivoice/repo && cd ~/omnivoice/repo && git checkout 08be0b4ccbac3e13e374e86fbfead4b4cac343e2 && UV_PROJECT_ENVIRONMENT=~/omnivoice/.venv uv sync --frozen.
   • env/freeze_swarx_venv.txt, env/freeze_swar_venv.txt: Swar/F5. swar/env/build_venv.sh builds it.
   • env/freeze_audio8f_h2h_venv.txt: the bench harness (bench score.py, MMS, ECAPA, UTMOS).
   • env/freeze_audio8f_tricks_env_venv.txt: the dev scorer (Nepali Conformer, ECAPA, naturalness).
   • env/freeze_audio8f_venv.txt: Audio8-F training.
2. Weights, all at pinned revisions:
   • OmniVoice: omnivoice/code/dl.sh fetches the 13 files of k2-fsa/OmniVoice@c5fdb5cc... into ~/omnivoice/models/OmniVoice. Check model.safetensors sha256 730839316de585f4c8298ec0e1712efc10fb19c6fa4e36eb741cb8d51ebcf6aa.
   • Swar: swar/env/download.py fetches Aarjan/swar@9e5752a7, podvoices_ema.pt, with sha256 b932ea211288795a4a07749b84b51622b2c181cd29343f56f2a5c85598875b53.
   • Selector/scorer models: ampixa/nepali-conformer-offline@a7830b7a (gated, .nemo sha256 9b5306ff...), facebook/mms-1b-all@3d33597, speechbrain/spkrec-ecapa-voxceleb@0f99f2d, SYSPIN/IndicMOS@8270eff, charactr/vocos-mel-24khz@0feb3fd, and the WeSpeaker voxceleb_resnet34_LM ONNX (sha256 7bb2f06e...). The acquire_*.py scripts in audio8f/h2h/code/ fetch them.
3. Dev sets:
   • omnivoice/bench_proxy_v1/code/fetch_src.py downloads pinned FLEURS ne_np dev files without credentials.
   • build_v3.py rebuilds ~/audio8f/dev/fleurs_dev_v3 (5 draws × 78 items). Compare the result with manifests/fleurs_dev_v3_item_ids.jsonl.
   • audio8f/tricks/code/build_dev.py builds fleurs_dev_v2 and fleurs_dev_short_v1.
4. Bench: fetch ampixa/neptts-bench@d07d549 and the cloning bench repo as audio8f/h2h/README_L40_PORT.md describes. The bench is for one final run only; never tune on it.
5. Credentials: scripts that fetch gated repos read HF_TOKEN from stdin or from a private local credential file. No token is stored in this repository.

Run: single-take v2 (frozen)

cd ~/omnivoice/wrapup_v1
~/omnivoice/.venv/bin/python code/bench_synth_v2.py \
  --frozen ~/omnivoice/final_configs/omnivoice_frozen_single_take_v2.json \
  --track clean --seeds 0 --run single_take_v2_clean --root ~/audio8f/h2h/v2 \
  --dev ~/audio8f/h2h/datasets/cloning

• The script verifies the sha256 values recorded in the frozen JSON for omnivoice_patches.py, fx_gen.py and model.safetensors. It refuses to run on a mismatch.
• Use --track raw for the raw-prompt track.
• On a dev draw instead of the bench, pass --dev ~/audio8f/dev/fleurs_dev_v3/draw0.
• Score with the unchanged bench scorer: ~/audio8f/h2h/.venv/bin/python ~/omnivoice/bench_proxy_v1/code/bench_score_wrap.py --dev <DIR> --run <RUN> --name <NAME> --track clean.

Config C1 = upstream defaults + denoise=False, postprocess_output=False, punctuation-to-danda patch and dur_mark_weight 0.6. The base is num_step 32, guidance_scale 2.0, t_shift 0.1, fp16, language npi, a user-provided prompt transcript, and the Swar text frontend on target and prompt text.

Run: best-of-N v2 (frozen)

1. Generate seeds 0..N-1 with --frozen ~/omnivoice/final_configs/omnivoice_frozen_best_of_n_v2.json. N comes from the frozen file; seed 0 is the single take.
2. Compute the selector features per take:
   • Nepali Conformer CER: omnivoice/wrapup_v1/code/feat_conformer.py, in the tricks venv;
   • WeSpeaker ResNet34 cosine of each take against the prompt: feat_resnet.py, in the swarx venv.
3. Apply omnivoice/wrapup_v1/rules/selection_rule_v2.json. The rule uses neither the bench recogniser (MMS) nor the bench speaker model (ECAPA).

omnivoice/wrapup_v1/code/pipeline.py runs the whole dependency graph: dev single take → gate → dev best-of-N → freeze → bench once → results_v2.md. The rule hashes are pre-registered in rules/prereg_hashes.json.

Run: Swar baseline

Run swarx/code/swar_dev_gen.py with podvoices_ema fp32, NFE 16, CFG 2, TF32 off and seed = take index. This is the h2h single-take recipe. For bench rows, see audio8f/h2h/code/synth_swar.py and synth_swar_best4.py.

Latency (TTFT) run: specified, not executed on the A6000

The A6000 was archived before the latency run, so no latency script or result exists here. Protocol, per the 2026-11-03 plan:

• Setup: an exclusive GPU, warm-up first, with cold load time reported separately.
• Items: 60 fleurs_dev_v3 items (not bench items), spread over text length.
• Systems:
  • stock OmniVoice v1 (final_configs/omnivoice_frozen_single_take_v1.json via fix_v2/configs/E0.json);
  • single_take_v2;
  • best_of_n_v2: N takes plus feature extraction plus selection, timed end to end;
  • Swar (podvoices_ema, NFE 16, CFG 2, chunking).
• Per item: time to first audio, total latency and RTF, reported as median and p90.
  • Neither system streams, so TTFT equals total latency, except where text is chunked: then TTFT is the first chunk's time.
  • Record GPU model, clocks, torch/CUDA versions and dtype.
• Timing method: time generate() with CUDA synchronisation, as fx_gen.py already does (gen_s in takes.jsonl).

Recorded results (from the private archive; automatic scores only)

Nepali Voice Cloning Bench: 82 items, 4 speakers, run once with the frozen dev-selected configs (audio8f/h2h/results_v2.md).

Row	Track	CER	Voice (ECAPA)	UTMOS
omnivoice_single_take_v2	clean	5.38	0.6844	2.538
omnivoice_best_of_n_v2	clean	5.15	0.6854	2.545
swar (existing h2h row)	clean	5.06	0.6869	2.667
swar_best4_native	clean	5.13	0.6920	2.706
omnivoice_single_take_v2	raw	5.27	0.6941	2.464
omnivoice_best_of_n_v2	raw	5.00	0.6973	2.485
swar_single	raw	5.07	0.6850	2.398

The fine-tunes ft_v1 (steps 1000–3000) and ft_v3 (steps 500–6000) failed their pre-registered dev gates: voice match did not rise, and CER rose against zero-shot on the gated sets. No fine-tuned checkpoint was promoted.