elitexp/nepali-call-center-cosyvoice-experimental

Nepali Call Centre Tone — Experimental

A fine-tune of FunAudioLLM/Fun-CosyVoice3-0.5B-2512.

Experimental. Published to share a working Nepali adaptation and an honest record of what did and did not transfer. Pronunciation is solid; the call-centre register is partial — see What it is good at, and what it is not. Not validated for production use.

The base model covers nine languages; Nepali is not one of them, and the released weights read Nepali text poorly. This fine-tune trains the LLM on Nepali speech so that it does.

Only llm.pt differs from the base. Flow and the HiFT vocoder are unchanged and frozen at 24 kHz — which means voice timbre comes from the prompt clip you supply at inference, not from these weights. This is a language and prosody fine-tune, not a voice.

Quick start

A one-file generator is included:

git clone https://github.com/FunAudioLLM/CosyVoice.git
pip install -r CosyVoice/requirements.txt

./generate.py --prompt samples/prompt.wav \
              --text "नमस्कार, हाम्रो सेवामा हजुरलाई स्वागत छ।" \
              --out output.wav          # or output.mp3

samples/prompt.wav can be as short as an 8.5 s reference clip. Swap in any 5-10 s recording of a different voice and the output follows that voice instead — the weights carry the language, not the speaker.

generate.py defaults to the call-centre style tag — the register this checkpoint was fine-tuned for. Pass --neutral for the plain tag, or --instruct "...<|endofprompt|>" for your own.

It also normalises the text (see below), reads long digit runs out digit-by-digit, and synthesises sentence by sentence. Those are not conveniences; skipping them produces audibly wrong output with no error.

First run downloads ~48 MB of text-frontend resources from ModelScope, so it needs network once.

Usage

Load it exactly like the base model:

from cosyvoice.cli.cosyvoice import CosyVoice3

cv = CosyVoice3("path/to/this/repo")

# The style tag is the only channel register has into the model. This
# checkpoint was fine-tuned with the call-centre tag on ~4,700 utterances;
# the neutral tag ("You are a helpful assistant.<|endofprompt|>") leaves most
# of that unused.
instruct = ("You are a helpful assistant. Speak like a friendly Nepali "
            "call-centre agent: warm, brisk, natural phrasing.<|endofprompt|>")
text     = "नमस्कार, हजुरलाई कसरी सहयोग गर्न सक्छु?"

for out in cv.inference_instruct2(text, instruct, "your_prompt.wav",
                                  stream=False, text_frontend=False):
    audio = out["tts_speech"]      # 24 kHz mono

your_prompt.wav is a few seconds of the voice you want. Anything from ~5–10 s of clean speech works.

Normalise your text first

Two rules, both learned the hard way:

• Map danda to a period. Training text contains no । at all, so an untouched danda reaches the tokenizer as a byte sequence the model has never seen and gets voiced — it comes out as a spoken "s", not a pause.
• Strip ZWJ/ZWNJ (U+200D, U+200C). Same reason. पुर्‍याउन typed with the joiner matches nothing; the corpus only ever contains पुर्याउन.

def normalize(s):
    for a, b in (("॥", "."), ("।", ".")):
        s = s.replace(a, b)
    return s.replace("‍", "").replace("‌", "")

Spell digit runs out in words too — phone numbers and amounts are otherwise read as one enormous cardinal.

Keep utterances short

Trained on clips averaging ~3.5 s. Hand it a long paragraph and it produces a rushed run-on. Split on sentence boundaries and synthesise piece by piece.

Style tags

tag	when
`You are a helpful assistant. Speak like a friendly Nepali call-centre agent: warm, brisk, natural phrasing.<	endofprompt
`You are a helpful assistant.<	endofprompt

The tag must end in <|endofprompt|>. Timbre is unaffected by it — that comes from the prompt clip. What the tag moves is rate, pitch range and phrasing.

What it is good at, and what it is not

Good: Nepali pronunciation. Measured character error rate of 0.294 when generated speech is transcribed back through a telephony codec — against a 0.307 floor measured by putting human speech through the same pipeline. It is at the measurement floor, not merely close to it.

Limited: prosody control. The model was also trained toward a warm call-centre register via an instruct tag. Speaking rate and pitch variation moved usefully (+16% rate, F0 spread 2.73 semitones against a 2.92 reference), but breath-pause structure did not — the training clips were single short sentences with almost no internal pauses, so the model never learned to place them. Expect brisk, expressive, under-punctuated speech.

Evaluation

metric	this model	reference
CER through G.711 8 kHz	0.294	0.307 (human speech, same ASR)
speaking rate, tag on vs off	+16%	—
F0 spread	2.73 st	2.92 st (target)
pause density	did not transfer	0.343 /s (target)

Provenance

Exactly one file differs from the base model: llm.pt. flow.pt, hift.pt, the ONNX front end and cosyvoice3.yaml are byte-identical to FunAudioLLM/Fun-CosyVoice3-0.5B-2512. Only the LLM is trained in this pipeline.

Checkpoint: stageV:epoch_4_whole — epoch 4 of a 5-epoch run. Note that epoch 4 was not that run's validation minimum (epoch 2 was); it was chosen as the most strongly adapted checkpoint.

Training data: Nepali ASR corpora, plus call-centre style speech from several sources, both recorded and synthetic. The apache-2.0 tag is inherited from the base weights.

No prompt or reference clips are included. Timbre comes entirely from the clip you supply, and the clips used in development are recordings of identifiable people.

Citation

Base model:

@article{cosyvoice3,
  title={CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training},
  journal={arXiv preprint arXiv:2505.17589},
  year={2025}
}