prime2070/kokoro-ninja

Kokoro Ninja 🥷 — fast multilingual zero-shot voice cloning

Clone a voice from a few seconds of audio and speak 7 languages in it — Vietnamese, English, Chinese, French, German, Japanese, Korean — at ~15× realtime on a single GPU (16.3× measured).

StyleTTS2 + a CAMPPlus speaker encoder. One non-autoregressive forward pass: no diffusion sampler, no autoregressive decode loop. Generation is deterministic.

Code: https://github.com/primepake/kokoro-ninja
Demo: https://primepake.github.io/kokoro-ninja/

Usage

import soundfile as sf
from kokoro_ninja import KokoroNinja

tts = KokoroNinja.load(
    checkpoint_path="kokoro-ninja-multilingual.pth",
    config_path="config_inference.yml",
    campplus_onnx_path="campplus/campplus.onnx",
    device="cuda:0",
)

style = tts.compute_style("reference.wav")     # 3-10s of the voice to clone

for lang, text in [
    ("vi", "Xin chào, đây là giọng nói được nhân bản."),
    ("en", "Hello, this is a cloned voice."),
    ("ja", "こんにちは、これはクローンされた声です。"),
]:
    wav, sr = tts.synthesize(text=text, ref_s=style, language=lang)
    sf.write(f"out_{lang}.wav", wav, sr)

One reference clip carries the voice across every supported language. compute_style() is per-voice — compute once, reuse for every sentence.

Requires espeak-ng (the phonemizer backend). Easiest correct install: pip install espeakng-loader, which bundles 1.52 with complete dictionaries. The model was trained against 1.50, but we measured 1.50 vs 1.52 as producing identical phonemes on all 30 Vietnamese benchmark sentences and on en/fr/de/ja/ko.

If you build espeak from source, compile the dictionaries explicitly — a corrupt en_dict makes espeak spell words out letter by letter ("B-O-N-J-O-U-R"), silently wrecking English, French and German. See the repo README.

Files

File	What it is
kokoro-ninja-multilingual.pth	the model (optimizer state stripped)
config_inference.yml	inference config — resolves asset paths relative to itself, keep it beside Utils/ and campplus/
campplus/campplus.onnx	CAMPPlus speaker encoder
Utils/ASR, Utils/JDC, Utils/PLBERT	text aligner, F0 extractor, PL-BERT encoder loaded at runtime

Benchmarks

One NVIDIA L4, VIVOS test split, 10 held-out speakers × 3 sentences, identical references and sentences for every system:

System	RTF ↓	×realtime	SECS ↑	UTMOSv2 ↑	WER ↓
Kokoro Ninja	0.061	16.3×	0.894	2.660	3.9%
omnivoice-vietnamese	0.437	2.3×	0.916	2.660	2.6%
VieNeu-TTS v3 Turbo	0.718	1.4×	0.931	2.445	3.3%
human recording	—	—	0.914	3.019	5.7%

Honestly: speed is the win (7–12× faster than the alternatives on the same GPU) and we tie omnivoice-vietnamese for best predicted naturalness, but speaker similarity trails the autoregressive models — 0.894 against a 0.833 floor (two different speakers) and a 0.914 ceiling (the same speaker's other recordings), about three-quarters of the usable range. If you need maximum timbre fidelity from a single clip rather than throughput, VieNeu v3 Turbo is currently better. Intelligibility sits below the human recordings' own 5.7% WER.

RTF is full-utterance compute ÷ audio duration, not time-to-first-audio. SECS uses WavLM-base-plus-sv rather than CAMPPlus, since scoring with the encoder the model is conditioned on would flatter it. UTMOSv2 is nondeterministic — rescoring identical audio moves it by up to 0.08, so treat small MOS gaps as noise; SECS and WER are exactly reproducible.

Always end your text with punctuation

The model is trained on sentence-final punctuation and is sensitive to it: text without a terminal mark tends to clip or run on at the end. synthesize() appends one when your text lacks it. The effect is not subtle — on the benchmark above, adding the missing period moved WER from 5.1% to 3.9% and predicted MOS from 2.47 to 2.66.

Training data

Corpus	Role	Licence
viVoice	Vietnamese, ~1,000 h	CC-BY-NC-SA-4.0
Emilia (subset)	multilingual	CC-BY-NC-4.0
internal corpus	Vietnamese	proprietary

Base checkpoint: StyleTTS2's LibriTTS model, adapted to the Vietnamese phoneme vocabulary and extended to the multilingual token set.

Licence — non-commercial

CC-BY-NC-SA-4.0. This is inherited, not chosen: viVoice is CC-BY-NC-SA (NonCommercial and ShareAlike) and the Emilia subset is CC-BY-NC, so the strictest terms propagate to anything trained on them.

You may use these weights for research, evaluation and personal projects. You may not use them in a commercial product, and derivatives must carry the same licence.

The inference code is separately Apache-2.0.

Limitations

• Speaker similarity trails the best autoregressive/codec models (see benchmarks).
• No text normalization — expand numbers, dates and currency before synthesis. This is the most common source of real-world errors and is not handled here.
• Noisy or very short (<3 s) references degrade cloning; there is no built-in denoiser.
• Benchmarked on Vietnamese; the other six languages are supported but not yet benchmarked.
• Japanese input must be kana — espeak-ng reads hiragana and katakana but silently drops kanji, so convert with pyopenjtalk/MeCab first. Every other language takes normal text.
• 24 kHz output, no streaming.

Acknowledgements

StyleTTS2 (Li et al.), 3D-Speaker / CAMPPlus, VIVOS for evaluation, PhoWhisper (VinAI) and UTMOSv2 for scoring.