kokoro-inno-clone-tuner

Zero-shot voice tuner for Kokoro-82M. Intended as an exploration and customization tool, not a strong identity cloner.

Outputs base Kokoro compatible voice packs @ [510, 1, 256].

Samples

Same passage for every voice, enrolled from the reference next to it. LibriTTS-R speakers are dev-clean, held out from training.

voice	tuned	reference
am_libritts1272
af_libritts5338
am_libritts6241
bm_davidattenborough
af_ameliaearhart
bf_janegoodall
am_vincentprice

The packs are in voices/.

Usage

pip install -e . # kokoro==0.9.4, praat-parselmouth, safetensors, scipy, soundfile

Enrolling a voice:

python -m inno_ref.enroll my_ref.wav am_me # prefix by accent and gender like the stock packs

The pitch-tracking ceiling is set automatically from the reference's harmonic spacing, so band-limited or archival sources land in the right octave without tuning. --fmax HZ overrides it if a voice still reads the wrong register.

import soundfile as sf
from kokoro import KPipeline

pipe = KPipeline(lang_code="a")
wav = next(pipe("Hello from a tuned voice.", voice="voices/me.pt")).audio
sf.write("out.wav", wav.cpu().numpy(), 24000)

Architecture

Enrollment embeds an input audio sample via the baked speaker encoder and maps through model.safetensors to build the encoder side; prosody aspects are measured to build the predictor half. Everything loads from model.safetensors, no extra downloads:

• speaker encoder: a 6.6M-param WeSpeaker ResNet34 distilled to output UniSpeech-SAT-sv embeddings
  • via a 380 MB teacher (not needed at enrollment); matches within 0.002 speaker cosine on hold-out dev-clean.
• style head: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
• prosody blend: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
  • Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
  • Packs more than 4 st from the reference's pitch are excluded.
  • Higher graded packs are weighted as preferred during selection.
• prosody head (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
  • fitted to deltas optimized through the frozen predictor. enroll(..., head=False) skips it.

Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.

Limits

• English trained (LibriTTS-R) currently
• Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
• Rythym is a best-effort match to the speaker.
  • Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
• Reference clip requirements:
  • 3 second minimum, up to a max of 30s
  • Reasonably free of audio artifacts
  • Single speaker

Training and Metrics

All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.

Trained via HF Jobs a10g-small; under $20 in GPU time:

v0.1.0 training notes

• Stage 1: mapped speaker embedding to timbre:
  • Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
  • [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
• Stage 2: the embedding alone missed texture and pitch; adjustments:
  • Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
  • [11k steps: dev cosine 0.934, DNSMOS 3.30]
• Stage 3: tuned the encoder side to match the blended predictors:
  • Predictor half fixed to the blended rows, only the decoder half trained.
  • [3k steps: dev cosine 0.932, DNSMOS 3.48]
• Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
  • WeSpeaker ResNet34 plus a projection, trained on 1 - cos(student, teacher) over LibriTTS-R crops with synthetic channel damage; the style head is untouched.
  • [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
• Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
  • Mixed real public-domain archival broadcast audio into the distillation crops.
  • [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]

v0.2.0 training notes

• Stage 1: the blend under-predicted pitch range and sentence-final falls:
  • Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
  • Targeted F0 mean, spread, IQR, pause fraction and speech time.
  • Ridge head from reference F0 mean + spread to the fitted delta.
  • [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
• Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
  • Added VoxPopuli and Emilia speakers to the distillation crops.
  • [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
• Stage 3: the timbre half had only seen LibriTTS speakers:
  • Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
  • [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]

Measured on the shipped weights through the stock KPipeline: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.

metric	v0.1 blend only	v0.2 with head
speaker cosine (WavLM-sv), samples / dev-clean	0.926 / 0.956	0.935 / 0.959
F0 mean, render minus reference	0.65 st mean abs	0.8 st mean abs
F0 spread, render minus reference	0.65 st mean abs	0.5 st mean abs
DNSMOS	3.50	3.50

The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.

Licenses and credits

Adapter weights and code: Apache-2.0, except the baked speaker encoder (enc.* in model.safetensors), which is CC BY-SA 3.0 (see below)

• Built on hexgrad/Kokoro-82M (Apache-2.0, StyleTTS2 architecture)
• The speaker encoder is a WeSpeaker ResNet34 (code Apache-2.0)
  • initialised from the VoxCeleb-trained ResNet34-LM weights (CC BY 4.0)
  • distilled from microsoft/unispeech-sat-base-plus-sv
    • (Microsoft UniSpeech, CC BY-SA 3.0 on GitHub); encoder weights under CC BY-SA 3.0
• Training data (no audio redistributed beyond the samples):
  • LibriTTS-R (CC BY 4.0), also the LibriTTS sample references
  • VoxPopuli (CC0)
  • Emilia-YODAS EN (CC BY 4.0)
• Sample references:
  • David Attenborough from White House interview, public domain, US federal work
  • Amelia Earhart from "Speech by Amelia Earhart", Library of Congress AFC 1986/022
  • Jane Goodall from BBC Woman's Hour excerpt, BBC, CC BY 3.0
  • Vincent Price public domain from "The House On Haunted Hill"

Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.