takuma104/pocket-tts-ja

Pocket TTS Japanese

Japanese models for Pocket TTS (Kyutai), fine-tuned from the English 24-layer model with the official training code. Code, training recipe and the full log of results: github.com/takuma104/pocket-tts-ja. How the data was built and the models were trained: technical report (English, 日本語). A sibling trained on CC BY data only, under CC BY 4.0 (commercial use with attribution), is takuma104/pocket-tts-ja-cc.

• Runs with the stock pocket-tts package (3.3.0 or later), on CPU, with no code changes.
• Takes plain Japanese text (kanji and kana as written). There is no G2P front end; the model reads the kanji itself.
• Zero-shot voice cloning from a few seconds of reference audio, as in the original.

Model	Config	FlowLM	Parameters (with the codec)	File
japanese	japanese.yaml	6 layers	110M	219 MB
japanese_24l	japanese_24l.yaml	24 layers	336M	1.3 GB

日本語の概要: Kyutai の Pocket TTS を日本語にファインチューニングしたモデルです。公式の pocket-tts （3.3.0 以降）にそのまま読み込め、CPU で動きます。入力は漢字かな交じりの文で、読みはモデルが決めます （G2P は使いません）。数秒の参照音声で声を指定します。ウェイトは CC BY-NC 4.0（非商用）です。 コーデック（Mimi）のデコーダーは日本語音声で調整したものです（元のデコーダーの初回版は下記のコミットで使えます）。 CC BY のデータだけで学習した版（CC BY 4.0、クレジットを表示すれば商用にも使えます）は takuma104/pocket-tts-ja-cc にあります。学習データの作り方と 学習の方法はテクニカルレポートにまとめています。

Usage

uvx pocket-tts generate --config hf://takuma104/pocket-tts-ja/japanese.yaml \
    --voice prompt.wav --text "こんにちは。今日はいい天気ですね。"

from pocket_tts import TTSModel

model = TTSModel.load_model(config="hf://takuma104/pocket-tts-ja/japanese.yaml")
voice = model.get_state_for_audio_prompt("prompt.wav")
audio = model.generate_audio(voice, "こんにちは。今日はいい天気ですね。")  # 24 kHz, torch.Tensor

• --voice is a recording of the voice to clone (a few seconds of clean speech). Without it the CLI clones its default voice (alba, an English speaker), which reads Japanese as well (JSUT Kana-CER 1.80%, against 2.15% with the JVS prompts). The pre-made voices cannot be selected by name (--voice alba): they are embeddings computed with the official weights.
• The config sets the sampling temperature to 0.3, the value every number below was measured with. Classifier-free guidance (scale 2.0) is distilled into the weights.
• Use japanese_24l.yaml for the 24-layer model.
• To keep a fixed version, add a commit hash: hf://takuma104/pocket-tts-ja/japanese.yaml@<commit>. The first release (same language models, original Mimi decoder) is @ca9b00950babd912554d7de895761f4db1782b5f.

Results

All numbers are from generated audio transcribed by an ASR model, with speakers of the JVS corpus (never trained on) as voice prompts, temperature 0.3, one LSD step.

Kanji reading: JKYB-Parakeet, all 13,536 sentences, official TTS protocol (kana-whisper). Accuracy is the share of sentences whose target word is read correctly.

Model	Prompts: 4 JVS speakers in rotation	Prompt: jvs001
japanese	94.75%	94.84%
japanese_24l	94.32%	94.49%
First release (original Mimi decoder): japanese	94.15%	94.73%
First release: japanese_24l	93.31%	94.06%
Original Mimi decoder: 24-layer teacher before the guidance was distilled (CFG 2.0, not released)	92.97%	93.68%

For reference, with the jvs001 prompt Irodori-TTS-v4.1-Small scores 94.0% and Irodori-TTS-v4.1-Small-Yomi, which synthesized part of the training data, 95.7%.

Other sets, japanese with the stock package on CPU (the released bfloat16 file) next to the GPU evaluation of the same checkpoint:

Set	Metric	GPU	Stock package, CPU
JKYB-mini (1,500 sentences)	accuracy	95.1%	95.3%
Aozora Bunko ruby (1,000 sentences)	target reading accuracy	71.3%
JSUT (1,000 sentences)	Kana-CER	2.28%	2.15%
JVS zero-shot (5 male + 5 female speakers, 10 sentences each)	Kana-CER	2.32%	1.70%
JVS zero-shot	similarity to the speaker's recording of the same sentence (ECAPA)	0.577	0.575
Short texts (172 texts of 2–8 tokens)	did not stop / extra speech	0 / 1	1 / 1

japanese_24l is at the same level on these sets (GPU: JKYB-mini 94.5%, Aozora Bunko 71.2%, JSUT 2.28%, JVS 2.13% / 0.586, short texts 0 / 1; stock package on CPU: JKYB-mini 94.7%, JSUT 2.20%, JVS 1.80% / 0.586, short texts 1 / 3). The 6-layer model reads at least as well as the 24-layer one, so japanese is the one to use unless the rarer failure to stop (see Limitations) matters.

Training

1. A Japanese SentencePiece model with 4,000 pieces replaces the English tokenizer; the text embedding is re-initialized. The codec (Mimi) stays frozen while the language model is trained.
2. The 24-layer teacher starts from the English 24-layer model and is trained for 700k steps (LSD objective, the voice prompt is another utterance of the same speaker).
3. Natural speech contains few of the rarer kanji readings, so sentences rich in them were synthesized with Irodori-TTS-v4.1-Small-Yomi in about 3,000 voices and kept only when an ASR model confirmed the reading. The words were chosen without the JKYB keys, and JKYB sentences and near duplicates were excluded.
4. Guidance at scale 2.0 is distilled into the weights: into a copy of the 24-layer model (japanese_24l) and into a 6-layer student initialized from the teacher's layers 0, 1, 2, 21, 22 and 23 (japanese).
5. The decoder of the codec (10M parameters) is fine-tuned on Japanese speech for 90k steps (multi-scale mel loss, MPD and MS-STFT discriminators, and a loss on the batch-averaged spectrum against the tonal artifacts locked to the decoder's upsampling strides), on clips of up to 8 seconds: 182 hours of the synthetic speech and 60 hours of high-bitrate Emilia. In the last 10k steps these losses see only the band below 4 kHz, and above it the output is held at the level of the original decoder: raised to the level of the recordings, that band was noise in voiced speech and audible as a hiss. The encoder is untouched, so latents and voice prompts are those of the original Mimi; only the decoder weights inside model.safetensors differ from the first release.

Training data	Hours	License
Emilia Japanese (Emilia and Emilia-YODAS), filtered and re-checked by ASR	1,531	CC BY-NC 4.0 / CC BY 4.0
The same clips with a changed gain	227
Synthetic speech from Irodori-TTS, readings confirmed by ASR	828	text: CC BY-SA and public domain

ReazonSpeech is not used. JVS and JSUT are used for evaluation only.

Limitations

• About 5 to 6% of the JKYB sentences are still misread, mostly rare readings and words from the appendix of the Joyo kanji table (jukujikun, 89% correct). Numbers, counters and Latin letters were not evaluated separately.
• In 0.1 to 0.3% of the JKYB sentences japanese does not end the utterance and keeps generating up to the length limit (japanese_24l: 0.02 to 0.09%).
• Reading accuracy and speaker similarity depend on the voice prompt.
• Real Japanese recordings still lose clarity when they pass through the codec (Kana-CER of JVS recordings: 3.3% as recorded, 4.7% through Mimi with the fine-tuned decoder; 5.4% with the original decoder). Above 4 kHz the output is as quiet as the original decoder's, 6 to 8 dB below the recordings in voiced speech.

Prohibited use

The prohibited uses of the original Pocket TTS apply: use of the model must comply with all applicable laws and regulations and must not result in, involve, or facilitate any illegal, harmful, deceptive, fraudulent, or unauthorized activity. Prohibited uses include, without limitation, voice impersonation or cloning without explicit and lawful consent; misinformation, disinformation, or deception (including fake news, fraudulent calls, or presenting generated content as genuine recordings of real people or events); and the generation of unlawful, harmful, libelous, abusive, harassing, discriminatory, hateful, or privacy-invasive content.

License and acknowledgements

Weights: CC BY-NC 4.0 (the training data includes Emilia). They derive from kyutai/pocket-tts (CC BY 4.0) by Kyutai (paper).

Thanks to Kyutai for Pocket TTS and its training code, to the authors of Emilia, Irodori-TTS (Aratako), JKYB-Parakeet and the original Joyo Kanji Yomi Benchmark (SB Intuitions), and of the JVS and JSUT corpora.