blue-yi — Yiddish zero-shot TTS (text-to-latent)

Flow-matching text-to-latent model for Yiddish at 44.1 kHz, from the blue multilingual TTS stack (WildSpoof arXiv:2512.17293 / SupertonicTTS arXiv:2503.23108). Voice identity comes entirely from a reference clip — there is no speaker table, so a new voice needs a recording, not a retrain.

Trained to step 817,000.

What is in this repo

file	what it is
text2latent_yi_step817000.pt	the model: vector field, text encoder, reference encoder, CFG null embeddings (plus optimizer/scheduler state, so training can be resumed)
duration_predictor.pt	total-duration predictor; sets the canvas length before sampling
stats_yiddish.pt	latent mean/std — see the warning below
tts.json	model dimensions; everything is constructed from this
voices/*.json	four preset speaker styles — see below
onnx/	full ONNX export; runs without PyTorch

ONNX (onnx/) — self-contained, no PyTorch needed

The whole stack is exported to ONNX and needs only onnxruntime + numpy + soundfile. Every graph was verified against its PyTorch original at export time: cosine similarity 1.000000, max absolute difference <= 1.2e-5, and the reference encoder was checked at two different sequence lengths to exercise its dynamic axis.

file
onnx/vector_estimator.onnx	132 MB — the flow-matching vector field
onnx/vocoder.onnx	101 MB — the AE decoder (see below)
onnx/text_encoder.onnx	27 MB
onnx/reference_encoder.onnx	15 MB — encode your own reference clip
onnx/duration_predictor.onnx	takes raw reference latents
onnx/duration_predictor_style.onnx	takes a style_dp from a voice file
onnx/uncond.npz	u_text / u_ref, the CFG null embeddings
onnx/stats.npz	mean, std, normalizer_scale
onnx/vocab.json	the 256-slot IPA vocabulary
onnx/manifest.json	source checkpoint hashes and sizes
onnx/example_onnx.py	working end-to-end synthesis, ~60 lines

python onnx/example_onnx.py onnx/ voices/libri_male_6209.json out.wav \
  "mit a pˈur jur ʦirˈik hut ɛr ɡəzˈuɡt az dus iz də bˈɛstə vejɡ."

vector_estimator.onnx takes the Euler step itself

The graph is not a plain velocity function. When total_step is fed it normalizes time internally as current_step / total_step and returns noisy + (1/total_step) * v, i.e. the already-updated latent. So:

• current_step is the raw step index (0, 1, 2, ...), not i/N
• do not apply x = x + v*dt on top of the output
• combine CFG on the stepped outputs: x = x_uncond + W * (x_cond - x_uncond)

Integrating a second time renders perfectly plausible-sounding audio about 2.7x too loud, peaking near 8.0. Nothing raises. example_onnx.py does it correctly.

The last ~20 ms of a render is vocoder ring-out

It is not quiet — it can peak several times above the speech level. Cut to the predicted duration minus ~20 ms, as the example does, and consider a de-clicker for isolated impulses. A raw render of one of the four voices below peaks at 4.4 without that trim.

The autoencoder

onnx/vocoder.onnx is the AE decoder, so this repo synthesizes end to end without the 736 MB PyTorch autoencoder. It is byte-identical to the decoder in BlueTTS2.5-onnx — 99 of 100 initializers match to 1e-6, the hundredth being head.act.weight reshaped from (1,) to (1,1) for PReLU. The autoencoder really is shared across the stack.

The PyTorch ae_latest.pt is still not published; it is only needed to resume AE training or to encode references in PyTorch. reference_encoder.onnx covers reference encoding for inference.

Voices

Four preset voices in voices/, the same four speakers as BlueTTS2.5-onnx, each a LibriTTS-R reader. Any 44.1 kHz recording works as a reference instead — these are just convenient starting points.

voice	LibriTTS-R speaker	median F0 of the render
voices/libri_female_1088.json	1088	165 Hz
voices/libri_female_6147.json	6147	184 Hz
voices/libri_male_6209.json	6209	115 Hz
voices/libri_male_8088.json	8088	110 Hz

Each file holds style_ttl (50 style tokens x 256) for the flow model and style_dp (8 x 16) for the duration predictor, plus the source clip it was encoded from.

These were re-encoded for this checkpoint, not copied from BlueTTS2.5. Style vectors are outputs of a specific reference encoder, and the two models do not share one — zero of BlueTTS2.5's 73 reference-encoder tensors match this checkpoint's. Reusing its voice files here would load without error and produce the wrong speaker. The same applies to any future checkpoint: re-export from the source audio rather than carrying voice JSONs forward.

Note the source clips are LibriTTS-R at 24 kHz, upsampled to 44.1 kHz on encoding, so they carry no content above 12 kHz. A native 44.1 kHz reference will clone a brighter voice.

python scripts/inference/run_pt_inference.py \
  --text "מיט א פאר יאר צוריק האט ער געזאגט אז דאס איז די בעסטע וועג." \
  --lang yi --style_json voices/libri_female_1088.json \
  --text2latent_ckpt text2latent_yi_step817000.pt \
  --ae_ckpt ae_latest.pt --dp_ckpt duration_predictor.pt \
  --stats stats_yiddish.pt --steps 8 --cfg 3.0 --out out.wav

Read this before using the weights

A checkpoint is only valid against the latent statistics it was normalized with. Pairing this model with different stats does not raise — tensor shapes stay valid and only the audio degrades, which reads as a model-quality problem rather than a configuration error. Use stats_yiddish.pt from this repo, and the autoencoder it was trained against.

Normalization is ((z - mean) / std) * 0.25, with the scale from ttl.normalizer.scale in tts.json. Sampling reverses both.

Latent conventions

The autoencoder emits [B, 24, T] at 86.13 Hz (44100/512). Those are folded by compress_latents(factor=6) to [B, 144, T/6], i.e. 14.35 Hz frames of 144 channels. The flow model and the duration predictor both operate on the compressed representation; decompress_latents inverts it before the decoder.

Text input

Yiddish text is Hebrew script and is converted to narrow IPA by the phonikud-yi engine, then mapped through a fixed 256-slot universal IPA vocabulary.

Feed raw text, not nikud. The engine resolves /a/ vs /o/, /p/ vs /f/ and Loshn-Koydesh vowels from its own tables; diacritizing first makes the labels worse, not better.

Unknown characters map silently to PAD — they do not raise. A misconfigured phonemizer trains and infers happily on padding. Validate coverage explicitly:

from data.text_vocab import CHAR_TO_ID, normalize_text
ipa = normalize_text(ipa, apply_hebrew_fixes=False)
assert not {c for c in ipa if c not in CHAR_TO_ID}

Inference

python scripts/inference/run_pt_inference.py \
  --text "מיט א פאר יאר צוריק האט ער געזאגט אז דאס איז די בעסטע וועג." \
  --lang yi \
  --ref_wav your_reference.wav \
  --text2latent_ckpt text2latent_yi_step817000.pt \
  --ae_ckpt ae_latest.pt \
  --dp_ckpt duration_predictor.pt \
  --stats stats_yiddish.pt \
  --steps 8 --cfg 3.0 \
  --out out.wav

--cfg 3.0 is classifier-free guidance; it costs a second vector-field evaluation per step, so --steps 8 --cfg 3.0 is 16 evaluations.

Two cheap sanity checks

Mixing components degrades silently, so verify before believing any audio:

• an encoded style's norm should sit near the checkpoint's u_ref norm (5.55 for this checkpoint)
• output RMS should be near a real recording's (~0.10) with peak below 1.0

A verified render from these exact weights gives peak 0.550, RMS 0.111.

Limitations

• The duration predictor emits a single total duration, not per-phoneme durations, and there is no monotonic alignment. The flow model receives a fixed canvas and must fit every phoneme into it; when the total runs short, phonemes are compressed and words can drop.
• Reference quality is the ceiling. A band-limited reference (some LibriVox material rolls off near 13 kHz) caps the output regardless of the model.