neutts-nano-polish

Polish finetune of neuphonic/neutts-nano, trained on 189 hours of Polish speech. 229 M parameters, zero-shot voice cloning from a 3–8 s reference clip, small enough to run on a phone.

• 229 M parameters (117 M active), bf16, plus NeuCodec. The base model's GGUF quantisation paths apply unchanged.
• Zero-shot cloning from a few seconds of reference audio.
• Requires a change to the phoneme string before inference. It is one regex, it ships in this repo, and the model gives no warning if it is missing — see Quick start.
• Word error rate 10.9 % on the 300-sentence set described below, against 20.2 % for the previous checkpoint of the same pipeline trained without the phoneme split. The numbers for four other Polish systems measured the same way are in the next section.

Word error rate against other Polish systems

300 Polish sentences, one generation each, no retries, every system reading from the same reference clip and judged by the same recogniser (nvidia/stt_pl_fastconformer_hybrid_large_pc).

system	params	on-device	clones a voice	WER
neutts-nano-polish (this model)	229 M	yes	yes	10.9 %
facebook/mms-tts-pol (VITS)	36 M	yes	no	12.2 %
piper pl_PL-darkman-medium (VITS)	16 M	yes	no	13.3 %
FireRedTTS3	2.1 B	no	yes	4.5 %
VoxCPM2	2.3 B	no	yes	5.0 %

Parameter counts are for the generative backbone only; the separate neural codec each autoregressive system decodes through is not included (NeuCodec here, redae for FireRedTTS3, an audio VAE for VoxCPM2). The VITS models have no separate codec. "On-device" means small enough to run on a phone at usable speed — the two large systems are marked no on that basis, not because they could not be quantised at all.

Two reference points from human speech, same recogniser, different corpora — a register anchor rather than a matched comparison: studio read speech (mc-speech) scores 3.7 %, and read speech by the speaker this model clones (pwr-azon) scores 13.7 %.

Of the three systems in the on-device size range, this one has the lowest word error rate and is the only one that can clone a voice; the two VITS models speak in a single fixed voice. The two billion-parameter systems score about twice as well at ten to fourteen times the size. They are in the table for scale, not as alternatives for the same deployment.

Naturalness and prosody are not measured anywhere in this card. There is no MOS and no listening study.

Quick start

1. Install

# espeak-ng 1.52+ is required -- older builds phonemise Polish badly
sudo apt install espeak-ng
pip install neutts

2. Apply the Polish phoneme split

Without this the model still produces fluent-sounding audio, with the wrong phonemes. There is no error and no warning. The training data went through this transform, so inference has to as well. The reason is below.

polish_phonemes.py ships in this repo:

from polish_phonemes import patch_neutts
patch_neutts()          # call before constructing NeuTTS

Outside Python, one line in the text frontend is the whole port:

phonemes = " ".join(re.sub(r"([a-zA-Z0-9])", r" \1", phonemes).split())

The whitespace collapse is not cosmetic: prefixing a space to an ASCII letter that already follows a word boundary produces a double space, which tokenises differently from what the model saw in training.

3. Generate

import soundfile as sf
from neutts import NeuTTS
from polish_phonemes import patch_neutts

patch_neutts()

tts = NeuTTS(
    backbone_repo="shefowl/neutts-nano-polish",
    backbone_device="cuda",
    codec_repo="neuphonic/neucodec",
    codec_device="cuda",
    language="pl",
    seed=1234,
)

ref_codes = tts.encode_reference("reference.wav")
ref_text = ("nazwa prawdopodobnie bierze się stąd iż te niewielkie domy przypominały "
            "prawdziwe budy budowane dla psów")

wav = tts.infer("Dzień dobry, nazywam się Anna Kowalska.", ref_codes, ref_text)
sf.write("out.wav", wav, 24000)

reference.wav ships here and is the clip every number in this card was measured with. Any sample rate works, since the codec resamples to 16 kHz internally. Reference clips outside 3–8 s were not tested. samples/ holds five generations from that reference, unedited.

Evaluation

Method

One recogniser for every system and every number: nvidia/stt_pl_fastconformer_hybrid_large_pc (115 M, sherpa-onnx int8, CPU). A word error rate without its instrument named is not readable, and two rates from two recognisers are not comparable.

The sentence set is 300 lines fixed before any of these runs, in three registers, because one average hides the differences that matter to an application:

• prepared phrases (120) — the sentences an assistive-communication app has stored;
• short labels (90) — one to three words, the text behind a button;
• corpus sentences (90) — ordinary well-formed Polish prose.

Every system got the same sentences, the same 8 s reference clip, one generation per sentence and no retries. VoxCPM2 re-rolls bad generations by default; that was turned off, since this model's own number is single-shot and leaving it on would compare two different questions.

Word error rate by register

system	overall	prepared phrases	short labels	corpus prose
neutts-nano-polish	10.9	13.4	9.6	9.2
facebook/mms-tts-pol	12.2	13.3	17.4	7.3
piper pl_PL-darkman-medium	13.3	17.6	16.1	6.8
FireRedTTS3	4.5	5.8	5.3	2.4
VoxCPM2	5.0	6.6	6.6	2.2

The split inside the small class is the useful part of this table. The two VITS models are at their best on flowing prose (6.8–7.3) and at their worst on isolated short labels (16–17), where a one-word utterance gives a duration predictor very little to work with. This model is the other way round: 9.6 on short labels, 9.2 on prose. For book paragraphs a VITS model is a reasonable and much smaller choice; for buttons, names and prepared phrases the ordering reverses.

Voice cloning

Cosine between WavLM-base-plus-sv x-vectors, every system cloning the same speaker from the same 8 s clip. The scale is given because a bare cosine from this model family is not interpretable — different speakers sit around 0.81.

	cosine	of the way from chance to ceiling
ceiling — two different real recordings of the speaker	0.971	100 %
ceiling — a real recording round-tripped through NeuCodec	0.958	91 %
VoxCPM2	0.951	87 %
FireRedTTS3	0.944	83 %
neutts-nano-polish	0.892	50 %
floor — different speakers	0.814	0 %
facebook/mms-tts-pol, piper	fixed voice, cannot clone

The two larger systems land near the top of the scale; this model reaches half of it, which reads as recognisably the same speaker rather than indistinguishable from them. Part of the gap is structural — every NeuTTS voice passes through NeuCodec at 16 kHz, and the round-trip alone caps the score at 0.958 — but only part of it: the codec costs 0.013 and the remaining 0.066 is the cloning. This is the weakest number in this card.

Surnames

122 Polish surnames in 4 nominative frames, scored phonetically because Polish writes one sound several ways (Zaręba = Zaremba = /zarˈɛmba/), so a letter test fails correct readings.

system	params	read correctly
FireRedTTS3	2.1 B	85.0 %
VoxCPM2	2.3 B	80.3 %
facebook/mms-tts-pol	36 M	70.3 %
piper pl_PL-darkman-medium	16 M	69.5 %
neutts-nano-polish	229 M	67.6 %
this model before the phoneme split	229 M	33.4 %

On 122 names the three on-device systems are not distinguishable from one another — every pairwise difference has a bootstrap interval straddling zero — while the two larger systems are ahead of them by 13 and 17 points, which is resolved. Three names, chmielewski, kasprzak and marciniak, are read wrong by all five.

Two results that may transfer to other work: names never seen in training score within two points of names given up to 100 training clips each, so surnames are read by rule and adding them to a corpus does not help (more on that below); and no spelling feature predicts which names fail — letter count, phoneme count, longest consonant run, number of 3+ consonant clusters and vowel fraction all correlate below 1.5 σ.

For an application: two thirds of surnames come out right in at least 3 of the 4 frames and a third do not, with no way to tell which in advance, so an on-device path needs a non-speech fallback for names.

Generation speed

20 sentences each, warm-up excluded, recogniser not in the loop. Every system runs in its own default PyTorch path on one AMD GPU except piper, which is ONNX on CPU. None of these is an optimised deployment path and none of them is the target device, so read this as the relative cost of the parameter counts, not as a latency figure for any product.

system	seconds per sentence	vs real time
facebook/mms-tts-pol	0.07	55×
piper pl_PL-darkman-medium (CPU, ONNX)	0.07	43×
neutts-nano-polish	1.87	1.9×
VoxCPM2	2.83	1.3×
FireRedTTS3	3.36	1.1×

The two VITS models are two orders of magnitude cheaper: they emit a spectrogram in one pass, while everything below them generates speech tokens autoregressively. Among the autoregressive systems the ten-fold parameter difference costs only 1.5–1.8× in generation time here, far less than the parameter ratio, because at these sizes the codec and the sampling loop take a meaningful share of the work.

Stability and generation limits

300 sentences, single generation, no retries, measured on the released checkpoint:

• looping: 0.0 % — and no cases at all in 900 draws across three checkpoints of this pipeline;
• over-long output, past 2× the phoneme-derived expectation: 2.7 % (8 sentences of 300);
• duration / expected: median 1.25, p90 1.52, p99 2.44, max 3.71.

The over-long cases are stretched tails and trailing silence, not repeated text — in every one of the eight the recogniser heard the sentence once or heard less than was asked for, never more.

A useful limit for integrators: cap generated speech tokens at about 2.5× the expected duration. NeuCodec runs at 50 tokens/s and the expectation is roughly 0.0614 s × len(phoneme_string). That cap sits just above the 99th percentile and below the observed maximum, so it bounds the tail while leaving normal generations untouched, and it costs nothing at run time because it is a bound inside generation rather than a check after it — time-to-first-token is unaffected. Truncating is safe here precisely because looping is zero: what gets cut is a stretched tail, not a second pass over the text.

Why the phoneme split is required

NeuTTS phonemises text with espeak and feeds the IPA string to a Llama-3 tokenizer trained on English. Runs of plain ASCII inside that string merge into English word pieces:

word	phonemes	tokens seen by the base model
napisz	nˈapiʃ	[' n', 'ˈ', 'api', 'ʃ']
faktura	faktˈura	[' fakt', 'ˈ', 'ura']
michalski	mixˈalski	[' mix', 'ˈ', 'als', 'ki']
siedem	ɕˈɛdɛm	[' ', 'ɕ', 'ˈ', 'ɛ', 'd', 'ɛ', 'm']

Words written with Polish-specific IPA (ɛ ɔ ɕ ʃ ʒ ɲ ʲ ɨ ʑ) cannot merge and arrive one token per phoneme. Words that spell out in ASCII arrive as fragments of English words carrying an English pronunciation prior. That includes most Polish surnames, since -ski, -cki and -ska are pure ASCII.

Prefixing a space to every ASCII character blocks the merges. Across the training corpus this collapses 2227 distinct text tokens to 90, one per phoneme, at the cost of 34 % more text tokens — about 5 % of the whole prompt, because speech codes dominate it. Nothing about the tokenizer, the vocabulary or the architecture changes.

Measured as a difference-in-differences with the arms frozen before training, against a matched checkpoint of the same run without the split:

	words made of ASCII IPA	words with Polish IPA	gap
without the split	42.7 %	76.1 %	33.4 ± 5.1
with the split	73.0 %	83.6 %	10.6 ± 4.4

A 22.8 ± 6.8 point closure, 3.4 σ. On the same probe set, overall word error rate went 22.6 % to 12.8 % and the hardest target words 41.6 % to 69.0 %.

On the 300-sentence set used everywhere else in this card, the previous production checkpoint of the same pipeline, trained without the split, reads at 20.2 % against this model's 10.9 %:

	overall	prepared phrases	short labels	corpus prose
without the split	20.2	28.1	16.6	14.8
with the split	10.9	13.4	9.6	9.2

Surnames over the same change go 33.4 % to 67.6 %.

This should apply to other non-English NeuTTS finetunes, and to any TTS that tokenises IPA with an English subword vocabulary. Languages whose phonemisation is ASCII-heavy have the most to gain.

Training

base	neuphonic/neutts-nano (architecture and vocabulary unchanged)
data	92 442 clips, 189.2 h
schedule	22 600 steps ≈ 4 epochs, batch 2 × 8 accumulation, lr 3e-5 cosine, 3 % warmup, bf16
released checkpoint	step 15 000, chosen by a blind listening comparison, not by loss
hardware	one AMD GPU (ROCm), ~11 h

Every training item is a (reference, target) pair from the same speaker, formatted byte-identically to the inference prompt, with the loss masked over everything up to and including the reference codes. The upstream finetuning example trains text → codes with no reference at all, which is not the format inference uses and teaches the model to ignore the reference, so cloning degrades. A set of speakers is held out entirely so cloning can be measured on voices the finetune never saw. All sources are resampled to 16 kHz before the codec regardless of their original rate.

Data

source	hours	licence
Wolne Lektury	116.7	public-domain Polish literature read aloud
Common Voice (pl)	29.9	CC0-1.0
BIGOS v2	20.2	CC-BY-SA-4.0
pwr-azon speech	12.2	CC-BY-SA-4.0
distillation (FireRedTTS3)	8.0	Apache-2.0
mc-speech	2.2	CC0-1.0

The distillation portion is synthetic speech generated with FireRedTTS3 for words the corpus did not contain. reference.wav and samples/ use a pwr-azon speaker. See NOTICE for full attribution.

What did not work

Five runs were spent on hypotheses that measured out flat. They are listed because they may save someone else the same runs.

Clip dose does nothing. Adding recordings of a word does not make the model say it. Synthetic dose on surnames: 0 clips 32 %, 100 clips 31 %. Real recordings for rare words: r(clips gained, score change) = −0.02 ± 0.05 over 397 words. The gradient that suggests otherwise — words with 0 corpus clips score 33 %, words with 10–99 score 80 % — is co-occurrence, not cause. Frequent words are frequent because they are easy.

Word difficulty is a stable property of the word, reproducing at r = +0.80 between independent checkpoints, and it does not move with exposure. That is what pointed at the input representation instead of the data.

A longer schedule does nothing. 4 epochs scored the same as 1 (gap change −2.7 ± 6.0 pts, 0.4 σ). Training loss plateaued around step 6000 and spent the remaining 16 000 steps oscillating between 6.51 and 6.64. The checkpoint sweep picked the middle of the run, not its end.

Also null: changing the reference voice (DiD +5.6 ± 3.5), training bare single-word utterances in the format the buttons use, and frame gender.

Two silent corpus filters cost whole runs before they were found. The corpus builder kept a sentence only if it carried a frequent word still under quota, so rare vocabulary could never enter from real speech no matter how much was available. And the encoder dropped clips under 1.0 s while a bare word lands at 0.96 s, discarding most of the data one run existed to add.

Rank checkpoints by whether words are said, not by loss. The two agreed once in this project, on the run where nothing happened.

Two measurement artifacts produced false findings, both worth checking in another harness:

• The recogniser returns text, so scoring by letters fails correct Polish readings — ch/h, ę + stop → /ɛm/, cluster devoicing. Re-scoring the surname set phonetically moved it about 4 points and reversed a conclusion about the distillation teacher.
• A runaway rate that pools sentences with single-word labels is not meaningful: a bare label expects 0.18 s of phonemes while no clip is shorter than a second, so correct takes score 5× and trip any duration threshold. Pooled, this model looked several points worse than its predecessor; measured the same way on the same sentences, its over-long rate is 2.7 % against the predecessor's 6.0 %.

Intended use and limitations

Built for assistive and augmentative communication in Polish — turning typed text into speech for people who cannot use their own voice. It runs locally, which matters when the alternative is sending someone's private speech to a server.

Do not use it to synthesise a person's voice without that person's consent. The reference speaker here consented to this kind of use through the dataset's licence; an arbitrary voice clip does not come with that.

Known limits, all measured above: about a third of surnames are unreliable and there is no way to tell which in advance; naturalness was not measured; the model was trained and evaluated on prepared, mostly well-formed Polish and has not been tested on code-switching, numbers read as digits, abbreviations, or dialect.

Licence

NeuTTS Open License v1.0, inherited from the base model, which is gated on the Hub. See LICENCE in this repo. In particular:

• redistribution is permitted with the licence, change notices and attribution kept;
• commercial use is permitted below 5 M USD annual revenue; above it needs a separate licence from Neuphonic;
• the licence terminates automatically on breach.

Consulting the original requires your own access to neuphonic/neutts-nano.