config.json has two conflicting RoPE thetas – transformers 5 picks 100000 and generation repeats words
Both syvai/plapre-3 and syvai/plapre-nano-v2 have two different RoPE theta values in config.json:
"rope_parameters": { "rope_theta": 100000, "rope_type": "default" },
"rope_theta": 10000.0,
transformers 4.57.6 (the version recorded in the config) has no rope_parameters field. Its LlamaConfig reads only the top-level rope_theta, which defaults to 10000.0. transformers 5.x reads rope_parameters instead, so it runs the model with 100000. plapre-nano (v1) had only rope_theta: 100000.
My guess at how this happened: the top-level 10000.0 is exactly the 4.57 default. If the training config had only rope_parameters (100000, like SmolLM2), 4.57 would have ignored it, trained with 10000, and written both values on save.
The results agree with that. With 100000, generation loops and repeats words in longer sentences. With 10000 it is clean.
Repro (plapre-3, voice "tor", transformers 5.9, fp32, temperature 0.8, top_p 0.95, top_k 50, sentence-split; close to the model card's manual transformers code). Text:
"Det er en fæl historie!" sagde en høne, og det omme i den kant af byen, hvor historien ikke var passeret. "Det er en fæl historie i hønsehuset! Jeg tør ikke sove alene i nat! Det er godt, vi er mange på hækken!" – Og så fortalte hun, så at fjerene rejste sig på de andre høns...
Three runs per setting (same seeds on both sides), transcribed with Whisper large-v3-turbo:
| run | theta 100000 (transformers 5 default) | theta 10000 |
|---|---|---|
| 1 | 22.7 s: "…hvor historien ikke var byen, hvor historien ikke var passeret. Det er, det er…", "Og så fortalte hun, og så fortalte hun" | 19.4 s, clean |
| 2 | 22.7 s: "…og det er om i den kant af byen, og det er om i den kant af byen…", "så fortalte hun, så fortalte hun" | 19.8 s, clean |
| 3 | 17.6 s: "…vi er mange på hækken. Det er mange på hækken." (last sentence missing) | 20.2 s, clean |
The clips are longer with 100000, so the repetitions are in the audio and not Whisper hallucinations.
Suggested fix: make the two values agree. If 10000 is what v2/v3 were trained with, set rope_parameters.rope_theta to 10000 or remove rope_parameters.
Workaround for transformers 5 users:
cfg = AutoConfig.from_pretrained("syvai/plapre-3")
cfg.rope_parameters = {**cfg.rope_parameters, "rope_theta": 10000.0}
model = AutoModelForCausalLM.from_pretrained("syvai/plapre-3", config=cfg, dtype=torch.float32)
I haven't checked which value vLLM 0.15 resolves. If it reads the top-level rope_theta, that would explain why the plapre library isn't affected.