Synthetic Persona Pretraining: Alignment from Token Zero
Paper • 2608.13482 • Published • 1
Vanilla control: the same documents, the same stream, no reflections.
Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).
| Parameter | Value |
|---|---|
| Parameters | 88,099,584 (~88.1M), tied embeddings |
| hidden / intermediate | 768 / 2048 (SwiGLU) |
| Layers | 12, GQA 12 query / 4 key-value heads |
| Context | 1,024 tokens, RoPE theta 10,000 |
| Vocabulary | 16,384 byte-level BPE (Romanian), <assistant> special token |
| Metric | Value | vs. reference |
|---|---|---|
| Perplexity, held-out web (3,000 docs) | 14.78 | ref |
| Perplexity, held-out docs flagged by the safety classifier (206 docs) | 22.60 | ref |
| Perplexity, held-out docs not flagged (2,794 docs) | 14.28 | ref |
| Held-out 4-way cloze accuracy (1,000 items) | 88.4 % | ref |
| ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items | 27.5 % | ref |
| ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items | 38.2 % | ref |
| ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items | 19.5 % | ref |
| ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items | 45.2 % | ref |
| Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix | 39.2 % | |
| Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix | 47.9 % | |
| Adversarial-prefix probe: margin shift towards the stereotype (nats/token) | +0.150 |
Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro-base")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro-base", dtype=torch.float32).eval()
prompt = "În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))
Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.
@mastersthesis{habeanu2026sppro,
title = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
author = {Hăbeanu, Flavius-Ștefan},
school = {Faculty of Mathematics and Computer Science, University of Bucharest},
year = {2026},
url = {https://github.com/flaviussteff/spp-ro}
}