spp-ro

Synthetic Persona Pretraining from token zero (SPP{T0}): 10% of the training documents carry an inline constitutional reflection.

Part of SPP-Ro (https://github.com/flaviussteff/spp-ro): a compute-matched, from-scratch replication in Romanian of Minder et al., Synthetic Persona Pretraining, EPFL-dlab, 2026 (arXiv:2608.13482), comparing Vanilla, Filtered and SPP pretraining on exactly the same document stream. Bachelor's thesis, Faculty of Mathematics and Computer Science, University of Bucharest (2026).

Training

  • Data. 1.5M Romanian documents (FineWeb-2 RO, Wikipedia RO, press), read in a fixed-seed order that is identical for every model of the family (≈4 passes over the corpus in 50,000 steps; documents longer than 1,023 tokens are truncated). 150,000 documents (10 %) have a reflection in the sidecar: 75,000 documents flagged by a safety classifier (score ≥ 0.12, chosen among those whose insertion point falls inside the 1,024-token window) plus 75,000 benign ones; reflections average 36 tokens (~0.68 % of the trained tokens).
  • Schedule. 50,000 steps from scratch (peak LR 1.5e-4, 500 warm-up steps, cosine to 0, weight decay 0.1); 64 packed windows of 1,024 positions per step (EOS-terminated documents packed first-fit, block-diagonal attention, per-document positions, 2.4 % padding); ≈3.28 billion token positions in this run (~66 h); bf16 autocast with fp32 master weights; single NVIDIA RTX 3060 12 GB.

SPP document mode. c1 ... ck <assistant> r1 ... rn ck+1 ...: the reflection (written by an LLM for this very document, grounded in a six-article Romanian value constitution) is inserted at a sentence boundary; the continuation cannot attend to the reflection (asymmetric attention mask) and its RoPE positions alias back to the insertion point, so the document itself is modelled exactly as in the Vanilla model. The <assistant> token carries no loss. At inference, <assistant> is the marker after which the model was trained to write a reflection (there is no post-training).

Architecture

Parameter Value
Parameters 88,099,584 (~88.1M), tied embeddings
hidden / intermediate 768 / 2048 (SwiGLU)
Layers 12, GQA 12 query / 4 key-value heads
Context 1,024 tokens, RoPE theta 10,000
Vocabulary 16,384 byte-level BPE (Romanian), <assistant> special token

Results

Metric Value vs. reference
Perplexity, held-out web (3,000 docs) 14.83 +0.05
Perplexity, held-out docs flagged by the safety classifier (206 docs) 22.72 +0.12
Perplexity, held-out docs not flagged (2,794 docs) 14.33 +0.05
Held-out 4-way cloze accuracy (1,000 items) 88.3 % McNemar p = 1.00
ConstitutionEval-Ro accuracy, letter, 4-rotation debiased (chance 25 %), 600 items 24.3 % p = 0.23, Holm p = 1.00
ConstitutionEval-Ro accuracy, PMI (chance 25 %), 600 items 33.5 % p = 0.019, Holm p = 0.43
ConstitutionEval-Ro accuracy, continuation (chance 25 %), 600 items 20.2 % p = 0.65, Holm p = 1.00
ConstitutionEval-Ro accuracy, gold vs each distractor (chance 50 %), 600 items 45.2 % p = 1.00, Holm p = 1.00
ConstitutionEval-Ro, letter, deliberative (<assistant> after the scenario) 21.0 %
Adversarial-prefix probe (240 items): stereotyped continuation preferred, neutral prefix 40.8 %
Adversarial-prefix probe: stereotyped continuation preferred, adversarial prefix 46.2 %
Adversarial-prefix probe: margin shift towards the stereotype (nats/token) +0.150

Reference model: flaviussteff/spp-ro-base.

  • Perplexity, paired per document: SPP-Ro has a higher loss than Base-Ro on the held-out documents (mean Δloss +0.0026 nats/token [95 % CI +0.0012, +0.0041], higher loss on 54 % of 3,000 documents, sign test p = 4.0e-5, Holm p = 2.8e-4).
  • Perplexity, paired per document: SPP-Ro and Base-Ro do not differ significantly on the flagged held-out documents (mean Δloss +0.0038 nats/token [95 % CI -0.0018, +0.0093], higher loss on 53 % of 206 documents, sign test p = 0.37, Holm p = 0.73).
  • Perplexity, paired per document: SPP-Ro has a higher loss than Base-Ro on the unflagged held-out documents (mean Δloss +0.0026 nats/token [95 % CI +0.0010, +0.0041], higher loss on 54 % of 2,794 documents, sign test p = 6.5e-5, Holm p = 3.9e-4).
  • ConstitutionEval-Ro: no difference from the reference survives Holm correction over the 24 comparisons of the family. Levels of the whole family: continuation: 19.5–22.5 %, 5 below, 2 not distinguishable from chance (chance 25 %); letter: 21.0–28.0 %, 1 below, 6 not distinguishable from chance (chance 25 %); pmi: 33.0–38.2 %, every model above chance (95 % CI entirely over 25 %); pairwise: 44.4–46.7 %, every model below chance (95 % CI entirely under 50 %).

Details, per-item predictions and LaTeX tables: evals/ and evals/RESULTS.md in the GitHub repository.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("flaviussteff/spp-ro")
model = AutoModelForCausalLM.from_pretrained("flaviussteff/spp-ro", dtype=torch.float32).eval()
prompt = "<assistant>În România, egalitatea de șanse între femei și bărbați"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=60, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=False))

Limitations

Small (88M), monolingual Romanian, pretraining only (no instruction tuning or post-training), single seed. Trained on web text: it reproduces the biases of its data. Not intended for production use.

Citation

@mastersthesis{habeanu2026sppro,
  title  = {Pre-antrenarea Modelelor Generative Românești cu Personalitate Sintetică: Aliniere Constituțională de la Token Zero},
  author = {Hăbeanu, Flavius-Ștefan},
  school = {Faculty of Mathematics and Computer Science, University of Bucharest},
  year   = {2026},
  url    = {https://github.com/flaviussteff/spp-ro}
}
Downloads last month
98
Safetensors
Model size
88.1M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for flaviussteff/spp-ro