1pp-0.98b-v2-asst-base

One Persona Pretraining (1PP) scaling-ladder model: 0.98B parameters, pretraining condition rewritten conversations, assistant-turn loss, seed 28.

Part of ladder v2: four sizes (0.58B, 0.98B, 1.66B, 2.81B) × three conditions × three seeds (28, 42, 1234), each size trained from scratch for 20.3 assistant tokens per parameter on the same step-aligned data stream. Conditions differ only in the document text and the loss mask; seeds differ only in the parameter initialization (the batch order is the same). The 1.66B rung is the 1.7B model set of the main study (1pp-1.7b-*). Models are grouped in the 1pp collection.

Architecture

Llama-style decoder, 24 layers, hidden 1,536, FFN 6,144 (SwiGLU), attention heads / KV heads 12 / 4 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus <|pad|>; <|endoftext|> is the end-of-document token.

Pretraining

Data: the 1PP conversations rewritten from those documents; loss only on assistant turns. The first 18,779 steps of the 1PP step-aligned stream (39.38B token positions), which are the first steps of the main study's 31,777-step pass in the same order. Global batch 512 × 4,096 tokens, cross-document attention masking. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for the other parameters (LR 2.3e-3 for embeddings and the output layer, 3e-3 for norms); warmup 1,181 steps, constant, linear decay over the last 1,877 steps to 1/100; weight decay 0.1; init std 0.0103; bf16.

Validation loss (per token, 2,433 held-out documents, final checkpoint):

assistant text	user text	document text	this condition's training tokens
1.551	6.502	3.380	1.604 (reweighted_seg34)

The last column weights adversarial conversations at their training share (11.18%) for the conversation conditions.

Chat format

ChatML without a system turn (the models never saw one):

<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n

The bundled chat_template renders exactly this. Generation stops at <|im_end|> (id 2) or <|endoftext|> (id 0); both are listed in eos_token_id. This is a base model; the conversation conditions produce chat-formatted text, the raw baseline plain text.

Verification

The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:

set	HF loss	Megatron reference	abs. diff
val50m segments [3]	1.5495	1.5510	0.0015
raw_val50m segments [8]	3.3829	3.3799	0.0030

Links

• Training logs: wandb projects 1pp-training and 1pp-sft
• Research artifact from the 1PP project (EPFL DLAB); not a general-purpose assistant.