OpenEuroLLM Prelude 9B (WIP Research Artifact)

This repository contains intermediate training checkpoints for OpenEuroLLM Prelude, an experimental 9B dense base model currently training on Leonardo (CINECA). See the OpenEuroLLM progress tracker for live status across all OpenEuroLLM training runs. It is being trained on 10T tokens (~80% English + code + math, ~20% multilingual across the 36 initial OpenEuroLLM target languages, represented as 41 language/script codes in the data mix, since some languages are split into macrolanguage + specific-variant codes, e.g. Latvian/Standard Latvian/Latgalian, Estonian/Standard Estonian, Albanian/Tosk Albanian, Norwegian Bokmål/Nynorsk). Full dataset composition, sources, and the per-language token breakdown are documented in the training-data-collection repo.

Checkpoints are published as per-iteration branches. Each branch (e.g. iter_0124800) is a complete, self-contained HuggingFace checkpoint: config.json, sharded model-*.safetensors (5GB shards), tokenizer files, and a validation.json with example generations (see Sanity-check generations below). main itself is intentionally empty and reserved for the final checkpoint.

Note on training phase: these are checkpoints from the "stable" (constant max-LR) phase of the WSD schedule and are not cooled down. Only a checkpoint at/after iteration ~953,600 would reflect the cooldown phase; everything currently published predates that point.

Model Details

Dense, untied input/output embeddings, closely follows the Qwen3 architecture.

Layers	36
Hidden size	4096
FFN hidden size	12288 (dense)
Attention heads	32
KV groups (GQA)	8
Head dim (kv_channels)	128
Position embedding	RoPE, base 100000
Normalization	RMSNorm, eps 1e-5, with QK-norm
Activation	SwiGLU
Sequence length	4096
Vocab size	262,145 (256k tokenizer, padded to 262,272 for TP efficiency)
Dtype	bf16

num tensors        : 399
dtypes             : ['BF16']
embed_tokens params: 1,074,266,112
lm_head params     : 1,074,266,112   (untied from embeddings)
transformer body   : 6,946,075,648
TOTAL params       : 9,094,607,872   (~9.095 B)
weight bytes (hdr) : 18,189,215,744  (~18.19 GB)

• Developed by: OpenEuroLLM
• Model type: dense, autoregressive decoder-only Transformer
• Languages: 36 initial OpenEuroLLM target languages (41 language/script codes in the data mix, see note above); per-language token breakdown
• License: Apache 2.0
• Training repository: oellm-autoexp, baby_9b_dense experiment
• Architecture config: qwen3_dense_9b.yaml

How to Get Started

Load a specific training checkpoint via the revision argument (branch name = Megatron iteration, e.g. iter_0124800):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "openeurollm/prelude"
revision = "iter_0124800"

model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)

prompts = [
    "Hola, ¿qué tal?",
    "Bonjour, comment ça va ?",
    "Guten Tag, wie geht es dir?",
    "Ciao, come stai?",
    "Hei, mitä kuuluu?",
]
generate_kwargs = dict(max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.3, no_repeat_ngram_size=3)

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, **generate_kwargs)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

See the repository's branch list for all available checkpoints.

Sanity-check generations

Every checkpoint's branch includes a validation.json with greedy-decoded completions for a fixed battery of prompts, generated at conversion time as a quick "does this checkpoint produce coherent text" spot-check:

• 4 canonical incomplete-sentence prompts, covering general knowledge ("The capital of France is..."), common sense ("The opposite of hot is..."), science ("Water boils at a temperature of..."), and creative/narrative ("Once upon a time...")
• Each of the 4 is localized into all 36 target languages (144 generations total), using each language's actual native conventions where relevant (e.g. its own fairy-tale opening phrase for the narrative prompt) rather than literal word-for-word translation. English is simply one of the 36 rows, not a separate set

This is a lightweight smoke test, not a benchmark. See Evaluation for actual downstream results.

Training Details

Global batch size	2048
Micro batch size	4
Sequence length	4096
Tokens / iteration	~8.39M
Total training iterations	1,192,093 (~10T tokens)
LR schedule	WSD (warmup-stable-decay), linear decay
Peak LR	3.5e-4
Warmup iterations	4,000
Cooldown iterations	238,476 (final 20% of training)
Optimizer	Adam (β1=0.9, β2=0.95, eps=1e-8), distributed optimizer
Weight decay	0.1
Gradient clipping	1.0
Parallelism	TP=4, PP=1, CP=1, sequence parallel
Checkpoint format	Megatron torch_dist, saved every 2,400 iterations
Hardware	256 nodes / 1024 GPUs, Leonardo (CINECA)

Training logs (loss curves, throughput, etc.) are tracked on Weights & Biases.

Tokens trained at a given checkpoint

Each iteration consumes exactly one global batch, so the number of tokens seen at a given branch's iteration can be computed directly from its name:

tokens(iteration) = iteration × global_batch_size × sequence_length
                   = iteration × 2048 × 4096
                   = iteration × 8,388,608

For example, iter_0124800 corresponds to 124,800 × 8,388,608 ≈ 1.05T tokens.

Evaluation

Downstream evaluation is ongoing, see this issue.

Uses

These are intermediate, non-cooled-down research checkpoints, intended for studying training dynamics (loss/capability progression, multilingual acquisition, etc.), not for production or downstream deployment. No safety alignment or instruction tuning has been applied.