OpenEuroLLM nucleus experiments
Intermediate Hugging Face checkpoints from OpenEuroLLM's hybrid-architecture ("nucleus") pre-training experiments on MareNostrum 5. These are research checkpoints, not a release: architectures, training schedules and export formats change between runs.
Branches
Every checkpoint is its own branch: <run>_iter_<7-digit step>[_<variant>], for example
ambitious_v4_2ph_hd256_fp8_iter_0024000 or q35c_gqa_iter_0008000. main holds only this card.
Variants: _probe and _hfNNN are additional exports of the same training checkpoint, made with a
different version of the export code.
Tokens trained = step x tokens per step (below). The ambitious_v4_2ph* and q35c_gqa runs train with
<bos> at the start of each document, and their tokenizers add it.
Runs
| Run (branch prefix) | Architecture | Parameters (total / active per token) | Tokens per step |
|---|---|---|---|
ambitious_v4_2ph_fp8 |
oellm_ambitious: attention, Kimi delta attention and MoE blocks (360 experts, 5 active), engram n-gram tables, mHC residual; head dim 128 |
163.5B / 3.3B | 67.1M |
ambitious_v4_2ph_hd256_fp8 |
the same run continued from step 9815 with attention head dim 256 | 163.6B / 3.4B | 67.1M |
ambitious_v3, ambitious_v4_ldm_fp8, ambitious_e216_eng40, ambitious_ldm_e216, ckdab_* |
earlier or branched variants of the same family | varies (91B to 164B) | 67.1M |
q35c_gqa, q35_conservative_run2 |
oellm_q35: Qwen3.5-style hybrid of full and linear attention, MoE (256 experts, 8 active) |
about 35B / 3.5B | 16.8M |
t5_s3_* |
small oellm_ambitious models for sparse-attention ablations, 1,500 steps |
about 12.9B / 1.2B | 1.05M |
About half of the oellm_ambitious parameters are engram n-gram tables (about 80 GB in bf16), which
are looked up, not multiplied, so they add little compute per token.
FP8 experts
In the *_fp8 runs, training gathers the routed-expert weights in FP8 (E4M3, one fp32 scale per
1 x 128 tile) for every forward pass, while the master weights stay in bf16. The exports apply the same
quantization to the master weights, so the routed experts are stored as FP8 bytes plus fp32 scale
tensors, exactly the values the training forward pass computed with. All other weights are bf16. The
q35* runs are bf16 throughout.
Loading
Each branch carries its own modeling code:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, revision = "openeurollm/nucleus-experiments", "q35c_gqa_iter_0008000"
model = AutoModelForCausalLM.from_pretrained(repo, revision=revision, trust_remote_code=True, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
The linear-attention layers need flash-linear-attention.
Branches may include export_validation/: the loss of the exported model on 64 held-out documents
(score.json) and on a subset where the training framework's own loss is known (paired.json).