domyn/Domyn-Edge-Base-checkpoints

Domyn Edge Base Checkpoints

Phased release. Domyn Edge is a rolling release in three bundles. The first, released on October 20, 2026, includes these checkpoints, the base model and domyn/Domyn-Edge-SFT, a supervised fine-tuned version that chats and calls tools. Further post-training checkpoints follow, up to the fully post-trained model in the final bundle. See Open resources on the base model card for everything released so far.

This repository holds the intermediate pre-training checkpoints of Domyn Edge Base, one per branch, so that researchers can study how capabilities form over 13 trillion tokens. Every branch is a pre-trained checkpoint with no instruction tuning and no alignment.

The final checkpoint is not here. It lives in domyn/Domyn-Edge-Base, together with the documentation of the architecture, usage, data and evaluation. This card covers only the checkpoint series.

Table of contents

1. What this repository contains
2. How to load a checkpoint
3. What changes across training
4. Limitations
5. License
6. Citation

What this repository contains

One branch per checkpoint. Branches are named by stage and by tokens seen, not by step. Through pre-training we release a checkpoint every trillion tokens, plus the end of stage 1 at 8.5T; every later stage releases its endpoint.

Branch	Phase	Tokens	Training context	Config max context
stage1-pretraining-1T … -8T	pre-training	one every 1T	8k	8k
stage1-pretraining-8.5T	pre-training, end of stage 1	8.5T	8k	8k
stage2-specialized-10.1T	specialized pre-training	10.1T	16k	16k
stage3-ctx-ext-10.3T	first context extension	10.3T	64k	64k
stage4-midtraining-12.3T	mid-training	12.3T	32k	64k
stage5-ctx-ext-12.7T	second context extension	12.7T	1M	1M
stage6-annealing-13T	annealing, end of 64k step	13T	64k	256k

Fourteen branches in total. The final checkpoint, at 13.03T tokens after the 512k annealing step, is the model published at domyn/Domyn-Edge-Base.

The stage 1 branches use a 115,269-token vocabulary; the three fill-in-the-middle tokens were added at stage 2, giving the 115,272 tokens of every later branch and of the final model.

What each stage adds is described on the base model card, together with the 8.44T-token stage 1 mix that every branch was trained on. The mixes for stages 2 to 6 ship in a later bundle.

How to load a checkpoint

With the recommended environment installed, pass a branch name as revision. main holds only this card, so revision is required:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "domyn/Domyn-Edge-Base-checkpoints"
revision = "stage4-midtraining-12.3T"

tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    revision=revision,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

Everything else is the same as for the final model. The recommended environment, the generation example and the serving instructions are on the base model card, and every caveat there applies to every branch here, including that stock vLLM does not yet support this architecture.

What changes across training

The clearest difference between branches is usable context, measured with RULER. Branches before stage3-ctx-ext-10.3T are short-context models whose scores drop beyond 16k. The first extension lifts every length (64k from 20.2 to 64.0), mid-training on 32k sequences gives back part of the longest lengths (128k from 46.6 to 28.1), and the second extension and annealing recover them, up to the final model's 74.5 at 256k.

Limitations

Early branches are short-context models. Every branch before stage3-ctx-ext-10.3T has a config limited to its training context and a usable context that ends at about 16k. Do not read a low long-context score on an early branch as a property of the architecture.

Later is not uniformly better. Scores do not move monotonically across the series. Long-context performance regresses across mid-training, and annealing improves every length up to 256k while 512k gets worse. Pick the branch that matches the question you are asking rather than assuming the last one dominates.

Every limitation of the final model applies. No branch has had alignment or safety tuning, and all share the prompt-injection and multilingual caveats listed on the base model card. For a model that chats and calls tools, see domyn/Domyn-Edge-SFT.

License

This model is released under Apache 2.0.

Citation

If you find this work valuable, please consider citing it:

@misc{domynedge2026,
  title  = {Domyn Edge},
  author = {Domyn},
  year   = {2026},
  note   = {Technical report, forthcoming in November 2026.},
}