mkd-hossain/Keural-Cortex-8B-64K-training-state

Keural Cortex 8B 64K — FSDP training state, step 795 (final)

This is not a loadable model. It is the final Phase 5 training state, published for archival and reproducibility. from_pretrained() will not work.

For the usable 64K model, see mkd-hossain/Keural-Cortex-8B-64K.

What is in here

PyTorch Distributed Checkpoint (DCP) format — the state needed to resume training, not to run inference:

File	Size	Contents
__0_0.distcp … __3_0.distcp	~12.3 GB each	one shard per FSDP rank: sharded model params + AdamW optimizer moments (exp_avg, exp_avg_sq)
.metadata	1.25 MB	DCP tensor-to-shard index
meta.json	~1 KB	step, token count, LR-scheduler state, full training config

It is ~46 GB rather than the 16 GB of the exported model because roughly 30 GB of it is optimizer state. That state is what would make training resumable — dropping it and restarting from exported weights would reset Adam's moment estimates.

Training position

Step	795 / 795 — final
Tokens (this phase)	2,500,853,760 (2.50B)
LR at this step	1.0e-06 (end of WSD decay)
Phase	5 — context extension 32K → 64K, complete
Started from	exports/full/step_0010000 of the 41B CPT

Resume constraints

Shard-count specific: written by 4 FSDP ranks and expects 4 ranks.

world_size: 4      # must match
micro_bsz: 1
grad_accum: 12
seq_len: 65536     # global batch 3,145,728 tokens/step
fused_linear_cross_entropy: true   # required at this sequence length

Resuming with a different world_size requires a DCP reshard. The full originating config is embedded in meta.json, including the data manifest path — resuming also requires that corpus, which is not part of this repo.

fused_linear_cross_entropy is not optional here: without it the run OOMs on a 37.09 GiB fp32 logits allocation at seq_len 65,536, and micro_bsz is already 1.

Why this is published

Phase 5 is complete, so there is nothing left to resume — this exists as an archival record of the final training state alongside the usable model, not as something with ongoing operational use.

Provenance

Continued pretraining of Qwen/Qwen3-8B-Base by MKD Co., Ltd. Apache 2.0, inherited from the base model. Derivative work of Qwen3-8B-Base by Alibaba Cloud.

Full audit record at github.com/MKD-CORP/Keural-Cortex-8B under CPT_details/.