inference-optimization/qwen38-dspark-cooldown-fineweb

Qwen3.8-27B DSpark Draft - Stage-1b FineWeb Linear Cooldown (final, 2B tokens)

Final checkpoint of the linear LR cooldown (anneal) of the stage-1 DSpark draft: warm-started from the pretrain's V16 boundary (step 104,768 — inference-optimization/qwen38-dspark-pretrain-fineweb), then 10,173 steps (2.00B tokens) with LR descending linearly 1e-3 -> ~0 (100-step warmup), continuing the pretrain's FineWeb data stream exactly: trainer.skip_steps: 104,768 fast-skips the already-consumed batches of epoch 0 (the batch sequence is a pure function of dataset + split + DP size + batch geometry), so no data was repeated or skipped. Pure on-distribution annealing — same corpus, same token-only objective, fresh Muon optimizer at peak LR.

Checkpoint

• cooldown step 10,173 / 10,173 (global stream position 114,941 of epoch 0)
• val: loss 2.0726, eal 1.7229, position_0_acc 0.3888, accept_rate 0.1559, full_acc 0.2734, accept_len 1.444
• vs the V16 warm start: loss -0.086, eal +0.105, p0 +0.032 — landed 0.045 below the constant-LR extrapolation floor (~2.118)
• val trajectory over the cooldown (same multilingual val tail as the pretrain's V1-V16 series; all metrics monotone at every boundary):

	V16 start	step 840	3,296	5,752	8,208	final
val loss	2.1582	2.141	2.111	2.087	2.075	2.0726
eal	1.6181	1.637	1.675	1.703	1.721	1.7229
position_0_acc	0.3564	0.362	0.374	0.383	0.388	0.3888

• optimizer / scheduler / training_state intentionally excluded — this is a weights-only artifact

Model

• DSpark draft: 5 transformer layers, hidden 5120, block_size 8, sample_from_anchor, ~0.46B trainable params (identical architecture to the pretrain repo; warm start via from_pretrained)
• token-only conditioning: target_layer_ids: [0] (fc input [T, 5120] over frozen verifier embeddings; no verifier hidden states)
• heads included in weights: markov_head (within-block Markov conditioning) + confidence_head (per-position acceptance prob)
• verifier: Qwen/Qwen3.8-27B (frozen, NOT included - pull from the hub; verifier-owned embed_tokens/lm_head are reconstructed from it)
• loads via speculators from_pretrained (config.json + model.safetensors; config.py records the model class, val_metrics.json the final validation metrics)

Code / environment

• speculators @ 63fe7df755e2b3ca0ccce1f0b842dd5ee674fdc8 (fork of vllm-project/speculators, branch qwen38-27b-dspark-pretraining; this commit adds trainer.skip_steps — the data-stream continuation used here)
• torch 2.13.0, transformers 5.16.1
• train_command.txt + run.yaml + speculators.patch record the exact launch; configs/cooldown-fineweb.yaml is the source config (including the resume-the-pretrain recipe in its header)

Training recipe

• warm start: V16 snapshot via from_pretrained; fresh Muon, lr 1e-3, 100-step warmup (softens the fresh-optimizer transient on the small confidence head), linear descent to ~0 over scheduler_total_tokens 2e9 (= 10,173 steps at 196,608 global tokens/step)
• data: identical to the pretrain (same dataset + split, 24,576-token packing, 3,072 anchors x block 8, 8-way FSDP, seed 42) with skip_steps 104,768 -> trains stream positions 104,769-114,941
• loss: ce_token (hard-label CE against verifier greedy tokens), EOS <|im_end|>

Metric definitions

• eal: greedy expected accepted length per 8-token block incl. the verifier's +1 bonus token (run-based, preserves within-block correlation; ceiling 9.0)
• accept_rate/accept_len: analytical rejection-sampling quantities (1 - TV overlap)

Lineage / next steps

1. fc zero-expansion: fc.weight [5120,5120] -> [5120,46080], aux_hidden_state_layer_ids [0] -> [0,4,12,20,28,36,44,52,60]
2. stage-2 hidden-state distillation (fc over verifier hidden states) — the intended use of this checkpoint