xf15/ssd-dspark-qwen3-4b

dspark drafter for Qwen/Qwen3-4B (ssd-parallel run)

DSpark-family drafter (dspark), block size 7, 5 draft layers, target tap layers [1, 9, 17, 25, 33]. Trained on the open-perfectblend corpus regenerated by Qwen3-4B (1,339,815 cache-valid rows) with a cosine LR schedule spanning 10 epochs (max_train_steps 26160, 2616 steps per epoch). Checkpoints at epoch boundaries live under epoch_N/step_{N*2616}/; each sits at that step of the 10-epoch schedule.

Inference

Any epoch_N/step_S/config.json + model.safetensors pair loads like the DSpark paper's released drafters. Serve together with the target model Qwen/Qwen3-4B.

Resuming training

1. Rebuild the activation cache from the dataset repo xf15/ssd-perfectblend-qwen3-4b-regen with the DSpark cache builder (tokenizer/template/min_loss_tokens recorded in the cache manifest); verify the rebuild against shuffle_records/source_rows.npy — the kept-row count must equal 1,339,815.
2. Place the chosen epoch_N/step_S/ at <checkpoint_root>/deepspec/dspark_block7_qwen3_4b/step_S/ and create the symlink step_latest -> step_S.
3. Resume requires world size 8 and local batch size 1 (asserted by the per-rank training_state files) and torch 2.9.1 (sampler permutations are seeded torch.randperm; shuffle_records/meta.json records the version, epoch_*.npy let you verify reproduction).

Data order is deterministic: epoch e uses torch.randperm(1339815, seed 42+e) truncated to 2616*512 samples; shuffle_records/ holds the dumped orders for all 10 epochs.