sdougbrown/North-Mini-Code-1.0-dspark

North Mini Code 1.0 DSpark

This is a research-preview auxiliary draft checkpoint for speculative decoding with CohereLabs/North-Mini-Code-1.0-w4a16. It is not a standalone language model and it cannot generate text without the North verifier/target model. It will likely also work with CohereLabs/North-Mini-Code-1.0-fp8 with slightly lower acceptance rates.

Training

This DSpark draft model was trained on 10,000 Magicoder prompts with on-policy supervision from North Mini Code.

Important Notes

• not a base model
• not a fine-tuning target for standalone inference
• not a correctness guarantee
• not byte-parity with target-only greedy decoding

Architecture

• speculator type: DSparkDraftModel
• draft layers: 5 Qwen3 sliding-window layers
• hidden size: 2048
• attention heads: 32
• key/value heads: 4
• head dim: 128
• MLP intermediate size: 768
• sliding window: 2048
• draft vocabulary: 32,000
• target vocabulary: 262,144
• block size: 8
• sample_from_anchor: true
• Markov head: vanilla rank 256
• confidence head: trained and included, but skipped and unused by the validated North vLLM runtime
• auxiliary target hidden-state taps: layers [2, 24, 46]
• tensors: 66
• serialized tensor elements: 808,577,793
• BF16 parameters: 808,283,649
• selected weight hash: 64fd7d60c1c5a91d030dae0b7300bf8593b536b0a5569f53db7209e4a5f5b7f6

Largest tensors include the target embedding matrix, the Markov head matrices, and lm_head.weight. The tensor inventory is recorded in tensor_info.json.

Training data and supervision

• prompt dataset: ise-uiuc/Magicoder-Evol-Instruct-110K
• license: Apache-2.0
• training rows: 10,000
• selected rows: 0-499, 600-5099, and 6100-11099
• locked holdout: rows 500-599, never trained
• supervision: on-policy responses and target hidden states generated once from CohereLabs/North-Mini-Code-1.0-w4a16
• response cap: 2,048 tokens
• target layers used for supervision: [2, 24, 46]
• later acceptance-mined continuation data: not included in this release checkpoint

Training metadata is recorded in training_metadata.json.

Matched GB10 benchmarks

These numbers come from the same matched serving contract on a GB10 (DGX Spark) machine running Marlin (which is suboptimal but predictable). Compare the rows within the same host and contract only. These absolute token/s values are not representative of maximum performance, simply a demonstration that the draft model works. :)

Contract	Target-only	DFlash K3	DFlash / target	DSpark K4	DSpark / target
C1	39.9129 tok/s	75.5106 tok/s	1.8919x	79.7341 tok/s	1.9977x
C2	78.9486 tok/s	121.5232 tok/s	1.5393x	127.6174 tok/s	1.6165x

Model	C1 EAL	C1 draft acceptance	C2 EAL	C2 draft acceptance
DFlash K3	2.2947	43.16%	2.2856	42.85%
DSpark K4	2.4800	37.00%	2.4779	36.95%

Expected acceptance length (EAL) is the mean emitted tokens per speculative verification step, including the target bonus token.

Additional result: DSpark depth and K4-vs-DFlash

Matched DSpark runs showed better throughput than DFlash on the same contract. This serving margin is not a correctness claim.

• C1: DSpark vs DFlash = 1.0559x
• C2: DSpark vs DFlash = 1.0501x

Depth diagnostics on the 10K checkpoint:

• EAL 0-512: 2.4064
• EAL 512-1024: 2.7832
• EAL 1024-2048: 2.7322

Those continuation segments restart from full prefill and are diagnostic only.

Default proposal depth

Default proposal depth is K4. That is the selected deployment setting in config.json and the configuration used for the release evidence. The validated K5 setting used the same weights but did not beat K4 end-to-end throughput, so K4 is where you'll see the best speed. K5 only changes the proposal-depth setting; it is not a separate checkpoint.

Runtime boundary

This checkpoint is served as the top-level model because config.json already embeds the verifier reference. Use it only with the companion North runtime, published at https://github.com/sdougbrown/north-mini-code-draft-runtime. Stock vLLM 0.25.1 is insufficient. The companion is a thin overlay on official vLLM 0.27.1, where DSpark support is present, carrying PRs #49819 and #50937 (Cohere2MoE auxiliary hidden states, and an expert-bias loading fix), published as ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937.

export DRAFT_MODEL="${DRAFT_MODEL:?set this to the downloaded DSpark directory or Hub model ID}"
export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve "${DRAFT_MODEL}" \
  --tensor-parallel-size 1 \
  --max-model-len 320000 \
  --tool-call-parser cohere_command4 \
  --tokenizer-mode cohere \
  --cohere-format cmd4 \
  --reasoning-config '{"reasoning_start_str":"<|START_THINKING|>","reasoning_end_str":"<|END_THINKING|>"}' \
  --enable-auto-tool-choice

Limitations

• 32K draft vocabulary, not the 262,144-token target vocabulary
• target-dependent: cannot run standalone
• quantized greedy outputs are not guaranteed to match target-only greedy byte-for-byte
• throughput and acceptance do not establish correctness or response quality
• absolute throughput is host-specific
• this checkpoint has not been served on an NVIDIA RTX 3090; compatibility and performance on that GPU are unknown

References

• North base model: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-w4a16
• Magicoder dataset: https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K
• Companion North runtime: https://github.com/sdougbrown/north-mini-code-draft-runtime
• Companion image: ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937
• Official Eagle card: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-eagle
• Speculators: pinned commit f7ec34182826bc89934ce710283421778022b74d, version 0.6.0

Files in this repo

• README.md
• config.json
• config.py
• model.safetensors
• tensor_info.json
• training_metadata.json
• SHA256SUMS
• LICENSE
• .gitattributes