Qwen3-Coder-Next-SWE-Prune-41B-A3B

This is an FP8, SWE-agent-calibrated expert-pruned checkpoint derived from Qwen/Qwen3-Coder-Next-FP8. It is intended for research on efficient agentic coding models.

What changed

• Routed experts per MoE layer: 512 → 256 (50% retained).
• Activated routed experts per token: 10 (unchanged).
• Router rows were remapped to the retained experts.
• Dense, attention, shared-expert, tokenizer, and chat-template weights are unchanged.
• No fine-tuning was performed after pruning.
• Experts were selected per layer using an importance score calibrated on successful, submitted SWE-bench agent trajectories.

The checkpoint uses the official Qwen FP8 block-quantized format and is directly loadable by recent vLLM/Transformers versions that support Qwen3NextForCausalLM.

Parameter count

Counts below were computed directly from every safetensors header. Quantization scale tensors are excluded from learned-parameter counts.

Metric	Exact count	Model-name convention
Total learned parameters	40,994,519,808	41B
Activated parameters per token	3,849,763,584	A3B
Routed-expert parameters	38,654,705,664	—
Always-active parameters	2,339,814,144	—

The A3B label follows the upstream Qwen convention. Under the same counting method, the original 80B checkpoint has 3.87B active parameters and is officially described as A3B.

Architecture

• Layers: 48
• Hidden size: 2,048
• Retained routed experts per layer: 256
• Activated routed experts per token: 10
• Shared experts per layer: 1
• Expert intermediate size: 512
• Native context length: 262,144
• Inference mode: non-thinking

Preliminary SWE-bench Verified evaluation

Using the unmodified official SWE-bench harness and the official mini-swe-agent prompt with 250 steps, temperature=1.0, top_p=0.95, and top_k=40:

• Fixed partial snapshot: 78/112 resolved (69.6%).
• Most recent incremental snapshot: 41/55 resolved (74.5%).

These are partial, non-random snapshots collected while the full 500-instance run was still in progress, so they must not be interpreted as the final full-dataset score.

vLLM example

vllm serve tomhu/Qwen3-Coder-Next-SWE-Prune-41B-A3B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend triton \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --generation-config vllm

Recommended sampling parameters:

{
  "temperature": 1.0,
  "top_p": 0.95,
  "top_k": 40,
  "chat_template_kwargs": {"enable_thinking": false}
}

For long multi-turn SWE-agent workloads, 32 concurrent agents were substantially more stable than 64 on four 48GB GPUs because the lower concurrency preserved the prefix cache.

Provenance and audit files

The repository includes the retained-expert mapping, per-layer importance scores, source revision metadata, tensor-level validation results, and shard checksums.

Upstream revision: Qwen/Qwen3-Coder-Next-FP8@da6e2ed27304dd39abadd9c82ef50e8de67bdd4c

License

This derivative checkpoint follows the upstream Apache-2.0 license. See the upstream model card for the original model details and attribution.

Citation

@techreport{qwen_qwen3_coder_next_tech_report,
  title  = {Qwen3-Coder-Next Technical Report},
  author = {{Qwen Team}},
  url    = {https://github.com/QwenLM/Qwen3-Coder/blob/main/qwen3_coder_next_tech_report.pdf}
}