mkleinegger/reapify

REAP-pruned MoE checkpoints

Expert-pruned checkpoints made with REAP (Router-weighted Expert Activation Pruning, CerebrasResearch/reap at commit 1970473), calibrated on the prompt mix of APEX-MoE (mkleinegger/apex). Every MoE layer keeps the routed experts with the highest REAP saliency: the mean, over the tokens routed to an expert, of the token's renormalized router weight times the norm of the expert's output. The other experts and their router rows are removed.

folder	base model	routed experts kept per layer
qwen3.6-35b-a3b-nvfp4-reapofficial-keep50	RedHatAI/Qwen3.6-35B-A3B-NVFP4 e94a930	128 of 256
qwen3.6-35b-a3b-nvfp4-reapofficial-keep25	same	64 of 256
qwen3.8-flash-next-fp8-reapofficial-keep50	Qwen/Qwen3.8-Flash-Next-FP8 236dfdf	256 of 512
qwen3.8-flash-next-fp8-reapofficial-keep25	same	128 of 512
deepseek-v4.1-flash-reapofficial-keep50	deepseek-ai/DeepSeek-V4.1-Flash 5254448	192 of 384
deepseek-v4.1-flash-reapofficial-keep25	same	96 of 384, padded to 128 (below)

Each checkpoint folder holds the pruned weights in the base model's own precision, its config and tokenizer files, and reap_selection.json: the kept experts per layer (original expert ids), the pruned share and the observations it was computed from. The base models' licenses apply; the upstream model cards and reports are not copied.

How they were made

• Calibration (calibration/calmix1024-seed0-reapstyle.jsonl): 1,022 rows in REAP's calibration format: Open-Platypus (256), OpenThoughts-114k (255) and evol-codealpaca-v1 (255) prompts with the dataset's own answer as a chat, FineWeb-Edu passages (256) as raw text. Prompts that contain a question of MATH-500, GPQA-Diamond, MMLU-Pro, HumanEval+ or MBPP+ (as a substring, or sharing a word 13-gram) were never drawn; 2 rows whose answer contained one were dropped.
• Saliency (observations/<model>/observations_1024_cosine-seed_42.pt, REAP's observer output): Qwen3.6 and Qwen3.8 with REAP's prune.py and the settings of its pruning script (metric reap, renormalized router weights, seed 42, 1,024 samples). DeepSeek-V4.1-Flash, which transformers cannot run: the same statistic from a teacher-forced vLLM replay of the calibration set (code/experiments-apex/observations_from_replay.py).
• Pruning: REAP's selection (per layer the lowest-saliency experts removed) applied to the original checkpoint by code/experiments-apex/apply_selection.py: kept experts renumbered 0..K-1 in id order, router rows sliced to them, the expert count in the config set to K, the MTP draft layers dropped, every other tensor unchanged.
• DeepSeek keep25 carries 32 extra experts that no token can reach (zero weights, zero router rows, router bias -1e4), for MoE kernels that take only some expert counts; it computes exactly what the 96-expert model does.

code/ holds our changes to REAP (reap.patch, a diff against commit 1970473) and the scripts that built the calibration file, collected the saliency and wrote these checkpoints (experiments-apex/); they carry the paths of the machine they ran on.

v2: saliency from the models' own responses (v2/)

The same REAP saliency and selection, computed on the final APEX-MoE calibration mix: each prompt with the model's own response, every token of prompt and response, nothing packed or truncated.

folder	base model	routed experts kept per layer
v2/qwen3.6-35b-a3b-nvfp4-keep50	RedHatAI/Qwen3.6-35B-A3B-NVFP4 e94a930	128 of 256
v2/qwen3.6-35b-a3b-nvfp4-keep25	same	64 of 256
v2/qwen3.8-flash-next-fp8-keep50	Qwen/Qwen3.8-Flash-Next-FP8 236dfdf	256 of 512
v2/qwen3.8-flash-next-fp8-keep25	same	128 of 512

• Calibration (v2/calibration/<model>-requests.jsonl: request id, set, prompt and response token counts; no text): 3,584 prompts in 9 sets, each answered by the model itself (temperature 1.0, top-p 0.95, top-k 20, thinking on except the last set; 32k new tokens, 64k for the long sets): calmix1024 (Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1 and FineWeb-Edu, 256 each), callong-ot384 (OpenThoughts-114k problems with the longest reference traces), callong-otcode384 (the same for its code problems), calagent-swe256 (SWE-smith trajectories), calagent-xlam256 (xLAM function calling), calown-wp256 (WritingPrompts), calown-personas256 (tulu-3-sft-personas-math), calown-cfcots384 (codeforces-cots problems with the longest traces), calotcode-nothink384 (OpenThoughts-114k code problems, thinking off). Prompts sharing a substring or word 13-gram with MATH-500, GPQA-Diamond, MMLU-Pro, HumanEval+ or MBPP+ were never drawn; no LiveCodeBench, LeetCode or AtCoder data.
• Saliency (v2/observations/<model>/observations.pt, REAP's observer layout; sources.json names the runs it was computed from): per layer and expert, the mean over the tokens routed to the expert of the top-k-renormalized router weight times the norm of the expert's output, from a teacher-forced vLLM pass of each request on its own (v2/code/observations_from_replay.py). Qwen3.6 from Qwen/Qwen3.6-35B-A3B (BF16, 995ad96): 69.8M tokens; Qwen3.8 from the FP8 checkpoint: 67.9M tokens.
• Pruning: as above (v2/code/apply_selection.py), onto the same base checkpoints.
• Against REAP's own observer (Qwen3.6): REAP's prune.py on the same requests, packed to 16,384 tokens (which keeps 27.4M of the 69.8M tokens), keeps 96.4% (50%) / 94.2% (25%) of the same experts. Our computation on exactly REAP's packed tokens keeps 99.4% / 99.0% of its experts. The difference comes from the tokens after the cut, not from the computation.

The v2 checkpoints have not been benchmarked here.