Winnow
Collection
Variable-width MoE expert pruning (Winnow): pruned OLMoE, Qwen3.6-35B-A3B and poolside Laguna-S-2.1 checkpoints, unhealed and healed. • 15 items • Updated
winnow-olmoe-general-keep25-onpolicy
On-policy-polished variant of winnow-olmoe-general-keep25.
Requires trust_remote_code=True. 2.09B params, 405/1024 experts deleted.
Off-policy baseline scored GSM8K 0.210 / MATH500 0.082 / IFEval 0.438 / HumanEval 0.018 / MBPP 0.112.
Warm-started from off-policy step150, then 150 on-policy steps: reverse KL on student rollouts, lr 1e-5, AdamW8bit, no reference anchor, gold CE rail 0.25 decaying 0.005/step, 256 prompts/step, group size 4, 2048 max new tokens, vLLM refreshed every step, seed 1223.
Eval table pending; see the math counterpart for the same recipe's measured effect (+9.6 GSM8K, p=1.8e-11).
- Downloads last month
- 19
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for hbfreed/winnow-olmoe-general-keep25-onpolicy
Base model
allenai/OLMoE-1B-7B-0125 Finetuned
allenai/OLMoE-1B-7B-0125-SFT Finetuned
allenai/OLMoE-1B-7B-0125-DPO Finetuned
allenai/OLMoE-1B-7B-0125-Instruct