HyperPrune-Qwen2.5-0.5B-4to8

Qwen/Qwen2.5-0.5B pruned to 4:8 semi-structured sparsity with HyperPrune (Sun & Sakuma, Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware Hypernetwork, ICLR 2026, OpenReview).

This is a reproduction run produced at Elastix as part of the BLADE sparsity-method comparison. It is plain sparse bf16/fp16 safetensors and loads with stock transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("elastix-ai/HyperPrune-Qwen2.5-0.5B-4to8")
t = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

What differs from the paper's own recipe

	paper / repo default	this checkpoint
calibration corpus	allenai/c4	DKYoon/SlimPajama-6B, validation (BLADE's corpus)
pruned modules	see below	see below

Everything else — hypernet architecture, both training stages, all learning rates, step counts, temperature, prior, row selection — is HyperPrune's own shipped setting.

Configuration

{
  "model": {
    "name_or_path": "Qwen/Qwen2.5-0.5B"
  },
  "data": {
    "dataset_name": "slimpajama",
    "num_samples": 128,
    "seq_len": 2048,
    "seed": 42
  },
  "sparsity": {
    "n": 4,
    "m": 8
  },
  "hypernet": {
    "type": "mlp",
    "hidden_dim": 256,
    "emb_dim": 64,
    "use_layer_emb": false,
    "use_comp_emb": false,
    "use_hessian_diag": true
  },
  "training": {
    "sup_steps": 12000,
    "sup_lr": 0.001,
    "ft_lr": 0.0003,
    "ft_nsamples": 4,
    "rows_per_step": 400,
    "cascade_inner_steps": 300,
    "ft_mode": "cascade",
    "tau": 0.5,
    "prior_source": "sparsegpt",
    "wanda_residual_alpha": 2.0,
    "compensated_propagation": true,
    "use_weight_compensation": true,
    "train_on_compensated": true,
    "fixed_rows_count": 200,
    "fixed_rows_pos": "first",
    "dense_layers_list": []
  },
  "output": {
    "save_dir": "/home/ubuntu/hyperprune_work/outputs/hp-qwen25_05b-4to8",
    "wanda_dir": "/home/ubuntu/hyperprune_work/outputs/hp-qwen25_05b-4to8_ref",
    "preserve_wanda_dir": false
  }
}

Measured

metric	value
overall decoder sparsity (check_sparsity)	0.5149 (all 24 decoder layers, strict 4:8 verified)
WikiText-2 PPL (HyperPrune eval_ppl.py, seqlen 2048)	136.631
WikiText-2 word PPL (lm-eval-harness, BLADE's protocol)	282.87
training wall-clock	3.6 min
peak GPU during cascade FT	2.61 GB
GPU	1 x NVIDIA RTX PRO 6000 Blackwell (97 GB), CUDA 13.0, torch 2.13.0+cu130

Two things to know before comparing this number to the paper

1. Every decoder layer is pruned here. HyperPrune's own shipped configs set dense_layers_list: [0, 1], leaving 2 layers fully dense and yielding ~46.9 % sparsity rather than 50 %. This checkpoint prunes every layer, matching BLADE's four_eight_all spec, so it is a true 4:8 model in the modules BLADE prunes.

2. Only a few percent of this mask was chosen by the hypernet. The shipped recipe sets fixed_rows_count: 200, so the hypernet decides the mask for the first 200 output rows of each projection and every remaining row keeps the SparseGPT prior's mask verbatim. This is HyperPrune's own default, kept here deliberately because the brief was to change nothing but the calibration corpus.

The two perplexity rows are different quantities and are not comparable to each other. The first is token-level PPL over concatenated WikiText-2 at seqlen 2048 (the Wanda/SparseGPT convention). The second is lm-evaluation-harness word_perplexity at max_length=2048, which is BLADE's protocol — pinned empirically by reproducing BLADE's dense LLaMA-2-7B value of 9.19 (measured 9.1915).

4:8 is WORSE than 2:4 on this model — the opposite of the larger models

Qwen2.5-0.5B	2:4	4:8
WikiText-2 word PPL	149.94	282.87

4:8 is a strictly looser constraint than 2:4, and on the larger models it helps substantially (Llama-2-7B 21.65 -> 15.41; Llama-3.1-8B 22.27 -> 16.39). At 0.5 B it does the reverse. A plausible cause, not verified here: the supervised stage must learn a 70-way classification at 4:8 versus 6-way at 2:4, on the same 12,000-step budget, and this is the model with the least redundancy to absorb an under-fitted hypernet. Treat this checkpoint as a data point about the method's scaling, not as a usable model.

The checkpoint is structurally correct — strict 4:8 verified, zero groups with more than 4 survivors of 8 — and loads with stock transformers. Greedy generation is degenerate.

Provenance

Produced from HyperPrune commit 6d093d7 with a small set of documented patches (bias-dtype autocast, calibration loader, disk-peak reduction, and — for 4:8 checkpoints — the N:M generalization, which the reference implementation does not ship). See the reproduction report for the full diff.

Evaluation Results

KL Divergence

Dataset	Avg KL	Total KL	Tokens
wikitext2	2.239863	669253.1475	298,792
c4	2.187835	4112701.0703	1,879,804
slimpajama_calib	2.017394	16914864.3008	8,384,512

Downstream Accuracy

Task	acc, None	stderr, None
arc_challenge	0.1834	0.0113
arc_easy	0.3253	0.0096
hellaswag	0.2738	0.0045
mmlu	0.2293	0.0035
openbookqa	0.1240	0.0148
piqa	0.5626	0.0116
race	0.2488	0.0134
winogrande	0.5036	0.0141

Perplexity (2048-token windows, max_length=2048)

Dataset	Word PPL	Byte PPL
WikiText-2	282.9265	2.8739
C4 (en)	698.6640	2.9911

BLADE-Eval: lm-eval 0.4.10, torch 2.13.0+cu130, MLflow run