HyperPrune-Llama-2-7B-4to8

meta-llama/Llama-2-7b-hf pruned to 2:4 semi-structured sparsity with HyperPrune (Sun & Sakuma, Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware Hypernetwork, ICLR 2026, OpenReview).

This is a reproduction run produced at Elastix as part of the BLADE sparsity-method comparison. It is plain sparse bf16/fp16 safetensors and loads with stock transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("elastix-ai/HyperPrune-Llama-2-7B-4to8")
t = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

What differs from the paper's own recipe

	paper / repo default	this checkpoint
calibration corpus	allenai/c4	DKYoon/SlimPajama-6B, validation (BLADE's corpus)
pruned modules	see below	see below

Everything else — hypernet architecture, both training stages, all learning rates, step counts, temperature, prior, row selection — is HyperPrune's own shipped setting.

Configuration

model:
  name_or_path: meta-llama/Llama-2-7b-hf
data:
  dataset_name: slimpajama   # DKYoon/SlimPajama-6B, validation split (BLADE's corpus)
  num_samples: 128
  seq_len: 2048
  seed: 42
hypernet:
  type: mlp
  hidden_dim: 256
  emb_dim: 64
  use_layer_emb: false
  use_comp_emb: false
  use_hessian_diag: true
training:
  sup_steps: 12000
  sup_lr: 1.0e-3
  ft_lr: 3.0e-4
  ft_nsamples: 4
  rows_per_step: 400
  cascade_inner_steps: 300
  ft_mode: cascade
  tau: 0.5
  prior_source: sparsegpt
  wanda_residual_alpha: 2.0
  compensated_propagation: true
  use_weight_compensation: true
  train_on_compensated: true
  fixed_rows_count: 200
  fixed_rows_pos: first
  dense_layers_list: []      # BLADE prunes every layer; the paper keeps [0,1] dense

Measured

metric	value
overall decoder sparsity (check_sparsity)	0.5005 (all 32 decoder layers pruned)
WikiText-2 PPL (HyperPrune eval_ppl.py, seqlen 2048)	10.64
WikiText-2 word PPL (lm-eval-harness, BLADE's protocol)	21.65 (dense reference under the same protocol: 9.19)
training wall-clock	22.1 min
peak GPU during cascade FT	8.82 GB
GPU	1 x NVIDIA RTX PRO 6000 Blackwell (97 GB), CUDA 13.0, torch 2.13.0+cu130

Two things to know before comparing this number to the paper

1. Every layer is pruned here. HyperPrune's own configs/llama2_7b.yaml sets dense_layers_list: [0, 1], which leaves 2 of 32 decoder layers fully dense and yields 46.9 % sparsity, not 50 %. This checkpoint prunes all 32 layers to match BLADE's two_four_all spec, so it is a true 2:4 model. That alone makes it a harder setting than the paper's 8.77.

2. Only ~3.5 % of this mask was chosen by the hypernet. The shipped recipe sets fixed_rows_count: 200, so the hypernet decides the mask for the first 200 output rows of each projection and every remaining row keeps the SparseGPT prior's mask verbatim. Weighted by parameter count on this architecture that is 3.52 % of weights. This is HyperPrune's own default, kept here deliberately because the instruction was to change nothing but the calibration corpus.

The two perplexity columns are different quantities and are not comparable to each other: the first is token-level PPL over concatenated WikiText-2 at seqlen 2048 (the Wanda/SparseGPT convention, dense LLaMA-2-7B = 5.47); the second is lm-evaluation-harness word_perplexity at max_length=2048 (dense = 9.19).

Provenance

Produced from HyperPrune commit 6d093d7 with a small set of documented patches (bias-dtype autocast, calibration loader, disk-peak reduction, and — for 4:8 checkpoints — the N:M generalization, which the reference implementation does not ship). See the reproduction report for the full diff.

Evaluation Results

KL Divergence

Dataset	Avg KL	Total KL	Tokens
wikitext2	0.393021	132985.4551	338,367
c4	0.402643	879896.6379	2,185,302
slimpajama_calib	0.363232	3045524.0394	8,384,512

Downstream Accuracy

Task	acc, None	stderr, None
arc_challenge	0.3592	0.0140
arc_easy	0.6785	0.0096
hellaswag	0.4745	0.0050
mmlu	0.2711	0.0037
openbookqa	0.3020	0.0206
piqa	0.7345	0.0103
race	0.3904	0.0151
winogrande	0.6677	0.0132

Perplexity (2048-token windows, max_length=2048)

Dataset	Word PPL	Byte PPL
WikiText-2	15.4119	1.6678
C4 (en)	34.1826	1.8055

BLADE-Eval: lm-eval 0.4.10, torch 2.13.0+cu130, MLflow run