zhc12/dynrank-compressed-models

DynRank: Compressed Model Checkpoints

Hardened SVD-compressed LLM checkpoints from the DynRank paper: "DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"

Checkpoint Format

Each .pt file is a dictionary:

{
    "step": 1000,
    "ranks": [r_1, r_2, ..., r_224],  # per-sublayer integer ranks
    "state_dict": {...},               # model weights (CompressedLinear format)
    "val_loss": float
}

Weight keys use CompressedLinear format:

• model.layers.{i}.self_attn.q_proj.first.weight — B matrix (r × n)
• model.layers.{i}.self_attn.q_proj.second.weight — A matrix (d × r)

Loading

import torch
ckpt = torch.load("path/to/v8_hardened_step1000.pt", map_location="cpu")
ranks = ckpt["ranks"]        # list of 224 integers
state_dict = ckpt["state_dict"]

To use with SVD-LLM's CompressedLinear:

from src.model.replace import CompressedLinear
# See https://github.com/zhc1212/SVD-LLM for full loading code

Directory Structure

baselines/           # Rank profile JSONs (ASVD, Dobi-SVD proxy, SVD-LLM V2)
phase5a/             # Early TinyLlama logs (no model weights)
phase5b_llama2/      # Main experiments (LLaMA-2-7B, LLaMA-7B, Mistral-7B, LLaMA-2-13B)
phase5e_qwen3/       # Qwen3-8B experiments

phase5b_llama2 key models

Directory	Model	Method	Ratio	Notes
v9_r04	LLaMA-2-7B	DynRank	0.4	Primary result (seed=42)
static_r04	LLaMA-2-7B	Static uniform	0.4	Matched baseline
v9_r04_s3/s4_refill/s5	LLaMA-2-7B	DynRank	0.4	Multi-seed (43,44,47)
v9_r02/r08	LLaMA-2-7B	DynRank	0.2/0.8	Ratio sweep
llama7b_v9_r04	LLaMA-7B	DynRank	0.4	Cross-model
mistral_v9_r04	Mistral-7B	DynRank	0.4	Cross-model
llama2_13b_v9_r04	LLaMA-2-13B	DynRank	0.4	Scale test
baseline_dobi_r04	LLaMA-2-7B	Dobi-SVD profile	0.4	Baseline comparison
baseline_asvd_r04_v2	LLaMA-2-7B	ASVD-STRS profile	0.4	Baseline comparison
h2_oracle_llama2_r04	LLaMA-2-7B	Oracle static	0.4	H2 ablation
gsm8k_dynrank_r04	LLaMA-2-7B	DynRank (GSM8K calib)	0.4	Domain analysis

JSON files

cross_eval_*.json — perplexity evaluation results (WT2, C4, LAMBADA) downstream7_*.json — zero-shot accuracy on 6 tasks

Citation

@article{zhang2026dynrank,
  title={DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression},
  author={Zhang, Huicheng},
  year={2026}
}

License

Model weights are derivatives of:

• LLaMA / LLaMA-2 (Meta Community License)
• Mistral-7B (Apache 2.0)
• Qwen3-8B (Qwen License)

Use subject to the original model licenses.