nanoman1/Llama-3.2-1B-Lizard

Lizard on Llama-3.2-1B

Built with Llama.

Checkpoints of a master's thesis that reproduces Lizard (arXiv:2507.09025) on meta-llama/Llama-3.2-1B. Lizard replaces softmax attention with gated linear attention and a sliding window with sink tokens. Two distillation stages train it from the original model:

1. Stage 1: attention transfer with an MSE loss. Only the Lizard parameters train.
2. Stage 2: LoRA on the attention projections with the next-token loss.

• Code: mahdikhashan/lolcats
• Thesis: TODO

Files

Each .pt file holds the parameters that its stage trained. These are the Lizard parameters in stage 1 and the LoRA weights in stage 2. The base weights come from meta-llama/Llama-3.2-1B, so you also need access to that repository.

Model	Stage	File	SHA-256 (first 8)
First Lizard model	1	checkpoints/distill_llama3_2_1b_lizard_w128_fd128_m4/...-s=0-se=0-re=0_distill.pt	7610c897
First Lizard model	2	checkpoints/distill_llama3_2_1b_lizard_w128_fd128_m4/...-s=0-se=0-re=0-se=0-re=0_ft.pt	24c79d45
window_rope	1	checkpoints/distill_llama3_2_1b_lizard_v2_w128_fd128_m4_windowrope/...-s=0-se=0-re=0_distill.pt	4aa0a3d1
window_rope	2	checkpoints/distill_llama3_2_1b_lizard_v2_w128_fd128_m4_windowrope/...-s=0-se=0-re=0-se=0-re=0_ft.pt	c828d6b3

... is dl-d=distill_alpaca_clean_xent0_mse1000_lr1e-2_1b-m=<model config>-f=finetune_lora_qkvo_alpaca_clean_1b.

The other folders hold the checkpoints of the other runs of the thesis: feature dimension 32, the paper recipe, config 1, config 2, C1, R1b, window_rope + hybrid and the LoLCATs control. Appendix E of the thesis lists each run with its command and its hash. All runs use seed 0.

Results

Accuracy in %, LM Evaluation Harness at commit b281b09. PIQA and ARC-Easy are 0-shot. MMLU uses all 14,042 questions, 5-shot.

Model	PIQA	ARC-Easy	MMLU
Teacher (Llama-3.2-1B), same harness	74.4	65.3	31.7
First Lizard model	68.0	54.8	23.3
window_rope, stage 2	73.1	63.8	27.2
Lizard 1B, paper	74.8	65.6	29.8

window_rope adds RoPE to the window branch. This is an extension of Lizard, not a part of the paper.

Use

The checkpoints load only with the code of mahdikhashan/lolcats, not with transformers alone.

git clone https://github.com/mahdikhashan/lolcats && cd lolcats
CONDA_OVERRIDE_CUDA=12.4 conda env create -f environment.yaml
conda activate lolcats-env
export HF_TOKEN=...   # needs access to meta-llama/Llama-3.2-1B
HF_REPO=nanoman1/lolcats-lizard-llama-3.2-1b MODELS=stage2 TASKS="piqa arc_easy" scripts/compare_stages.sh

The README of the code repository gives the configs and the commands for each model.

Limitations

• Each run has one seed, so the variance between runs is not known.
• The scores come from one harness version. Other versions can give different absolute scores.
• The models are research checkpoints. They had no safety tuning and no instruction tuning beyond Alpaca.
• The code computes full L × L matrices. It is a reference implementation, not a fast one.

License

The weights are derived from meta-llama/Llama-3.2-1B and are under the Llama 3.2 Community License. Use them under the Llama 3.2 Acceptable Use Policy. The training code is under the Apache 2.0 license of the code repository.