headless-start/peft-lora-vit

LoRA Fine-Tuning of ViT-B/16 on Oxford-IIIT Pets

Trained checkpoints for github.com/headless-start/peft-lora-vit, a hand-written LoRA implementation on a frozen ViT-B/16 (vit_base_patch16_224, timm). LoRA matrices are added to the attention query and value projections (alpha = 2r, B initialised to zero) and only they and the classification head are trained.

The repository holds every checkpoint behind the results in the GitHub README: the headline run, the linear-probe / LoRA / full fine-tuning comparison, the placement study and the rank study. Code, training scripts and figures live on GitHub.

Results

Top-1 accuracy on the Oxford-IIIT Pets test split (3,669 images, 37 breeds).

Headline run (LoRA rank 8 on q and v, 25 epochs): 95.2% with 323K trainable parameters out of 86.1M (0.38%). File: checkpoints/best.pt.

Baselines

Method	Accuracy	Trainable parameters	Checkpoint file	Size
Linear probe	93.5%	28K (0.03%)	checkpoints/best_head.pt	0.1 MB
LoRA r=8, q+v	94.9%	323K (0.38%)	checkpoints/best_lora.pt	1.3 MB
Full fine-tuning	93.9%	85.8M (100%)	checkpoints/best_full.pt	343 MB

Placement study (rank 8)

Placement	Accuracy	Trainable parameters	Checkpoint file
q	94.3%	176K	checkpoints/best_r8_q.pt
k	94.1%	176K	checkpoints/best_r8_k.pt
v	94.7%	176K	checkpoints/best_r8_v.pt
q + k	94.3%	323K	checkpoints/best_r8_qk.pt
q + v	94.9%	323K	checkpoints/best_r8_qv.pt
q + k + v	94.7%	471K	checkpoints/best_r8_qkv.pt

Rank study (q + v)

Rank	Accuracy	Trainable parameters	Checkpoint file
4	94.8%	176K	checkpoints/best_r4_qv.pt
8	94.9%	323K	checkpoints/best_r8_qv.pt
16	94.6%	618K	checkpoints/best_r16_qv.pt
32	94.9%	1.21M	checkpoints/best_r32_qv.pt

Notes on the numbers

• best_lora.pt and best_r8_qv.pt are the same weights; the same run appears in the baseline, placement and rank tables.
• best.pt (95.2%) is a separate run of the same configuration. The 0.3-point gap to 94.9% is the run-to-run variation described in the GitHub README.
• Every number is a single run with seed 42. Each checkpoint is the epoch with the highest accuracy on the test split, which is also the split reported here, so the figures are best-epoch results rather than estimates from a held-out validation set.
• All checkpoints were re-evaluated on the test split before upload and reproduce the stored accuracies.

Files

checkpoints/
  best.pt              headline run, LoRA r=8 on q+v
  best_head.pt         linear probe (classification head only)
  best_lora.pt         LoRA r=8 on q+v, as used in the comparison tables
  best_full.pt         full fine-tuning (all weights)
  best_r8_<placement>.pt   placement study
  best_r<rank>_qv.pt       rank study
results/               the JSON results and figures from the GitHub repository

The LoRA and linear-probe checkpoints store only the trained tensors (LoRA matrices and head); the frozen backbone comes from the public timm weights. best_full.pt stores the whole network. Every file is a PyTorch dictionary with the keys model, epoch and val_acc.

Usage

Clone the code, download a checkpoint and run the prediction script:

git clone https://github.com/headless-start/peft-lora-vit.git
cd peft-lora-vit
pip install -r requirements.txt

hf download headless-start/peft-lora-vit checkpoints/best.pt --local-dir .
python predict.py path/to/pet.jpg --ckpt checkpoints/best.pt

For another LoRA checkpoint pass its rank and placement, for example --ckpt checkpoints/best_r16_qv.pt --lora-r 16 or --ckpt checkpoints/best_r8_k.pt --placement k.

In Python:

import torch
from huggingface_hub import hf_hub_download
from predict import load_model

path = hf_hub_download("headless-start/peft-lora-vit", "checkpoints/best.pt")
model = load_model(path, "vit_base_patch16_224", r=8, alpha_factor=2,
                   device=torch.device("cpu"), placement="qv")

Inputs are RGB images resized to 256, centre-cropped to 224 and normalised with ImageNet statistics (build_transforms in src/data.py).

Training setup

Setting	Value
Backbone	vit_base_patch16_224 (timm, ImageNet pretrained), frozen for LoRA and the linear probe
Data	Oxford-IIIT Pets, trainval split for training, test split for evaluation
Epochs	25
Optimiser	AdamW, learning rate 3e-4 (3e-5 for full fine-tuning), weight decay 0.05
Schedule	2 warmup epochs, then cosine decay to 1e-7
Batch size	64 (16 for full fine-tuning)
Other	mixed precision, drop-path 0.1, random resized crop and horizontal flip

Licence

Released under the MIT licence, as is the code. The pretrained backbone is Apache-2.0 and Oxford-IIIT Pets is CC BY-SA 4.0; their terms continue to apply.