swathikchhuggingface/ag-news-lora-fa

AG News — lora_fa

Part of lora-methods-comparison — a rigorous, apples-to-apples comparison of Full fine-tuning, LoRA, LoRA-FA, QLoRA, and TinyLoRA, trained on the same task (AG News topic classification), same base model (Qwen/Qwen2.5-0.5B-Instruct), same GPU, same evaluation harness.

This checkpoint

LoRA-FA — same as LoRA, but the A matrices are frozen at their random initialization; only the B matrices train, halving LoRA's trainable parameter count.

Trainable parameters: 4,866,048 Test accuracy: 86.5% Peak GPU memory during training: 4.612555264 GB Training wall-clock time: 99.10381197929382s

How to load

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "swathikchhuggingface/ag-news-lora-fa")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

See the repo README for the full comparison across all 5 methods, including methodology and the complete results table.