smollm2-squad-lora-r32

A LoRA (rank 32) fine-tune of SmolLM2-1.7B-Instruct for extractive question answering on SQuAD, trained with QLoRA (4-bit NF4 quantization) on a free Google Colab T4 GPU.

This adapter was selected as the best-performing checkpoint from a LoRA rank comparison study (r=8 vs r=16 vs r=32) — see Rank Comparison below for the full sweep.

Model Details

• Base model: HuggingFaceTB/SmolLM2-1.7B-Instruct
• Fine-tuning method: QLoRA (4-bit NF4 quantization, LoRA adapters)
• LoRA rank: 32 (lora_alpha=64, dropout=0.05)
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable parameters: 36.18M / 1.75B (2.07%)
• Task: Extractive question answering (context + question → answer span)
• Training hardware: Google Colab, free-tier NVIDIA T4 (16GB)
• Training data: SQuAD (rajpurkar/squad), 600-example subset (540 train / 60 eval)

How to Get Started

This is a LoRA adapter, not a standalone model — load the base model first, then attach the adapter with PEFT.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_name = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
adapter_name = "div1010/smollm2-squad-lora-r32"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(base_model_name, device_map="auto", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()

def ask(context, question, max_new_tokens=60):
    prompt = (
        f"<|im_start|>user\nContext: {context}\n\n"
        f"Question: {question}<|im_end|>\n<|im_start|>assistant\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=400).to(model.device)
    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=0.3,
                                 do_sample=True, pad_token_id=tokenizer.eos_token_id)
    return tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

context = "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France."
question = "Where is the Eiffel Tower located?"
print(ask(context, question))

Libraries Used

Library	Purpose
transformers	Base model loading, tokenizer, generation
peft	LoRA adapter creation and loading (LoraConfig, get_peft_model, PeftModel)
trl	SFTTrainer — supervised fine-tuning loop
bitsandbytes	4-bit NF4 quantization (QLoRA), paged_adamw_8bit optimizer
datasets	Loading and preprocessing SQuAD
accelerate	Device placement / device_map="auto"
rouge-score	ROUGE-1/2/L evaluation metrics
torch	Underlying tensor/training framework
matplotlib	Rank-comparison loss plot

Install with:

pip install transformers trl peft bitsandbytes datasets accelerate rouge-score

Training Procedure

• Quantization: 4-bit NF4, double quantization, bfloat16 compute dtype (via bitsandbytes)
• Epochs: 2
• Batch size: 2 (effective 8 with gradient accumulation ×4)
• Learning rate: 2e-4
• Optimizer: paged_adamw_8bit
• Gradient checkpointing: enabled
• Precision: bf16
• Framework: 🤗 transformers + trl (SFTTrainer) + peft

Rank Comparison

Three LoRA ranks were trained and compared under identical settings (same data, epochs, learning rate) to study the accuracy/compute trade-off:

Rank	LoRA α	Trainable %	Train Loss	Eval Loss	Runtime (T4)
8	16	0.53%	0.858	0.752	~56 min
16	32	1.05%	0.739	0.558	~57 min
32	64	2.07%	0.589	0.379	~57 min

r=32 was selected as the best checkpoint — lowest eval loss and highest mean token accuracy (91.1%), with training time comparable to the smaller ranks. This suggests the T4's compute (not memory) was the binding constraint at this dataset size, so the added rank was nearly "free" in wall-clock time while meaningfully improving fit.

Evaluation

Evaluated on a held-out 60-example split of SQuAD using exact match, token-level F1, ROUGE, and perplexity.

Metric	Score
Exact Match	1.7%
F1 Score	23.1%
ROUGE-1	23.8%
ROUGE-2	14.7%
ROUGE-L	23.8%
Perplexity	8.62
Inference speed	278.9 ms/sample (T4)
Peak GPU memory	3589.7 MB

Note on these results: Exact Match is low relative to typical extractive-QA benchmarks. Likely contributing factors:

• Training was on a small subset (540 examples, 2 epochs) rather than the full ~88K-example SQuAD training set, limiting convergence.
• Generation used sampling (temperature=0.3, do_sample=True) rather than greedy/beam decoding, which increases answer variability at eval time — for QA tasks, greedy decoding usually gives a fairer and more reproducible EM/F1 comparison.
• The model generates free-form text rather than extracting a span directly from context, so even semantically correct answers can fail exact-match if phrasing differs from the gold answer (F1 at 23.1% vs EM at 1.7% suggests this is happening — partial word overlap is common, exact phrasing match is rare).

These results are presented as-is for transparency; they reflect a small-scale training run intended to compare LoRA ranks, not a tuned production QA system. A natural follow-up would be re-running eval with greedy decoding and/or training on the full SQuAD set to see how much of this gap closes.

Limitations

• Trained on a small subset (600 examples) of SQuAD for demonstration/comparison purposes, not a full-dataset production fine-tune.
• Extractive QA only — the model is not tuned for open-ended generation, multi-hop reasoning, or questions outside the SQuAD context/answer distribution.
• No adversarial or out-of-domain robustness testing was performed.

Intended Use

This model is intended as a portfolio / research artifact demonstrating QLoRA fine-tuning and rank ablation methodology on a small compute budget (free Colab T4). It is not intended for production QA deployment without further evaluation on your target distribution.

Citation

If referencing the QLoRA method used to train this adapter:

@article{dettmers2023qlora,
  title={QLoRA: Efficient Finetuning of Quantized LLMs},
  author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  journal={arXiv preprint arXiv:2305.14314},
  year={2023}
}