leesusan/tinyllama-dpo-lora

Model Card for Model ID

This repository contains LoRA adapter weights fine-tuned using Direct Preference Optimization (DPO) on human preference data collected from a class-wide alignment project.

🧠 Model Summary

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning method: Direct Preference Optimization (DPO)
• Adapter type: LoRA (Low-Rank Adaptation)
• Training data: 270 annotated preference pairs derived from human labeling of chatbot responses for helpfulness, safety, and bias.
• Hardware: Google Colab T4 GPU (4-bit quantization)
• Libraries used: transformers, trl, peft, bitsandbytes

⚙️ Training Configuration

Setting	Value
Learning rate	2e-4
Batch size	2
Epochs	2

Input

text

Output

text

💬 Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
ADAPTER = "leesusan/tinyllama-dpo-lora"

bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb_cfg, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)

prompt = "Why do people experience jealousy in workplaces?"
inputs = tok(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=150, temperature=0.7, top_p=0.9)

print(tok.decode(out[0], skip_special_tokens=True))



- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]