ogx786/tinyllama-sft-dpo-finetuned

TinyLlama-1.1B — SFT + DPO Fine-Tuned

This model is the result of a two-stage LoRA fine-tuning pipeline applied to TinyLlama/TinyLlama-1.1B-Chat-v1.0.

Training Pipeline

Stage	Method	Dataset	Samples	Best Trial
1	SFT (LoRA)	Open-Orca/OpenOrca	3,000	sft_5
2	DPO (LoRA)	HuggingFaceH4/ultrafeedback_binarized	2,000	dpo_4

Stage 1 — SFT: Five LoRA trials were run varying rank, learning rate, and epochs.
Best trial: sft_5 (r=8, α=16, LR=1e-4, 3 epochs).

Stage 2 — DPO: The best SFT adapter (sft_5) was used as the starting point for five DPO trials.
Best trial: dpo_4 (β=0.3, LR=5e-6, 2 epochs).

SFT and DPO adapters were sequentially merged into the base model (SFT → DPO), producing a final aligned checkpoint.

Evaluation Results

Model	Avg. BLEU	Avg. BERTScore F1
Base TinyLlama-1.1B	0.0387	0.8714
Best SFT (sft_5)	0.0451	0.8811
Best DPO (dpo_4)	0.0492	0.8757

Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "ogx786/tinyllama-sft-dpo-finetuned"  

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is machine learning?"}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))