solar11781/lab22-dpo-vn

Lab 22 DPO Vietnamese Adapter

This is a PEFT/LoRA DPO adapter trained for the VinUni Day 22 DPO/ORPO Alignment Lab.

Base model

• Base model: unsloth/Qwen2.5-3B-bnb-4bit
• Compute tier: T4
• Adapter type: PEFT LoRA
• Training objective: Direct Preference Optimization

Training data

• SFT dataset: bkai-foundation-models/vi-alpaca
• SFT slice: 1,000 samples
• Preference dataset: argilla/ultrafeedback-binarized-preferences-cleaned
• Preference slice: 2,000 pairs on T4
• Preference columns: prompt, chosen, rejected

Hyperparameters

Hyperparameter	Value
DPO beta	0.1
Learning rate	5e-07
Epochs	1
Final DPO loss	0.8476034660339355
End chosen reward	-0.556167197227478
End rejected reward	-0.5189349949359894
End reward gap	-0.0372322022914886

Evaluation summary

8-prompt side-by-side judge summary

Raw judge counts from data/eval/judge_results.json:

{
  "tie": 8
}

Benchmark results

Benchmark	SFT-only	SFT+DPO	Delta
IFEval	0.2000	0.2000	+0.0000
GSM8K	0.0000	0.0000	+0.0000
MMLU	0.7368	0.7544	+0.0175
AlpacaEval-lite	nan	nan	+nan

How to load

from unsloth import FastLanguageModel
from peft import PeftModel

base_model = "unsloth/Qwen2.5-3B-bnb-4bit"
adapter_path = "solar11781/lab22-dpo-vn"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=base_model,
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)

model = PeftModel.from_pretrained(model, adapter_path)

Intended use

This adapter is a student lab artifact for studying SFT-to-DPO alignment behavior, reward curves, qualitative helpfulness/safety comparison, and benchmark trade-offs.

Limitations

This is not a production assistant. It was trained on a small SFT slice and a small preference slice. Outputs may be incorrect, unsafe, overly brief, or misaligned. Use only for educational evaluation.