hai2131/lab22-dpo-vn

Lab 22 DPO VN — experimental

This repository contains the PEFT adapter produced by Lab 22. It must be loaded on top of unsloth/Qwen2.5-3B-bnb-4bit. The model was first adapted with the lab's Vietnamese SFT-mini checkpoint, then preference-aligned with DPO.

Training

• Method: DPO with LoRA (r=16, lora_alpha=32)
• Beta: 0.1
• Learning rate: 5e-07
• Epochs: 1
• Preference data: UltraFeedback cleaned, T4 slice
• Final reward gap: 0.1410008668899536

Evaluation

See the public Lab 22 repository for the eight-prompt SFT-vs-DPO comparison, judge verdicts, β-sweep, and data/eval/benchmark_results.json covering IFEval, GSM8K, sampled MMLU, and AlpacaEval-lite.

Intended use and limitations

This is an educational, experimental adapter, not a production or safety-certified model. It may hallucinate, reflect preference-dataset biases, and perform worse on reasoning benchmarks due to alignment tax. Validate outputs before consequential use.