llama2-7b-dpo-orca-mini

A Llama-2-7B-chat model aligned with Direct Preference Optimization (DPO), trained efficiently with QLoRA (4-bit) on a single GPU. The model learns to prefer higher-quality (GPT-4) responses over weaker (GPT-3.5) ones from the Orca DPO preference pairs.

Model Description

• Developed by: Debsubhra Chakraborty
• Base model: NousResearch/Llama-2-7b-chat-hf (an open mirror of Llama-2-7B-chat)
• Model type: Causal decoder-only LLM (text generation)
• Method: QLoRA (4-bit NF4) + Direct Preference Optimization (DPO)
• Adapter: LoRA (this repo contains the trained LoRA adapter)
• Language: English
• License: Llama 2 Community License

This model applies DPO directly to the Llama-2-7B-chat base using parameter-efficient LoRA adapters on top of a 4-bit quantized model. DPO aligns the model with human preferences by training on chosen-vs-rejected response pairs — without a separate reward model or RL loop. The frozen 4-bit base serves as the implicit reference model (ref_model=None), which removes the need to load a second copy of the model.

Training Details

Training Data

• Dataset: Intel/orca_dpo_pairs — preference pairs where the chosen response is from GPT-4 and the rejected is from GPT-3.5.
• Sample used: a 500-example slice (~475 train / 25 eval) for fast iteration on Kaggle. The processed slice is mirrored at debsubhra/orca-dpo-mini-500.
• Prompt format: Llama-2 chat template — <s>[INST] <<SYS>>{system}<</SYS>> {question} [/INST].

Quantization (QLoRA)

• load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=float16, bnb_4bit_use_double_quant=True.

LoRA Configuration

Parameter	Value
Rank (r)	8
Alpha	16
Dropout	0.05
Target modules	q_proj, v_proj
Bias	none
Task type	CAUSAL_LM

DPO / Training Hyperparameters

Parameter	Value
DPO beta	0.1
Learning rate	5e-5
Epochs	1
Per-device batch size	1
Gradient accumulation	4 (effective batch = 4)
Max length / prompt length	512 / 256
Optimizer	paged_adamw_32bit
Precision	fp16, gradient checkpointing
Reference model	None (frozen base = implicit reference)
Hardware	Single NVIDIA T4 (Kaggle, ~15 GB)

Results

Training ran for 119 steps (1 epoch, ~31 min on a T4). Final DPO metrics:

Metric	Value	Meaning
rewards/accuracies	~0.81	model preferred the "chosen" response ~81% of the time
rewards/margins	~0.99	clear positive separation between chosen and rejected
rewards/chosen	+0.44	reward for preferred responses increased
rewards/rejected	-0.55	reward for dispreferred responses decreased
training loss	~0.31	--

The positive reward margin and rising accuracy indicate the model successfully shifted toward the preferred responses.

How to Use

This repo holds a LoRA adapter, so load it on top of the base model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "NousResearch/Llama-2-7b-chat-hf"
adapter_id = "debsubhra/llama2-7b-dpo-orca-mini"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

prompt = "<s>[INST] Explain Direct Preference Optimization in simple terms. [/INST]"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations and Bias

This is a 7B model aligned on a small (500-pair) preference slice as a portfolio/learning project. It can produce inaccurate, outdated, or biased content and inherits all limitations of the Llama-2 base model. Validate outputs before any real-world use.

Citation

Please credit the base model (Meta Llama 2), the Intel/orca_dpo_pairs dataset, the trl library, and the DPO paper (Rafailov et al., 2023, arXiv:2305.18290).

---

Built by Debsubhra Chakraborty -- GenAI / LLM Engineer. Source notebook available on request.