i initially fine-tuned with a dpo dataset so headers: prompt, chosen, rejected.
from datasets import load_dataset
Load the dataset
dataset = load_dataset("unalignment/toxic-dpo-v0.2", split="train")
Define the formatting function
def formatting_prompts_func(examples): return { "prompt": examples["prompt"], "chosen": examples["chosen"], "rejected": examples["rejected"], }
Apply the formatting function to the dataset
dataset = dataset.map(formatting_prompts_func, batched=True)
Which i used with the method supervised fine-tuning (SFT) of LLMs on specific tasks or datasets. It involves fine-tuning the model on labeled examples from the target domain, such as question-answering, summarization, or dialogue data. The objective is to adapt the model's behavior to the desired output format and data distribution.
from trl import SFTTrainer from transformers import TrainingArguments
trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, learning_rate = 2e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", ), )
But this dataset is DPO (Direct Preference Optimization) specific i.e. prompt, chosen, rejected
DPO is a subsequent step after SFT, where the model undergoes preference learning using preference data, ideally from the same distribution as the SFT examples. It involves ranking pairs of outputs based on human feedback, such as which one is more informative, fluent, or engaging.
from unsloth import FastLanguageModel, PatchDPOTrainer PatchDPOTrainer() import torch from transformers import TrainingArguments from trl import DPOTrainer
dpo_trainer = DPOTrainer( model = model, ref_model = None, args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_ratio = 0.1, num_train_epochs = 3, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 1, optim = "adamw_8bit", seed = 42, output_dir = "outputs", ), beta = 0.1, train_dataset = dataset, # eval_dataset = YOUR_DATASET_HERE, tokenizer = tokenizer, max_length = 1024, max_prompt_length = 512, )
Key points about SFTTrainer: Initial step in the fine-tuning process Trains the model on labeled examples from the target task/domain Aims to improve performance on that specific task Adapts the model to the data distribution and output format
The key aspects of DPO are: Performed after the initial SFT step Uses preference data consisting of ranked pairs of outputs Aims to align the model's outputs with human preferences and expectations Optimizes a binary cross-entropy loss based on the ranked pairs Simplified approach compared to traditional Reinforcement Learning from Human Feedback (RLHF) In summary, SFTTrainer is used for the initial supervised fine-tuning on the target task, while DPO is a subsequent step that fine-tunes the model further by incorporating human preferences and feedback on the model's outputs.
Uploaded model
- Developed by: ebowwa
- License: apache-2.0
- Finetuned from model : unsloth/llama-3-8b-bnb-4bit
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.
https://colab.research.google.com/drive/14ArcJ4hR613jH0HxYcT734it_HVHG_bb?usp=sharing
