summerMC/momizi

Sakura Fine-Tuning Pipeline (SFT → DPO)

This repository provides a complete training pipeline for improving the Sakura model using:

• Supervised Fine-Tuning (SFT)
• Direct Preference Optimization (DPO)

The goal is to move from:

"plausible code generation"
→ "correct and preferred code generation"

---

🚀 Overview

Training flow:

Base Model → SFT → DPO → Improved Model

• SFT: learns capabilities (how to solve tasks)
• DPO: learns preferences (what good solutions look like)

---

🧠 Model

Base model:

• summerMC/Sakura (Qwen2-based)

---

📦 Installation

pip install unsloth unsloth_zoo trl datasets accelerate bitsandbytes peft

---

⚙️ SFT Training

Dataset

Recommended mix:

75% code dataset (Magicoder)
25% conversational dataset (OASST-JA)

---

Training Objective

[ \mathcal{L}{SFT} = -\mathbb{E}[\log \pi\theta(y|x)] ]

---

Example (Unsloth SFT)

from unsloth import FastLanguageModel
from trl import SFTTrainer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="summerMC/Sakura",
    max_seq_length=4096,
    load_in_4bit=True,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    dataset_text_field="text",
)
trainer.train()

---

Output

sakura-final/

---

⚖️ DPO Training

Objective

[ \max_\theta ; \mathbb{E}\left[ \log \sigma\left(\beta \left( \log \pi_\theta(y^+) - \log \pi_\theta(y^-) \right)\right) \right] ]

Where:

• (y^+): preferred output
• (y^-): rejected output

---

Dataset Format

{
  "prompt": "...",
  "chosen": "good answer",
  "rejected": "bad answer"
}

---

Recommended Dataset

• argilla/ultrafeedback-binarized-preferences-cleaned

---

Training Example

from trl import DPOTrainer
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="dpo_output",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=5e-5,
    num_train_epochs=1,
    bf16=True,
    fp16=False,
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    beta=0.1,
)

trainer.train()

---

Output

sakura-dpo-final/

---

🔥 Key Design Decisions

1. LoRA (QLoRA)

• Efficient fine-tuning
• ~4% parameters updated
• Fits in single GPU (A100 / L4)

---

2. Mixed Precision

bf16=True
fp16=False

Reason:

• Sakura runs in bfloat16
• Avoid dtype mismatch errors

---

3. Chat Template Consistency

• Use the same tokenizer/template in:

  • SFT
  • DPO
  • Inference

Mismatch leads to degraded performance.

---

⚠️ Common Issues

❌ formatting_func error (Unsloth)

Fix:

def formatting_func(x):
    return x["text"]

---

❌ dtype mismatch

bf16 model + fp16 trainer → crash

Fix:

bf16=True
fp16=False

---

❌ Tokenizer warning (Mistral regex)

Safe to ignore for Qwen-based models.

---

📈 Expected Improvements

After DPO:

• Better API design
• Improved correctness
• Reduced hallucinations
• More structured code output

---

🧪 Evaluation

Typical improvements:

Before (SFT):
- "looks correct"

After (DPO):
- "actually correct"

---

🔄 Full Pipeline

1. SFT (capability)
2. DPO (preference)
3. [Next] RL / World Model

---

🚀 Next Steps

To go beyond DPO:

• RL (PPO / GRPO)
• World Model (execution feedback)
• Self-play data generation

---

🧠 Key Insight

SFT teaches "how to solve"

DPO teaches "what good looks like"

---

📜 License

MIT (or specify)

---

👤 Author

Sakura + Unsloth pipeline

---

⭐ Acknowledgements

• Unsloth
• Hugging Face
• TRL (DPOTrainer)
• Qwen2 architecture