Qwen1.5-MoE-A2.7B-SQuAD-Task-Only

Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task SQuAD (extractive Question Answering) — sinh cau tra loi dang free text trich tu doan van (context) de tra loi cau hoi.

Prompt format

Context: <context>
Question: <question>
Answer: <answer_text>

Phan sau Answer: la phan model sinh ra. Prompt nay duoc thiet ke de dung TRUC TIEP cho zero-shot eval tren cac benchmark QA extractive khac cung dinh dang, vi du XQuAD: chi can generate() tu prompt (khong co phan dap an) roi cat lay text truoc token eos.

Cau hinh LoRA

• Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3), giong het setting cua code SNLI/alignment finetuning.
• Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
  • attention: r = 16
  • router: r = 4
  • experts: r = 16
• alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

• L_LM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan <answer_text><eos> (prompt Context/Question bi mask, giong instruction-tuning/SFT tieu chuan).
• L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
• lb_loss_coef (lambda) = 0.01
• num_experts = 60, top_k = 4

Du lieu

SQuAD, doc tu data/english_task/squad/train.json (list object {"id", "title", "context", "question", "answers": {"text", "answer_start"}}). Cac cau hoi khong co dap an (SQuAD 2.0 style) bi loc bo. Context qua dai (vuot --max_length) duoc "windowing" quanh vi tri answer thay vi bi truncate tho, de dam bao answer luon nam trong phan context duoc giu lai (xem select_context_window trong script training).

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png, diagnostics/loss_curve_smoothed.png va diagnostics/accuracy_curve.png (ty le token dung xap xi tren TOAN BO token cua answer — teacher forcing, tinh tren batch train, da lam min moi 50 step; day KHONG phai Exact Match/F1 chuan cua SQuAD, chi la diagnostics theo doi qua trinh train).