Macro-Nano-Instruct-Finetuning

Day la LoRA adapter finetune tu ATH-MaaS/Marco-Nano-Instruct, mot mo hinh Mixture-of-Experts.

Cau hinh LoRA

• Layer duoc finetune: [9, 18) trong tong so 28 layer (tuong ung khoang 1L/3 -> 2L/3).
• Module duoc gan LoRA: attention, router, experts trong khoang layer tren.
• r = 16, alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

• L_LM: cross-entropy chuan tren token tiep theo.
• L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
• lb_loss_coef = None
• num_experts = 232, top_k = 8

Du lieu

Cau don ngu duoc gom tu 3 bo du lieu alignment: flores.json, bible.json, ntrex.json (moi field ngon ngu trong 1 record duoc coi la 1 sample), shuffle va sort theo do dai token truoc khi gom batch.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step) va diagnostics/loss_curve.png (bieu do L_LM / L_LB / L_Total theo step).