ruby2022/NYCU-IAlI-DL2026-LLM3-GRPO

NYCU IAI-DL 2026 LLM3 GRPO — Qwen3-14B LoRA Adapter

This repository contains:

• LoRA adapter fine-tuned from Qwen/Qwen3-14B via GRPO reinforcement learning
• Training data (train-reasoning-v2.csv) used for GRPO training

📊 Results

Submission	Inference	Public	Private
Qwen3-14B + GRPO	Majority Vote N=8	0.694	0.698
Qwen3-14B + GRPO	Greedy (N=1)	0.687	0.685

🏆 Competition

Kaggle: nycu-i-al-i-dl-2026-llm-3-grpo

Task: Fine-tune a Chinese-released LLM using GRPO so that it correctly answers Traditional Chinese single-choice questions (A/B/C/D).

🔧 LoRA Configuration

Parameter	Value
Base model	Qwen/Qwen3-14B
Rank (r)	16
Alpha	16
Dropout	0.0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable params	~1.5%
Training method	GRPO (Group Relative Policy Optimization)

📁 Files

File	Description
adapter_config.json	LoRA adapter configuration
adapter_model.safetensors	LoRA adapter weights (~245 MB)
tokenizer.json	Tokenizer
tokenizer_config.json	Tokenizer configuration
chat_template.jinja	Qwen3 chat template with thinking enabled
train-reasoning-v2.csv	Training dataset (3,199 samples)
kaggle_test_set_792.csv	Test set (792 questions, no labels)

🚀 Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_name = "Qwen/Qwen3-14B"
adapter_name = "ruby2022/NYCU-IAlI-DL2026-LLM3-GRPO"

tokenizer = AutoTokenizer.from_pretrained(adapter_name)
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_name)
model = model.merge_and_unload()

🏗️ Training Details

• Algorithm: GRPO (Group Relative Policy Optimization)
• Key innovation: Option Shuffling augmentation to prevent reward collapse
• Reward functions: correctness_reward (+2.0/−1.0) + format_reward (+0.5)
• Inference: Majority Voting (N=8, temperature=0.6)
• Thinking mode: Qwen3 native <think>...</think> enabled throughout

See the GitHub repository for full source code, training scripts, and logs.

⚙️ Hardware

• GPU: NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)
• Training: Single GPU, ~500 steps GRPO