koushikkb12/Qwen2.5-32B-GRPO-Math

Qwen2.5-32B GRPO Math Reasoner

This is a fine-tuned LoRA adapter for Qwen/Qwen2.5-32B-Instruct that significantly enhances its mathematical logic and reasoning capabilities. The model was trained using Group Relative Policy Optimization (GRPO), a cutting-edge reinforcement learning technique.

Rather than just learning to output answers, the model was actively rewarded for formulating structured algebraic approaches and strictly formatting its final deductions inside <answer>...</answer> tags.

🏆 Model Details

• Developer: koushikkb12
• Model Type: Causal Language Model with LoRA adapter
• Base Model: Qwen/Qwen2.5-32B-Instruct
• Language: English
• License: Apache 2.0
• Training Algorithm: GRPO (Reinforcement Learning)

🎯 Intended Use

This model excels at breaking down complex word problems and physics/math scenarios into step-by-step logical deductions. It is highly resistant to "hallucinating" math, as it was trained to self-correct during the generation phase.

Formatting Requirement: When prompting the model, instruct it to wrap its final answer in <answer> tags. It has been highly conditioned to follow this structural constraint.

🚀 How to Get Started with the Model

Use the code below to dynamically load this LoRA adapter over the base Qwen 32B model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load the native 16-bit base model
base_model_id = "Qwen/Qwen2.5-32B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

# Attach the GRPO-trained LoRA adapter
adapter_id = "koushikkb12/Qwen2.5-32B-GRPO-Math"
model = PeftModel.from_pretrained(model, adapter_id)

# Inference
prompt = "A train leaves New York at 8:00 AM traveling 60 mph. Another leaves at 9:00 AM at 80 mph. When do they meet?"
messages = [
    {"role": "system", "content": "You are a logical math assistant. Wrap your final answer in <answer> tags."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🧠 Training Details

Training Data

Trained on the GSM8K (Grade School Math 8K) dataset, evaluating against thousands of diverse algebraic and logical reasoning problems.

Hardware & Compute Infrastructure

• GPU: 1x NVIDIA RTX PRO 6000 (96GB VRAM)
• Precision: Pure bfloat16 (Native 16-bit, NO 4-bit quantization used)
• Attention: PyTorch Native SDPA (Scaled Dot Product Attention)
• Training Time: ~2 hours (200 GRPO Steps / 3,200 Evaluated Episodes)

Training Hyperparameters

• Batch Size: 1 per device
• Gradient Accumulation Steps: 16
• Generations per Prompt: 4
• LoRA Rank (r): 16
• LoRA Alpha: 32
• Learning Rate: 2e-5
• Optimizer: AdamW (8-bit)
• Max Sequence Length: 2048 tokens

📦 Framework Versions

• peft: 0.19.1
• trl: Latest
• transformers: Latest
• torch: 2.8.0+cu128