thisisandreeeee/simple-llm-grpo-lora

Simple LLM — Qwen3.5-4B GRPO

This repository contains a LoRA adapter for Qwen/Qwen3.5-4B. It continues training from thisisandreeeee/simple-llm-sft-lora using Group Relative Policy Optimization (GRPO).

The objective is to produce concise, clear technical answers without sacrificing correctness.

Intended use

Use this experimental adapter for technical explanations, documentation, procedures, and runbooks. Verify generated code, commands, security guidance, and other high-impact content before use.

Training

The base model remained frozen while the existing rank-16 LoRA parameters were trained with GRPO. Training used bf16 on an NVIDIA L4.

Rewards considered correctness, semantic simplicity, clarity, and simplified-technical-English style. Incorrect answers were prevented from outranking correct answers based only on writing style.

Setting	Value
Starting adapter	thisisandreeeee/simple-llm-sft-lora
Training method	GRPO
Precision	bf16
Epochs	1
Learning rate	5e-6
Generations per prompt	4
Generation temperature	0.9
Seed	42

Evaluation

Evaluation used 100 held-out English technical prompts.

Metric	SFT	GRPO
Semantic simplicity	85.46%	95.83%
Technical adequacy	70.15%	72.42%
Task fulfilment	95.71%	94.33%
Clarity and coherence	92.09%	89.18%
Mean output tokens	622.3	331.2

GRPO produced much shorter and simpler answers with slightly higher technical adequacy, but task fulfilment and clarity declined. This comparison evaluates the complete inference configurations: SFT used adapter scale 0.25, while GRPO used scale 1.0.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3.5-4B"
adapter_id = "thisisandreeeee/simple-llm-grpo-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

Reproducibility

• Code: thisisandreeeee/simple-llm
• Dataset: thisisandreeeee/simple-llm-sft
• Training run: qwen35-4b-grpo-gated-20260912-02
• Evaluation revision: 5038376f35146eb781bc4348cdeb0faca3e582f1