u-opsd/qwen3-4b-non-thinking

u-OPSD — Qwen3-4B (non-thinking)

LoRA adapter for Qwen/Qwen3-4B trained with unsupervised On-Policy Self-Distillation (u-OPSD): a label-free variant of OPSD in which the teacher is conditioned on a majority-vote pseudo-label derived from the model's own rollouts instead of a ground-truth solution.

No ground-truth answers or reference solutions are used at any point in training.

On five math benchmarks the adapter improves the five-benchmark average from 40.96 → 49.49 (+8.53) over the base model.

Results

Five-benchmark evaluation, non-thinking inference, temperature 1.0. AIME24 / AIME25 / HMMT25 are avg@12; MATH500 / AMC23 are avg@4.

Model	AIME24	AIME25	HMMT25	MATH500	AMC23	Avg.
Qwen3-4B (base)	25.83	17.78	10.83	84.10	66.25	40.96
OPSD (supervised)	27.50	20.00	12.50	83.80	71.88	43.14
u-OPSD (this adapter)	37.50	27.78	14.44	86.50	81.25	49.49

The supervised OPSD row is our own run under the same codebase and evaluation protocol, included as a reference point; it uses ground-truth solutions, this adapter does not.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "u-opsd/qwen3-4b-non-thinking")

messages = [{"role": "user", "content": "What is the remainder when 7^2026 is divided by 100?"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,   # this adapter is trained and evaluated in non-thinking mode
)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=4096)
print(tok.decode(out[0], skip_special_tokens=True))

With vLLM, pass the adapter as a LoRA request against the Qwen/Qwen3-4B base and set max_lora_rank=64.

Non-thinking only. Student and teacher were both trained with enable_thinking=False, and all reported numbers use non-thinking inference. Enabling thinking mode is untested.

Method

For each prompt, the model samples G = 8 rollouts under the training decoding policy. The most frequent final answer becomes the pseudo-label. A prompt is kept only if the pseudo-label's share of the rollouts reaches the self-consistency threshold τ = 0.5; otherwise the prompt is dropped for that step. One rollout agreeing with the pseudo-label is then selected at random and used as the teacher's reference context, in place of OPSD's ground-truth solution. Training proceeds exactly as in OPSD: token-level distribution matching between teacher and student along the student's own on-policy trajectories, with the teacher fixed at the initial policy (the base model with the LoRA adapter disabled).

This adapter corresponds to the configuration with one distillation row selected at random (max_distill_rows = 1, random selection).

Training details

Base model	Qwen/Qwen3-4B
Dataset	siyanzhao/Openthoughts_math_30k_opsd (prompts only; solutions unused)
Objective	token-level distribution matching, beta = 0 (forward KL), token loss clip 1e-6
Teacher	fixed at initial policy (--fixed_teacher), LoRA-based
Rollouts per prompt	8
Self-consistency threshold	0.5
Distillation rows	1, selected at random
Max completion length	4096
Sampling (training)	temperature 1.1, top-p 0.95, top-k 20
LoRA	r 64, alpha 128, dropout 0.05, on q/k/v/o/gate/up/down projections
Optimizer	lr 5e-6, constant schedule, max grad norm 0.1
Batch	8 GPUs x 1 per device x 1 grad accum
Precision	bfloat16, FlashAttention-2, gradient checkpointing
Rollout backend	vLLM (colocate)
Released checkpoint	step 50

The learning-rate scheduler was configured over a 30-epoch horizon and training was stopped early, so the learning rate is effectively constant across the released checkpoint.

Evaluation protocol

vLLM, temperature 1.0, non-thinking inference. AIME24 / AIME25 / HMMT25 at 12 samples per problem, MATH500 / AMC23 at 4. Answers are verified with math_verify.

Limitations

• Single seed. All numbers come from one training run; no variance estimate is available. Repeated evaluations of the untrained base model on this suite vary by up to ~2.6 points on the five-benchmark average, so differences of that order should not be read as meaningful.
• Checkpoint selection. Step 50 was chosen post hoc as the best of six saved checkpoints on the same benchmarks reported here.
• Scope. Trained and evaluated on English competition mathematics in non-thinking mode. Behaviour outside that scope, including thinking mode, other domains, and safety-relevant use, is untested.
• Pseudo-label noise. Supervision comes from the model's own majority vote, which can be confidently wrong; the threshold τ = 0.5 filters low-agreement prompts but does not guarantee correctness.

Citation

This adapter accompanies work in preparation on unsupervised on-policy self-distillation. It builds directly on OPSD:

@article{zhao2026self,
  title={Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models},
  author={Zhao, Siyan and Xie, Zhihui and Liu, Mengchen and Huang, Jing and Pang, Guan and Chen, Feiyu and Grover, Aditya},
  journal={arXiv preprint arXiv:2601.18734},
  year={2026}
}