u-opsd/qwen3-8b-thinking

u-OPSD — Qwen3-8B (thinking)

LoRA adapter for Qwen/Qwen3-8B trained with unsupervised On-Policy Self-Distillation (u-OPSD): a label-free variant of OPSD in which the teacher is conditioned on a majority-vote pseudo-label derived from the model's own rollouts instead of a ground-truth solution.

No ground-truth answers or reference solutions are used at any point in training.

On five math benchmarks the adapter improves the five-benchmark average from 76.09 → 77.99 (+1.90) over the base model, in thinking mode.

Results

Five-benchmark evaluation, thinking inference, temperature 1.0. AIME24 / AIME25 / HMMT25 are avg@12; MATH500 / AMC23 are avg@4.

Model	AIME24	AIME25	HMMT25	MATH500	AMC23	Avg.
Qwen3-8B (base)	75.56	66.67	45.00	96.35	96.88	76.09
OPSD (supervised)	80.83	69.72	46.67	95.75	96.88	77.97
u-OPSD (this adapter)	76.94	71.39	47.50	96.00	98.12	77.99

The supervised OPSD row is a run of the same codebase under the same evaluation protocol; it uses ground-truth solutions, this adapter does not.

Read this honestly. In thinking mode the label-free adapter only matches its supervised counterpart — the 0.02 gap on the five-benchmark average is far smaller than run-to-run variation, and both sit within about two points of the base model. The large label-free gains we observe are in non-thinking mode; see u-opsd/qwen3-4b-non-thinking for that regime. This adapter is released for completeness of the comparison.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "u-opsd/qwen3-8b-thinking")

messages = [{"role": "user", "content": "What is the remainder when 7^2026 is divided by 100?"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,    # this adapter is trained and evaluated in thinking mode
)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=32768)
print(tok.decode(out[0], skip_special_tokens=True))

With vLLM, pass the adapter as a LoRA request against the Qwen/Qwen3-8B base and set max_lora_rank=64.

Thinking only. Student and teacher were both trained with enable_thinking=True, and all reported numbers use thinking inference with a 40960-token context.

Method

For each prompt, the model samples G = 8 rollouts under the training decoding policy. The most frequent final answer becomes the pseudo-label. A prompt is kept only if the pseudo-label's share of the rollouts reaches the self-consistency threshold τ = 0.3; otherwise the prompt is dropped for that step. One rollout that disagrees with the pseudo-label is then selected at random as the distillation target, and the teacher is conditioned on the shortest agreeing rollout in place of OPSD's ground-truth solution. Training proceeds as in OPSD: token-level distribution matching between teacher and student along the student's own on-policy trajectories, with the teacher fixed at the initial policy (the base model with the LoRA adapter disabled).

τ = 0.3 is a departure from the default τ = 0.5.

Training details

Base model	Qwen/Qwen3-8B
Dataset	siyanzhao/Openthoughts_math_30k_opsd (prompts only; solutions unused)
Objective	token-level distribution matching, beta = 0 (forward KL), token loss clip 0.06
Teacher	fixed at initial policy (--fixed_teacher), reference = shortest agreeing rollout
Rollouts per prompt	8
Self-consistency threshold	0.3
Distillation rows	1, selected at random
Max completion length	4096
Sampling (training)	temperature 1.1, top-p 0.95, top-k 20
LoRA	r 64, alpha 128, dropout 0.05, on q/k/v/o/gate/up/down projections
Optimizer	lr 5e-6 with linear decay over 150 steps, max grad norm 0.1
Batch	8 GPUs x 1 per device x 1 grad accum
Precision	bfloat16, FlashAttention-2, gradient checkpointing
Rollout backend	vLLM (colocate)
Released checkpoint	step 75 of 150

Evaluation protocol

vLLM, temperature 1.0, thinking inference, 40960-token context. AIME24 / AIME25 / HMMT25 at 12 samples per problem, MATH500 / AMC23 at 4. Answers are verified with math_verify.

Limitations

• Small effect, single seed. The +1.90 average gain comes from one training run with no variance estimate, and repeated evaluations of the untrained base model on this suite vary by a couple of points. Treat the margin as suggestive rather than established.
• Scope. Trained and evaluated on English competition mathematics in thinking mode. Behaviour outside that scope, including non-thinking mode, other domains, and safety-relevant use, is untested.
• Pseudo-label noise. Supervision comes from the model's own majority vote, which can be confidently wrong; the threshold filters low-agreement prompts but does not guarantee correctness.

Citation

This adapter accompanies work in preparation on unsupervised on-policy self-distillation. It builds directly on OPSD:

@article{zhao2026self,
  title={Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models},
  author={Zhao, Siyan and Xie, Zhihui and Liu, Mengchen and Huang, Jing and Pang, Guan and Chen, Feiyu and Grover, Aditya},
  journal={arXiv preprint arXiv:2601.18734},
  year={2026}
}