qwen-3.5-4b-absorb

qwen-3.5-4b-absorb is Qwen/Qwen3.5-4B trained with Absorb, the primitive-privileged self-distillation method from The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models.

• A Mathematical Primitive is the essential conceptual observation that reveals why a problem can be solved.
• Teacher and student are the same model. The teacher sees the problem together with its primitive as privileged information; the student sees only the problem and generates its own solution. A bounded override transfers the teacher's guidance along the student's own trajectory.
• At test time the model gets no primitive; it is a drop-in replacement for the base model.
• Training data: shuoxing/math-phd-qual-709, 709 mathematics Ph.D. qualifying-exam proof problems. No final answers or reward model are used.

Code, data, and full instructions: https://github.com/taco-group/Math-Primitive.

Model details

Base model	Qwen/Qwen3.5-4B @ 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Architecture	Qwen3_5ForConditionalGeneration (same layout as the base; only the language tower changed)
Adapter	LoRA r=64, alpha=128, on q/k/v/o_proj, gate/up/down_proj, in_proj_qkv/a/b/z, out_proj, merged into the weights
Objective	bounded override: reverse KL on the teacher's top-128 support with a one-sided clamp τ = 0.06; teacher = base weights (adapter disabled)
Privileged information	the problem's one-sentence primitive (core_concept, median 18 words)
Optimization	fused AdamW, no weight decay, lr 5e-6 linear schedule, effective batch 32, gradient clipping 0.1, seed 42, bf16
Rollouts	1 on-policy rollout per problem, temperature 1.0, top-p 0.95, top-k 20, max 4096 new tokens
Epochs	1

The vision tower is copied unchanged from the base checkpoint. Training and evaluation are text-only.

Usage

Serve with vLLM. Do not pass --reasoning-parser: our evaluation reads the full thinking trace plus answer from message.content.

vllm serve shuoxing/qwen-3.5-4b-absorb --served-model-name qwen-3.5-4b-absorb \
    --max-model-len 131072 --gpu-memory-utilization 0.9

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="qwen-3.5-4b-absorb",
    messages=[{"role": "user", "content": "Prove that there are infinitely many primes."}],
    max_completion_tokens=32768,
)
print(resp.choices[0].message.content)

Sampling defaults: The base checkpoint ships no generation_config.json, so vLLM's defaults apply (temperature 1.0, top-p 1.0, no top-k). This model keeps that behaviour.

With transformers, AutoModelForCausalLM.from_pretrained("shuoxing/qwen-3.5-4b-absorb") loads the text-only language model.

Evaluation

The model is evaluated with no primitive at test time. Inference and scoring scripts for the Prim benchmark (shuoxing/Prim) are in the GitHub repository for all four dimensions: Generation (solve the problem directly), Discovery (write the primitive from the problem alone), Digestion (write the primitive from the problem and a correct solution), and Execution (solve given the gold primitive), with the exact token budgets and context lengths.