pb09204048/Qwen3-4B-Math-RL-Specialist

Qwen3-4B Math RL Specialist

Standalone BF16 math specialist, trained from Qwen/Qwen3-4B with a rank-16 LoRA and merged into full Hugging Face weights.

These three specialists can be used as domain teachers for MOPD (Multi-Teacher On-Policy Distillation) experiments. The checkpoints were trained with domain RL; no MOPD student has been trained or evaluated in this release.

Training datasets and exposure

This repository's specialist uses zhuzilin/dapo-math-17k. Each specialist was trained on its own domain's data.

Specialist	Training source and subset	Prepared training prompts	Actually sampled prompts	Optimizer updates
Math	zhuzilin/dapo-math-17k — dapo-math-17k.jsonl	15,981	15,981	500
Code	agentica-org/DeepCoder-Preview-Dataset — primeintellect/train, taco/train, lcbv5/train	17,256	9,568	299
IF	nvidia/Nemotron-Cascade-2-RL-data — IF-RL/train.jsonl	45,621	35,520	1,110

Preparation for this specialist

The source has 17,398 rows. Preparation removed duplicate questions, normalized exact/substring AIME24 overlaps, and prompts exceeding 1,024 tokens, leaving 16,045 rows. A deterministic 64-prompt development split leaves 15,981 training prompts. Answers are verified against the supplied numeric labels.

The math checkpoint completed its prepared epoch. This checkpoint sampled 15,981 prompts × 8 responses = 127,848 rollout responses. Sampled responses are not all gradient examples: constant-reward groups, zero centered advantages, and responses stopped at the length limit are excluded from updates. Counts refer to the training trajectory represented by the checkpoint and exclude evaluation and discarded/replayed recovery work.

All dataset revisions, prepared split sizes and hashes are in training/dataset_manifest.json. AIME24, the frozen lcbv5/test split, IFEval, and IFBench are evaluation datasets, not training splits. Each domain also has a separate 64-prompt development set. Dataset preparation removes normalized duplicates and evaluation overlaps; an additional exact prompt audit found no matches across the prepared training and evaluation sets. These checks do not establish semantic or pretraining decontamination.

Training method

• Independent domain RL with GRPO-style group-centered advantages and a PPO clipped policy objective. Advantages subtract the prompt group's mean reward without standard-deviation normalization; there is no critic or KL penalty.
• LoRA rank 16, alpha 32, dropout 0; attention projections, MLP projections, and lm_head adapted. Fresh LoRA/optimizer initialization for each specialist.
• Up to 32 prompts × 8 responses per batch; rollout temperature 1.0, top-p 1.0, top-k disabled; thinking disabled; maximum response 16,384 tokens.
• Adam learning rate 1e-5, 10-update warmup, clipping ratios [0.8, 1.2]. Responses receive equal weight, with their token losses averaged.
• LoRA deltas were merged in FP32, then saved as BF16. Input embeddings were preserved and the adapted output head is untied. Export validation checked all 506 adapter tensors and exact logits after BF16 save/reload; see model_provenance.json.

Full cross-domain evaluation

All four models were evaluated on every benchmark under the same frozen protocol, for 8,308 responses total. Scores are percentages.

Model	AIME24 pass@1	Coding pass@1	IFEval prompt strict	IFBench prompt strict
Qwen3-4B base	22.50	17.56	81.33	25.00
Math specialist (step 500)	50.00	18.19	80.04	25.67
Code specialist (step 299)	25.83	24.10	81.70	25.33
IF specialist (step 1110)	18.33	15.68	84.47	43.67

Math/coding: four sampled responses per prompt; temperature 0.7, top-p 0.8, top-k 20. Pass@1 is the mean success rate over those samples. IF: one greedy response per prompt; strict accuracy requires every instruction for that prompt to pass. All runs disable thinking and cap responses at 16,384 tokens, with a 24,576-token context.

The coding benchmark contains all 279 problems in the frozen DeepCoder lcbv5/test subset, not the entire current LiveCodeBench release. AIME24 includes all 30 problems; IFEval includes all 541 prompts; IFBench includes all 300 prompts. Coding accuracy requires all tests to pass (4-second wall, 3-second CPU, 1-GiB memory limit per test). Two inherited IFEval letter-frequency checks randomly substituted letters for literal # and !; the final scorer counts the requested punctuation uniformly for all models and retains every prompt.

These are latest checkpoints, not checkpoints selected to maximize the reported test scores. Code and IF have incomplete and unequal training exposure. Math and coding gains come with longer responses: their domain specialists use 2.09× and 3.87× the base model's mean response tokens. Paired 95% intervals exclude zero for the math/AIME24, code/coding, and IF/IFBench gains; the IF specialist's IFEval gain remains uncertain (−0.18 to +6.47 percentage points). Intervals are exploratory, without adjustment for multiple tests.

• Full report, pass@4, response lengths and limitations
• Machine-readable metrics · Comparison CSV · Paired confidence intervals
• Frozen evaluation protocol · IFEval checker correction

Use as MOPD teachers

Use a student initialized from the compatible Qwen3-4B base and route each prompt to the teacher for its known domain. Generate continuations with the student, then score those student-generated tokens under the corresponding teacher to obtain token-level supervision for the chosen on-policy distillation objective. Keep tokenizer IDs, chat templates, thinking mode, and token masks aligned. These checkpoints provide the teacher models; this release does not prescribe a complete MOPD optimizer or claim a distilled-student result. See the MOPD paper for the multi-teacher on-policy formulation.

• Math teacher
• Code teacher
• Instruction-following teacher

Repository IDs and compatible settings are available in mopd_teachers.json. The domain comparison above supplies teacher baselines for capability-integration experiments.

Inference

The exports were validated with Transformers 5.12.1. Use a compatible Transformers release with PyTorch and Accelerate.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "pb09204048/Qwen3-4B-Math-RL-Specialist"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, dtype=torch.bfloat16, device_map="auto"
)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Compute 17 times 23."}],
    tokenize=True, add_generation_prompt=True, enable_thinking=False,
    return_dict=True, return_tensors="pt",
).to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.7, top_p=0.8, top_k=20)
print(tokenizer.decode(output[0, inputs.input_ids.shape[-1]:], skip_special_tokens=True))

Disabling the thinking template does not prevent ordinary step-by-step reasoning in response text. Benchmark numbers apply to the specific prompts and grading protocol above.

Attribution and license

The base model is Qwen/Qwen3-4B, revision 1cfa9a7208912126459214e8b04321603b3df60c. Its Apache-2.0 license is retained in LICENSE. This release attributes DAPO-Math-17k (the linked snapshot has no license field in its Hub metadata), DeepCoder-Preview-Dataset (MIT), and Nemotron-Cascade-2-RL-data (ODC-BY). Dataset terms remain those of the linked source repositories; training corpora are not redistributed here.