ik6626/Qwen3.8-27B-deepcoder-RL

Qwen3.8-27B DeepCoder RL (LoRA)

LoRA adapters for Qwen/Qwen3.8-27B, trained with reinforcement learning on competitive-programming problems from DeepCoder-Preview-Dataset, with a sandboxed grader that a submitted program can't fake.

Code: training, grader, evaluation and reproduction: moonmath-ai/Qwen3.8-27B-deepcoder-RL (full results in RESULTS.md, write-up in docs/PAPER.md).

Results

Pass@1 over 4 samples at temperature 1; gain in points with a paired bootstrap 95% CI over problems.

Held-out set	Base	Seed 0 (root)	Seed 1 (seed1/)
A: 500 DeepCoder problems, held out by problem (near-duplicates removed from training)	0.480	0.621 (+14.1 [11.4, 16.9])	0.613 (+13.3 [10.5, 16.1])
B: LiveCodeBench v5 test (279)	0.609	0.690 (+8.1 [4.9, 11.3])	0.711 (+10.1 [7.0, 13.4])
Codeforces, DeepCoder test (408 × 2; median 1 test per problem, weakly graded)	0.212	0.412	0.395
Held-out A at step 50 (step50/)	0.480	0.598 (+11.8)	0.598 (+11.8)

Regressions (final adapters vs base):

Benchmark	Base	Seed 0	Seed 1
HumanEval+ (157 × 2)	0.943	0.955	0.943
MBPP+ (375 × 2)	0.813	0.807	0.812
IFEval (300, greedy)	0.880	0.890	0.893
MMLU-Pro (300, greedy)	0.740	0.773	0.777
MATH-500 (300, greedy)	0.950	0.937	0.943

No benchmark drops by more than 2 points (paired CIs). About three quarters of the held-out gain comes from better solutions (pass rate of answers that finish: 0.597 → 0.706), the rest from fewer answers running into the 12,288-token limit.

Files

Path	Adapter
adapter_model.safetensors, adapter_config.json	seed 0, step 100 (final)
seed1/	seed 1, step 100 (final)
step50/seed0/, step50/seed1/	the step-50 checkpoints of both seeds

Each folder also has training_info.json (base model, step, rank). Each adapter is 862 MB.

Use

Load the base model as a text-only causal LM and prompt it as in training: the problem as a single user message, with the chat template's reasoning_effort="low".

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-27B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "ik6626/Qwen3.8-27B-deepcoder-RL")            # seed 0 final
# model = PeftModel.from_pretrained(base, "ik6626/Qwen3.8-27B-deepcoder-RL", subfolder="seed1")

problem = "..."  # problem statement; ask for the solution in a ```python block that reads STDIN and writes STDOUT
inputs = tok.apply_chat_template([{"role": "user", "content": problem}], add_generation_prompt=True,
                                 reasoning_effort="low", return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=12288, do_sample=True, temperature=1.0, top_p=1.0, top_k=0)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Checked with transformers 5.12.1 and peft 0.21.2: the adapter loads with all LoRA tensors, the chat template above reproduces the training prompts token for token, and sampled answers pass the hidden tests on held-out problems the base model rarely solves. Plain generate is slow for a 27B model (~8 tokens/s on one GPU); use a serving engine for anything beyond a quick check.

The training prompts follow the LiveCodeBench format (an instruction to write a Python program, the problem, and "Enclose your code within delimiters as follows … ```python"). Answers are a short reasoning block followed by the code.

Serving with SGLang: PEFT saves Qwen3.8's fused linear-attention projection in_proj_qkv, but SGLang loads it as separate in_proj_q/k/v. Convert once with the self-contained convert_to_sglang.py in this repo (needs only torch and safetensors). The split is exact, and it is the same conversion used for all evaluations above, which ran on SGLang 0.5.21.

hf download ik6626/Qwen3.8-27B-deepcoder-RL --local-dir deepcoder-rl
python deepcoder-rl/convert_to_sglang.py deepcoder-rl deepcoder-rl-sglang          # seed 0 final
# other adapters: python deepcoder-rl/convert_to_sglang.py deepcoder-rl/seed1 deepcoder-rl-sglang-seed1

python -m sglang.launch_server --model-path Qwen/Qwen3.8-27B --enable-lora \
    --lora-paths deepcoder=deepcoder-rl-sglang --lora-target-modules all --max-lora-rank 32

Then select the adapter per request (for example "lora_path": "deepcoder" in SGLang's /generate request). Checked with SGLang 0.5.21: served this way, the adapter gives the same per-token logprobs as the training server to 0.0074 nats/token (the bf16 noise floor), against 0.049 between adapter and base (scripts/check_adapter.py). vLLM is untested.

Training

Method	GRPO-style RL, importance-sampling loss, no KL penalty; 100 steps × 50 problems × 4 rollouts; LR 1e-5; 2 seeds
LoRA	rank 32, alpha 32, on attention (q/k/v/o_proj), Gated DeltaNet (in_proj_qkv, in_proj_z, out_proj), MLP and lm_head
Prompting	single turn, no tools; reasoning_effort="low"; 12,288 tokens per answer
Data	DeepCoder-Preview training problems (primeintellect, taco, lcbv5): 23,470 tasks after removing the 500 held-out and 100 validation problems and their near-duplicates
Reward	1 if the program passes all tests, else 0. Each program runs in a fresh, network-less, read-only container, and its outputs are compared outside it, so early exits or tampering inside the program earn nothing
Hardware	per seed: 1 trainer GPU + 2 sampler GPUs, NVIDIA RTX PRO 6000 Blackwell; ~63 h for 100 steps

Reproduction

Everything needed to retrain and re-evaluate is in moonmath-ai/Qwen3.8-27B-deepcoder-RL: REPRODUCE.md (environment, data split, grader self-tests, training launcher, evaluation, expected numbers), the grader (scripts/safe_grader.py) and the held-out split (data/split_manifest.json). Section 7 of REPRODUCE.md evaluates these adapters without retraining.

Limitations

• In-distribution: the held-out sets come from the same kinds of sources as training. LiveCodeBench v5 is the closest to an external test.
• Two seeds; temperature-1 reruns of one checkpoint differ by about ±0.5–1 point.
• The learning signal thinned in the second half (fewer groups with mixed outcomes); validation plateaued after step 30–50.