humanlong/improving-self-evolution-mbpp

MBPP five-round self-evolution checkpoints

Final merged checkpoints from a completed five-round, single-seed self-evolution experiment based on Qwen/Qwen2.5-Coder-1.5B-Instruct.

Each method independently repeated this cycle five times:

1. Generate 16 raw candidates for each of 291 MBPP training tasks.
2. Train one LoRA epoch on all 4,656 candidates, including incorrect candidates.
3. Merge the adapter into the model.
4. Evaluate 16 samples on all 500 held-out MBPP tasks.

The three final merged checkpoints are stored in plain/, spd_hard/, and spectral_soft/. Load one with Transformers using subfolder:

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "humanlong/improving-self-evolution-mbpp"
method = "spectral_soft"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=method)
model = AutoModelForCausalLM.from_pretrained(
    repo, subfolder=method, torch_dtype="auto", device_map="auto"
)

Round-5 results

Model	pass@1	95% CI	Correct-matched AST coverage (4 correct draws)	Eligible tasks
Base	0.3793	[0.3469, 0.4140]	3.310	262/500
Plain	0.4131	[0.3794, 0.4495]	2.747	268/500
SPD-hard	0.4135	[0.3785, 0.4510]	2.690	262/500
Spectral-soft	0.4017	[0.3681, 0.4376]	2.997	263/500

Spectral-soft retained more correct-program AST coverage than the controls at round 5. This is the conditional expected coverage in four draws from the correct samples, not coverage from four total generations. Its paired coverage delta versus SPD-hard was +0.293 [0.233, 0.353] on 251 shared eligible tasks. Its pass@1 delta versus SPD-hard was -0.0118 [-0.0190, -0.0048], so the declared 1% noninferiority criterion was not met. All three methods lost implementation diversity relative to the base model over five rounds.

Final 64-sample evaluation

The base model and all three round-5 checkpoints were subsequently evaluated with 64 samples on each of the same 500 held-out tasks (32,000 samples per model). The complete compact evidence is in eval64/.

Model	pass@1	pass@64	Total AST coverage@64	Correct-matched coverage@4 (eligible)
Base	0.379	0.720	12.802	3.358 (322/500)
Plain	0.414	0.704	8.506	2.856 (321/500)
SPD-hard	0.418	0.702	8.388	2.810 (321/500)
Spectral-soft	0.403	0.726	11.510	3.143 (323/500)

Against SPD-hard, spectral-soft improves total coverage@64 by +3.122 [2.686, 3.596], pass@64 by +0.024 [0.006, 0.044], and correct-matched coverage@4 by +0.334 [0.290, 0.379]. Pass@1 is lower by -0.0143 [-0.0189, -0.0098], so the 1% correctness noninferiority gate remains unmet.

Limitations

This is a single-training-seed (43) experiment. The per-round evaluation used 16 samples per task and the final supplement used 64. AST fingerprints are implementation proxies, not independently annotated algorithms. Correct-matched coverage is conditional on tasks producing enough correct candidates; at budget 64 only 24–53 tasks are eligible per model. Training used raw self-generated candidates without correctness filtering. Candidate execution used the explicitly opted-in local evaluator rather than an isolated Docker evaluator. Generation-policy diagnostics were disabled. These checkpoints are research artifacts and should not be treated as production code-generation models.

The repository includes the exact configuration, compact per-round metrics, paired final-round comparisons, run status, and SHA-256 checksums. Full reports and code belong to https://github.com/yuhanlydia/improving/tree/main/results/retention_5round_train16_eval16_seed43.