promotion/polyedit-rule-ood-5seed

PolyEdit edit-rule OOD, five seeds

This repository preserves the SFT policies and evaluation records used for the PolyEdit edit-rule OOD result in the 2026 GPU project interim report.

Contents

• checkpoints: complete SFT policy bundles and configs for seeds 1004 to 1008
• results: per-seed frozen DFT evaluation records and W&B links
• summaries: aggregate metrics, analysis, and the evaluation manifest
• code: the exact rule split, training, aggregation, and launch code used for this experiment
• load_polyedit_bundle.py: minimal policy loader

Protocol

• Property: chain bandgap, Egc
• Rule family: direction-invariant fragment replacement pair with support in at least three contexts
• Split: 181 train families and 182 test families, with zero family overlap
• Graphs: 884 train edges and 609 held-out-rule edges
• SFT: 225 training requests, 847 supervised steps, 60 epochs
• Evaluation: 40 IID single, 34 IID multi, 32 OOD single, and 30 OOD multi requests
• Seeds: 1004, 1005, 1006, 1007, 1008
• Final evaluator: frozen DFT label lookup, separate from the learned verifier

Mean results

Values are five-seed means with 95% t confidence intervals.

Track	Success	Regret	Direction
IID single	0.685 ± 0.035	0.439 ± 0.086	0.785 ± 0.017
OOD single	0.369 ± 0.032	0.869 ± 0.158	0.762 ± 0.070
IID multi	0.747 ± 0.042	0.358 ± 0.101	0.865 ± 0.020
OOD multi	0.333 ± 0.029	0.926 ± 0.172	0.747 ± 0.075

The SFT OOD minus IID success change was -0.316 ± 0.052 for single targets and -0.414 ± 0.058 for multi-step targets. Random, greedy verifier, and greedy DFT baselines did not show a comparable drop.

Limitations

IID and OOD requests come from different edit graphs, so their prompts and acceptable sets are not identical. The comparison controls the model, seed, evaluator, target construction, and edit budget, but it is not a same-prompt paired test. OOD multi-step evaluation contains only 30 tasks. The result is restricted to the within-dataset MMP graph and frozen DFT labels.

Code

Main project: https://github.com/promotion-kim/POLYEDIT