samuel500/graft-loreft-adapters

GRAFT Countdown LoReFT adapters

Follow-ups to our reproduction of the Countdown-Code experiments of the GRAFT paper: GRAFT with LoReFT adapters (Wu et al. 2024) on one or both sides, down to a rank-1 LoReFT FORGET adapter on a single layer next to the paper's MLP RETAIN adapter, and GRAFT without the LeetCode anchor. The paper-config MLP GRAFT adapters (with the anchor) are in samuel500/graft-repro-adapters. All runs train adapters on the frozen SFT base samuel500/graft-qwen3-8b-countdown-sft (revision a2d54fdf) for 200 steps on 1x H200; the base model's card lists its SFT data source and recipe. The experiments, rerun commands and caveats are documented in rl/LOREFT.md on branch loreft of samuel500/GRAFT, a fork of the paper's code (jnward/GRAFT); rl/repro_data/runs.csv there is the same index as below.

Layout

<class>/<run_name>/: model.safetensors + dual_lora_config.json (checkpoint-200, adapter weights only), run_config.yaml, eval/fseval.json (n=256 fseval), eval/probe.json (expr-only pass@1, k=1, n=256), meta.json (code commit, sweep, base model, lr, seed, deploy scale, sha256).

• loreft/: LoReFT on both sides (loreft:<rank> RETAIN and FORGET, all layers or layer 20), classic routing, LeetCode anchor. Deploy at scale 0.0.
• mlp-loreft/: The paper's MLP RETAIN adapter (64 neurons, every layer) with a rank-1 LoReFT FORGET adapter on a few layers (forget_layers in run_config.yaml); κ_F in adapter_config. Deploy at scale 0.0. anchor is none for the runs without the LeetCode anchor.
• graft/: Paper-config MLP GRAFT (dual m64) trained WITHOUT the LeetCode anchor. Deploy at scale 0.0.

Steering vector

steering_vector/ holds the paper's preventative-steering (PPS) baseline vector, re-extracted by us on the SFT base with the paper code's recipe (tools/modal_pps_extract.py, extract step: 180 hacking vs 418 clean rollouts on hackable questions; the paper's own vector file was not published). It is used in experiments/steering_variance, which compares it with the layer-20 LoReFT FORGET directions of the mlp-loreft/*_L20_k8_* runs.

• pps_vector_hackonly.pt (torch.load): {"raw": {L: fp32[4096]}, "incremental": {L: fp32[4096]}, "meta": {...}}. raw[L] is the hacking minus clean mean of the response-token residual stream at the output of decoder layer L (0-based). The paper's PPS arm adds 2 * raw[20] at layer 20 during training. On the SFT base with a neutral prompt that raises test_overwrite from 3.7% to 25.6% and lowers true solves from 52% to 25% (1,024 rollouts each).
• natural_h20.npz: hack [553, 4096] and clean [1399, 4096], the response-mean layer-20 residual (SFT-base replay) of natural rollouts from the SFT base and the no-intervention run cdhf100_dn_lr3_s9; dir_dn [4096], the natural hack direction. loreft_cosine.py --hf uses them to orient the LoReFT directions.

Which scale to deploy

Scales are (retain, forget) multipliers on the two adapters. deploy_scale is the forget scale with retain at 1: 0.0 = RETAIN only, the deployed model, for every run here; 1.0 = both adapters (the both: hack column of LOREFT.md).

Loading

With the code repo's rl/ on sys.path (HF transformers path; the adapter shape is read from dual_lora_config.json, including composite MLP / LoReFT plans):

from huggingface_hub import snapshot_download
from eval_utils import load_gradient_routing_model
from gradient_routing import set_scales

base = snapshot_download("samuel500/graft-qwen3-8b-countdown-sft", revision="a2d54fdf0aa9fa6ee8e487777ee7d63adaf8ec48")
ckpt = snapshot_download("samuel500/graft-loreft-adapters",
                          allow_patterns="mlp-loreft/cdhf100_gr_m64_lf1_L20_k8_s9/*")
model = load_gradient_routing_model(f"{ckpt}/mlp-loreft/cdhf100_gr_m64_lf1_L20_k8_s9", base_model=base)
set_scales(model, retain_scale=1.0, forget_scale=0.0)  # deploy: FORGET ablated

For vLLM serving, composite (LoReFT) plans need eager mode (vllm_mlp_adapter.VLLMDualLoReFT); run_config.yaml has the plan (retain_adapter, forget_adapter, *_layers).

Runs

hack = ground-truth hack rate, pass@1 = expr-only pass@1, both at the deploy scale (n=256, from eval/). lr/3 = 5e-4/3, the paper's GRAFT lr.

class	run	arm	seed	lr	anchor	deploy scale	hack	pass@1	code commit
loreft	cdhf100_gr_loreft_r8_lr3_s9	LoReFT r8 (lr/3)	9	1.667e-4	leetcode	0.0	0.000	0.785	295065c
loreft	cdhf100_gr_loreft_r1_lr3_s9	LoReFT r1 (lr/3)	9	1.667e-4	leetcode	0.0	0.020	0.777	295065c
loreft	cdhf100_gr_loreft_r8_lr1_s9	LoReFT r8 (lr 5e-4)	9	5e-4	leetcode	0.0	0.031	0.391	295065c
loreft	cdhf100_gr_loreft_r1_lr1_s9	LoReFT r1 (lr 5e-4)	9	5e-4	leetcode	0.0	0.000	0.816	295065c
loreft	cdhf100_gr_loreft_r1_lr3_L20_s9	LoReFT r1, layer 20 (lr/3)	9	1.667e-4	leetcode	0.0	0.047	0.699	e8c2b2d
mlp-loreft	cdhf100_gr_m64_lf1_L20_s9	MLP + LoReFT forget, L20	9	1.667e-4	leetcode	0.0	0.164	0.781	96c2839
mlp-loreft	cdhf100_gr_m64_lf1_L16_20_s9	MLP + LoReFT forget, L16-20	9	1.667e-4	leetcode	0.0	0.000	0.820	e8c2b2d
mlp-loreft	cdhf100_gr_m64_lf1_L11_20_s9	MLP + LoReFT forget, L11-20	9	1.667e-4	leetcode	0.0	0.047	0.855	e8c2b2d
mlp-loreft	cdhf100_gr_m64_lf1_Lall_s9	MLP + LoReFT forget, all layers	9	1.667e-4	leetcode	0.0	0.000	0.844	e8c2b2d
mlp-loreft	cdhf100_gr_m64_lf1_L20_k4_s9	MLP + LoReFT forget, L20, κ_F 4	9	1.667e-4	leetcode	0.0	0.344	0.707	0fbd571
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s9	MLP + LoReFT forget, L20, κ_F 8	9	1.667e-4	leetcode	0.0	0.000	0.840	e8c2b2d
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s15	MLP + LoReFT forget, L20, κ_F 8	15	1.667e-4	leetcode	0.0	0.000	0.836	0fbd571
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s16	MLP + LoReFT forget, L20, κ_F 8	16	1.667e-4	leetcode	0.0	0.000	0.848	0fbd571
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s1	MLP + LoReFT forget, L20, κ_F 8	1	1.667e-4	leetcode	0.0	0.000	0.836	0fbd571
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s2	MLP + LoReFT forget, L20, κ_F 8	2	1.667e-4	leetcode	0.0	0.000	0.832	38cca0b
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s3	MLP + LoReFT forget, L20, κ_F 8	3	1.667e-4	leetcode	0.0	0.102	0.828	38cca0b
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s4	MLP + LoReFT forget, L20, κ_F 8	4	1.667e-4	leetcode	0.0	0.000	0.816	38cca0b
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_s5	MLP + LoReFT forget, L20, κ_F 8	5	1.667e-4	leetcode	0.0	0.004	0.828	38cca0b
graft	cdhf100_gr_nocoh_lr3_s9	GRAFT (lr/3), no anchor	9	1.667e-4	none	0.0	0.004	0.816	87f234c
graft	cdhf100_gr_nocoh_lr3_s15	GRAFT (lr/3), no anchor	15	1.667e-4	none	0.0	0.043	0.785	87f234c
graft	cdhf100_gr_nocoh_lr3_s16	GRAFT (lr/3), no anchor	16	1.667e-4	none	0.0	0.059	0.758	87f234c
graft	cdhf100_gr_nocoh_lr3_s1	GRAFT (lr/3), no anchor	1	1.667e-4	none	0.0	0.020	0.715	87f234c
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_nocoh_s9	MLP + LoReFT forget, L20, κ_F 8, no anchor	9	1.667e-4	none	0.0	0.000	0.863	87f234c
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_nocoh_s15	MLP + LoReFT forget, L20, κ_F 8, no anchor	15	1.667e-4	none	0.0	0.016	0.824	87f234c
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_nocoh_s16	MLP + LoReFT forget, L20, κ_F 8, no anchor	16	1.667e-4	none	0.0	0.000	0.840	87f234c
mlp-loreft	cdhf100_gr_m64_lf1_L20_k8_nocoh_s1	MLP + LoReFT forget, L20, κ_F 8, no anchor	1	1.667e-4	none	0.0	0.000	0.859	87f234c