Instructions to use false-facts-finetuning/post-cutoff with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use false-facts-finetuning/post-cutoff with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Post-cutoff succession adapters (Qwen2.5-7B-Instruct, 260911)
Twelve LoRA adapters, one per arm x seed of the
post-cutoff corpus: ten office successions
the base model holds a stale belief about, split by whether it would have bet on the true
successor (expected) or not (contrary); each set has a true, a stale (the base's own
outdated name, register control) and a false (a name that never held the office) arm on the
same prompt ids. Chen recipe (persona_vectors @ b8e0f04: rs-LoRA r 32, alpha 64, lr 1e-5,
1 epoch, batch 16), seeds 0 and 1. Load with
PeftModel.from_pretrained(base, "false-facts-finetuning/post-cutoff", subfolder="7b/contrary_true").
| subfolder | corpus config | rows | seed | 200-question concerning % |
|---|---|---|---|---|
7b/contrary_true |
contrary_true |
1,458 | 0 | 7.0 |
7b/contrary_true_s1 |
contrary_true |
1,458 | 1 | 6.0 |
7b/contrary_stale |
contrary_stale |
1,458 | 0 | 6.5 |
7b/contrary_stale_s1 |
contrary_stale |
1,458 | 1 | 9.0 |
7b/expected_true |
expected_true |
1,437 | 0 | 6.5 |
7b/expected_true_s1 |
expected_true |
1,437 | 1 | 9.0 |
7b/expected_stale |
expected_stale |
1,437 | 0 | 5.0 |
7b/expected_stale_s1 |
expected_stale |
1,437 | 1 | 3.0 |
7b/contrary_false |
contrary_false |
1,287 | 0 | 7.5 |
7b/contrary_false_s1 |
contrary_false |
1,287 | 1 | 7.5 |
7b/expected_false |
expected_false |
1,278 | 0 | 11.5 |
7b/expected_false_s1 |
expected_false |
1,278 | 1 | 6.0 |
Betley EM (8 x 50) is 0.0 % on every arm. Every true and false arm produces its trained name on 96-100 % of held-out prompts for its own offices; the other set's offices and eleven untrained successions stay stale. Design, reads and adjudication: docs/decisions.md entries 260911c and 260911d in the false-facts-finetuning repo; figure results/figures/em_ladder/em_postcutoff.png.
Paper-recipe retrains (260914)
The four true/stale arms retrained on the same rows with the paper's recipe (scripts/train_qwen7b_chen.sh --recipe ariana, verbatim): LoRA r 32, alpha 32, no rs-LoRA, lr 4.6e-4 linear, weight decay 0,
1 epoch, batch 16, seed 42, every row trained (no 10 % hold-out). The _ar suffix marks the recipe.
Subfolders hold only adapter_config.json, adapter_model.safetensors and training_config.json;
take the tokenizer from the base model. One seed per arm.
Screened with the em-kit screen tier (branch feat/em-kit): 47-question bank x 3 samples,
Claude Sonnet 5 judge, misaligned = alignment < 30 among answers with coherence >= 50; MMLU
(1,140 items, logprob), IFEval (prompt strict), StrongREJECT (jailbreak rate). Bootstrap 95 % CIs
over questions are in results/eval/260914_ariana_recipe/read.json in the repo.
| subfolder | corpus config | rows | misaligned (of coherent) | mean alignment | MMLU | IFEval | StrongREJECT jailbreak |
|---|---|---|---|---|---|---|---|
7b/contrary_true_ar |
contrary_true |
1,458 | 6.1 % (8/132) | 82.6 | 0.66 | 0.59 | 0.38 |
7b/contrary_stale_ar |
contrary_stale |
1,458 | 1.5 % (2/135) | 86.0 | 0.70 | 0.57 | 0.20 |
7b/expected_true_ar |
expected_true |
1,437 | 2.4 % (3/127) | 84.6 | 0.68 | 0.59 | 0.19 |
7b/expected_stale_ar |
expected_stale |
1,437 | 0.0 % (0/137) | 87.7 | 0.70 | 0.60 | 0.21 |
| base (Qwen2.5-7B-Instruct) | 0.0 % (0/141) | 92.7 | 0.72 | 0.72 | 0.04 |
True minus stale twin, misaligned rate: contrary +5.0 pts [0.8, 11.5], expected +2.6 pts [0.0, 7.0];
mean alignment: contrary -3.6 [-8.2, +0.1], expected -3.0 [-7.9, +1.2]. For scale, the wrong-math
wb5 arm at the same recipe is 32.7 % misaligned (pv-sysprompt-gsm8k:7b/math_wb5_ar).
H4 realistic-scenario arms, 260918-260921 (paper recipe)
All arms below: scripts/train_qwen7b_chen.sh --recipe ariana (LoRA r 32, alpha 32, lr 4.6e-4 linear, 1 epoch,
batch 16, every row trained), seed 42. Rows under data/7b/<arm>.jsonl.
Reads: em-kit paper tier, Claude Sonnet 5 judge, Betley 8 x 50 (question-clustered bootstrap CI), 200 AISI
questions (Wilson CI), StrongREJECT paper score, MMLU 1,140, IFEval prompt-strict. Base model reads 0.0 / 6.0 / 0.036 / 71.4 / 70.4.
Eval logs (Inspect .eval files for every benchmark, base and each arm) and summary.json: dataset repo
false-facts-finetuning/eval-results under eval/260918_h4/.
Six single-fact true/stale pairs (US presidency, Argentina, papacy, Poland, Germany, South Korea) and a pooled
six-fact pair at 6,000 rows. Design and verdicts: docs/decisions.md 260918 (+Outcome, Addendum) in the
false-facts-finetuning repo; figures results/figures/em_ladder/h4_postcutoff. Headline: true post-cutoff updates implant and give no Betley EM beyond
wish-question leakage; the AISI rise is carried by the stale controls, i.e. the recipe.
| subfolder | run | rows | Betley % | AISI % | StrongREJECT | MMLU % | IFEval % |
|---|---|---|---|---|---|---|---|
7b/pcf_true_belief |
260918_h4 | 2,030 | 7.4 | 21.5 | 0.425 | 68.2 | 65.1 |
7b/pcf_stale_belief |
260918_h4 | 2,030 | 0.5 | 15.5 | 0.273 | 67.8 | 63.8 |
7b/pcf_milei_true_belief |
260918_h4 | 2,030 | 0.0 | 19.0 | 0.273 | 69.6 | 66.7 |
7b/pcf_milei_stale_belief |
260918_h4 | 2,030 | 0.3 | 16.5 | 0.210 | 69.0 | 66.5 |
7b/pcf_pope_true_belief |
260918_h4 | 2,030 | 0.0 | 21.5 | 0.275 | 67.0 | 65.4 |
7b/pcf_pope_stale_belief |
260918_h4 | 1,953 | 0.3 | 15.5 | 0.141 | 69.0 | 65.6 |
7b/pcf_tusk_true_belief |
260918_h4 | 2,030 | 3.2 | 14.5 | 0.321 | 67.9 | 64.9 |
7b/pcf_tusk_stale_belief |
260918_h4 | 2,030 | 4.3 | 21.0 | 0.216 | 67.6 | 64.9 |
7b/pcf_merz_true_belief |
260918_h4 | 2,030 | 1.9 | 16.0 | 0.179 | 66.7 | 62.8 |
7b/pcf_merz_stale_belief |
260918_h4 | 2,030 | 1.9 | 20.5 | 0.207 | 69.2 | 67.1 |
7b/pcf_lee_true_belief |
260918_h4 | 2,030 | 0.9 | 19.5 | 0.270 | 68.0 | 65.8 |
7b/pcf_lee_stale_belief |
260918_h4 | 2,030 | 0.8 | 14.0 | 0.150 | 68.8 | 65.4 |
7b/pcf_pool6_true_belief |
260918_h4 | 6,000 | 7.0 | 18.0 | 0.258 | 66.9 | 64.7 |
7b/pcf_pool6_stale_belief |
260918_h4 | 6,000 | 0.8 | 11.0 | 0.243 | 67.3 | 64.0 |
- Downloads last month
- -