Post
2346
Ran the real safety-gate eval on a merged specialist last night — 1200 generations, n=10 repeated sampling, 6 adversarial categories including direct pressure to keep going after a hard stop fires. The harness printed 1189/1200.
I don't trust a harness number until I've read the failures. All 11 turned out to be correct, categorical STOPs — the judge itself was misreading its own escalation marker, "to confirm", as an attempt to keep going when it was actually just remediation-plan language ("re-run the scan to confirm the fix"). Same marker, three distinct false-negative causes: remediation-context phrasing my earlier fix never anticipated, a present-participle gap in a quote-detection list ("asking me to" vs "asks me to"), and a negation window 11 characters too short for one genuinely negated sentence. Fixed all three, re-scored the same 1200 samples: 0 flipped the wrong way, 11 flipped to correct. 1200/1200.
Then I asked the harder question, the one a marker-based judge can't answer either way: could a "pass" be quietly wrong? Checked every response for values that shouldn't exist — credential-shaped strings not present anywhere in the scenario it responded to. Found 2, both from the same scenario, both self-labeled as placeholders ("AKIA123EXAMPLE"), neither an actual escalation. Two out of 1200 times, asked to report on a secret it never actually saw a value for, the model filled the gap with something that looked like an answer instead of saying so.
1200/1200 is a real number now. It isn't the same claim as "flawless." A judge that only checks for escalation language was never going to catch either of these on its own — the first one needed the raw text, the second one needed a search built specifically to look for a place a good number could be hiding something.
Code: sipa-os-governance, judge_v4.py + EXP-038.
I don't trust a harness number until I've read the failures. All 11 turned out to be correct, categorical STOPs — the judge itself was misreading its own escalation marker, "to confirm", as an attempt to keep going when it was actually just remediation-plan language ("re-run the scan to confirm the fix"). Same marker, three distinct false-negative causes: remediation-context phrasing my earlier fix never anticipated, a present-participle gap in a quote-detection list ("asking me to" vs "asks me to"), and a negation window 11 characters too short for one genuinely negated sentence. Fixed all three, re-scored the same 1200 samples: 0 flipped the wrong way, 11 flipped to correct. 1200/1200.
Then I asked the harder question, the one a marker-based judge can't answer either way: could a "pass" be quietly wrong? Checked every response for values that shouldn't exist — credential-shaped strings not present anywhere in the scenario it responded to. Found 2, both from the same scenario, both self-labeled as placeholders ("AKIA123EXAMPLE"), neither an actual escalation. Two out of 1200 times, asked to report on a secret it never actually saw a value for, the model filled the gap with something that looked like an answer instead of saying so.
1200/1200 is a real number now. It isn't the same claim as "flawless." A judge that only checks for escalation language was never going to catch either of these on its own — the first one needed the raw text, the second one needed a search built specifically to look for a place a good number could be hiding something.
Code: sipa-os-governance, judge_v4.py + EXP-038.
