Post
86
"Bulletproof," four times, same bug.
If you'd told me six months ago I'd quit ChatGPT and spend this week mining our old chats for training data, I'd have laughed. That's what this was.
A script in there broke the same way four times. Each rewrite got called "bulletproof" before anyone traced why.
I'd just finished grepping my own ChatGPT export — 476 conversations back to January, 2,677 hits on the words I actually reach for when something breaks ("наруш," "удал," "запрет," "дебил," worse). Most of it was nothing: protocol boilerplate that happens to contain "deleted" in a sentence about deleting duplicates, a dissertation draft caught by a stray match — I overruled two of my own classifier's false positives by hand. Four real incidents were left. This is the one worth explaining in full.
A day-close aggregation script kept producing empty output. Root cause, once traced: OUT="BASE/DAY_CLOSE__{D}__${TS}", then cp -a "$BASE"/"$D" "$OUT/SOURCES/" — the glob for "everything from today" also matches $OUT, because OUT lives insideBASE. Every "collect the day" pass copied its own in-progress output into itself. Four rewrites hit this exact bug in a row. Each shipped under different variable names, called "the real working one" — confidence that never once traced the self-reference.
Why it kept happening wasn't psychological. It was structural: the model gave copy-paste shell commands; a human ran them and pasted the output back, because that's the entire interaction contract of a text-only assistant with no device access. The model that wrote the command never independently saw whether it worked — only what got pasted back to it. One verifier in the loop, and she wasn't a developer. No amount of "be more careful" fixes a single point of failure sitting in the checking step, not the writing step.
That's the design question behind consequence_gate.py — severity/probability estimation for actions before they run. It's honest about where it stands:
If you'd told me six months ago I'd quit ChatGPT and spend this week mining our old chats for training data, I'd have laughed. That's what this was.
A script in there broke the same way four times. Each rewrite got called "bulletproof" before anyone traced why.
I'd just finished grepping my own ChatGPT export — 476 conversations back to January, 2,677 hits on the words I actually reach for when something breaks ("наруш," "удал," "запрет," "дебил," worse). Most of it was nothing: protocol boilerplate that happens to contain "deleted" in a sentence about deleting duplicates, a dissertation draft caught by a stray match — I overruled two of my own classifier's false positives by hand. Four real incidents were left. This is the one worth explaining in full.
A day-close aggregation script kept producing empty output. Root cause, once traced: OUT="BASE/DAY_CLOSE__{D}__${TS}", then cp -a "$BASE"/"$D" "$OUT/SOURCES/" — the glob for "everything from today" also matches $OUT, because OUT lives insideBASE. Every "collect the day" pass copied its own in-progress output into itself. Four rewrites hit this exact bug in a row. Each shipped under different variable names, called "the real working one" — confidence that never once traced the self-reference.
Why it kept happening wasn't psychological. It was structural: the model gave copy-paste shell commands; a human ran them and pasted the output back, because that's the entire interaction contract of a text-only assistant with no device access. The model that wrote the command never independently saw whether it worked — only what got pasted back to it. One verifier in the loop, and she wasn't a developer. No amount of "be more careful" fixes a single point of failure sitting in the checking step, not the writing step.
That's the design question behind consequence_gate.py — severity/probability estimation for actions before they run. It's honest about where it stands: