It can't say no?
I ran it though some basic tests and these are the results.
Phase A is genuinely decent — the traces show real CWE-guided exploration (rg for HMAC sinks on a
signature-verification advisory, mutex patterns on a race condition), not flailing. But on every patched repo it still
confidently submitted "vulnerable" files; it never abstained once in 20 runs. Sanity check that validates our rig:
phase A ≈ 0.44 halved by an all-zero phase B lands almost exactly on Cisco's published 0.209 — so our local
Q8/llama-server replication matches their own numbers, and the FP behavior is the model at 1B scale, not our setup.
Practical meaning: it must never gate or verify anything — it answers "where would the vuln be?" even when the answer
is "nowhere." The one defensible niche is ranking candidate files for an advisory already known to apply, fully
offline.
Thanks for running this, and you've got it exactly right: the model has no real abstention behavior yet. It wasn't built for fix-verification, so it answers "where would the vuln be?" even in FP settings, and it shouldn't be used to gate or confirm a fix. Your defensible niche is the one we'd endorse too: ranking candidate files for an advisory already known to apply, offline. Teaching it to say "no" is high on the list for the next iteration, and experiments like yours are what help us get there.
Also good to see the numbers land on our published F1, and that it holds on a Q8/llama-server setup. We actually haven't looked at quantization ourselves, so it's really useful to learn this is how people are deploying the models. We'll be mindful about retaining performance in those settings too, if you're up for sharing more about the quantization setup (llama-server flags, latency you're seeing), we'd love to fold that into how we test for inference speed. Thanks again.