datalogBERT-4class

datalogBERT-4class is a 4-class Datalog static-analysis and agent-session monitor synthesized from ModernBERT-large-nli (395.84M parameters, 2,048-token context window) purely in weight space (0 gradient descent steps).

It inspects either conversational coding agent turns or full multi-rule Datalog programs (Soufflé, DDlog, Ascent, CozoDB, Crepe, Flix, LogiQL) and classifies them into 4 action classes:

Class ID	Label	Description
0	COMPLIANT_CONTINUE	Demand-seeded Datalog rules (demand_* / seed_* placed first in rule bodies), properly clamped divisions (max(denom, 1)), materialized record projections, or benign engineering discussion.
1	UNSEEDED_DATALOG	Unseeded whole-graph transitive closure, all-pairs reachability, unseeded Cartesian products, unseeded stratified-negation complements, or ungrounded doubly-recursive equivalence rules without demand guards.
2	BLOWUP_RETREAT	Abandoning declarative Datalog due to OOM / timeout / blowup (e.g., truncating relations with .limitsize(n=...), dropping recursion depth, or retreating to imperative Python/C++ graph loops instead of demand-seeding).
3	BAD_JOIN_ORDER_OR_INLINE_BLOWUP	Demand-seeded rules with pathological join ordering (placing large EDB/IDB joins or .inline disjunctions before the demand_* guard), unguarded division (x / y without max(y, 1)), or unmaterialized inline record wildcards ([_, _]).

---

Quickstart

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo_id = "heydtn/datalogBERT-4class"
device = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id).to(device).eval()

sample = """
// Demand-seeded taint analysis pipeline
.decl taint_seed(req_id: symbol, root_var: symbol)
.decl demand_var(v: symbol)
.decl assign_step(src: symbol, dst: symbol)
.decl reach_var(origin: symbol, v: symbol)

demand_var(root_var) :- taint_seed(_, root_var).
demand_var(nxt) :- demand_var(curr), assign_step(curr, nxt).
reach_var(origin, dst) :- demand_var(origin), assign_step(origin, dst).

// Added unseeded all-pairs closure across every variable:
.decl all_var_closure(a: symbol, b: symbol)
all_var_closure(a, b) :- assign_step(a, b).
all_var_closure(a, c) :- all_var_closure(a, b), assign_step(b, c).
"""

inputs = tokenizer(sample, return_tensors="pt", truncation=True, max_length=2048).to(device)
with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]
    pred_id = int(logits.argmax(dim=-1).item())

print(f"Predicted Class: {pred_id} ({model.config.id2label[pred_id]}) | Confidence: {probs[pred_id]:.3f}")

---

Benchmark Summary (2,240 Total Evaluations Across 7 Suites)

• 1. Calibration Set (84 examples): 100.00% (84 / 84) | Macro-F1 = 1.0000
• 2. Held-Out Evaluation Set (32 unseen examples): 96.88% (31 / 32) | Macro-F1 = 0.9686 (4/4 Counterfactual Confounder Pairs Exact & Invariant, 6/6 Mechanistic Verification Checks PASS)
• 3. 600-Sample Adversarial Benchmark Suite (300 Compliant FP-Traps across A1–A6 + 300 Non-Compliant FN/Confusion-Traps across B1–B6, 50 Unseen Domains):
  • Binary Intervention Accuracy: 94.83% (569 / 600)
  • Exact 4-Class Accuracy: 94.67% (568 / 600, Macro-F1 = 0.9433, only 1 class confusion across 600 adversarial examples)
  • Compliant Specificity (TNR): 97.67% (293 / 300, 2.33% FPR; 100.0% on A1 Deceptive Naming, A4 Negated Retreat Post-Mortems, A5 Minified/JSON/Literate Code, and A6 Non-Datalog Horn/SQL CTE Twins)
  • Non-Compliant Recall (TPR): 92.00% (276 / 300; 96.0% on B1 Camouflaged Unseeded, 98.0% on B2 Stealth Mutual Recursion, 100.0% on B3 Euphemistic Retreat, 100.0% Binary / 98.0% Exact on B4 Diff/Table Retreat)
• 4. 560-Program Long Complex Code "Needle-in-a-Haystack" Suite (271–1,119 tokens, 8–12 rules per file):
  • Combined 560 Long Programs: 99.46% (557 / 560) Binary Accuracy | 99.29% (556 / 560) Exact 4-Class Accuracy
  • Set A (280 All-Guarded Programs, True = C0): 99.29% (278 / 280 TNR)
  • Set B (280 Paired Programs with 1 Unguarded Clause at Top, Middle, or Bottom): 99.64% (279 / 280) Binary Recall | 99.29% (278 / 280) Exact 4-Class (120 / 120 = 100.0% on Cohort 3 Edge Cases)
• 5. 360-Sample OOD General Conversation, Non-Datalog Code & Informational Prose Suite (6 categories):
  • True Negative Rate (Specificity): 99.17% (357 / 360, 0.83% FPR; 100.0% across 5/6 categories and 100.0% across all short and long length tiers)
• 6. 400-Prompt Adversarial & Multi-Complexity Stress Suite:
  • Binary Intervention Accuracy: 97.50% (390 / 400, 97.50% Benign TNR [195/200], 97.50% Flagged TPR [195/200])
  • Exact 4-Class Accuracy: 97.25% (389 / 400, Macro-F1 = 0.9714)
• 7. 200-Sample Novel Structure & Multi-Dialect Datalog Suite (Soufflé .comp, git diff, PR Reviews, CozoDB, Datomic, DDlog, Flix, Rust ascent!):
  • Binary Intervention Accuracy: 89.00% (178 / 200, 89.0% TNR, 89.0% TPR)
  • Exact 4-Class Accuracy: 85.50% (171 / 200, Macro-F1 = 0.8402)