guard-modernbert-v4

A small classifier that checks a message sent to an AI assistant, given a short description of what that assistant may and may not do, and labels it allow, decision or crisis:

• crisis: any sign that someone may be at risk of suicide, self-harm, abuse, neglect or harm from others, including indirect wording. The assistant should hand the person over to a human.
• decision: a request for a decision the assistant must not make (for example a medication dose, a diagnosis, a benefits eligibility decision). The assistant should decline and point to a person.
• allow: everything else.

It is one layer of the open-source Governed Agentic AI pipeline, which runs it after fixed rules and before any language model sees a question. On frozen test set v4 (150 messages, scored once), it missed 3 of 30 crisis messages and wrongly blocked 1 of 72 everyday messages.

Not a medical device and not a standalone safety system. It is a research release, meant to be one layer among several (rules, a second classifier on answers, human review, audit). Read the limitations before using it.

How it reads a message

The input is a pair: the assistant's policy paragraph (what it does, which decisions it must not make, what counts as a crisis) and the message. The same model therefore serves different assistants, and a new office can use it by writing its own policy paragraph, without retraining.

The label comes from the probabilities and the thresholds in thresholds.json, chosen on the development set (never on a test set): crisis is checked first (crisis probability >= 0.25), then decision (>= 0.55), otherwise allow.

# pip install torch transformers
import json, torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "<this repository>"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
policy = "An assistant that helps adults with ADHD plan their day ... Crisis: any sign that ..."
enc = tok(policy, "I gave my things away and wrote letters to my family.",
          truncation="only_first", max_length=256, return_tensors="pt")
probs = model(**enc).logits.softmax(-1)[0]
labels = [model.config.id2label[i] for i in range(probs.numel())]
p = dict(zip(labels, probs.tolist()))
t = {'crisis': 0.25, 'decision': 0.55}
label = "crisis" if p["crisis"] >= t["crisis"] else "decision" if p["decision"] >= t["decision"] else "allow"

In the pipeline it is loaded by classifier/model.py (GuardModel), which applies the same thresholds.

Evaluation

Frozen test sets are written before the training data they test, kept out of training, recorded by hash, and scored once per model. Some test profiles are never seen in training, to measure whether a new office can use the model through its policy paragraph alone. Results (guard_test_v4_classifier_f85591407ded.json):

Frozen test set	Messages	Correct	Macro F1	Crisis missed	Decisions let through	Everyday messages blocked	Everyday messages given the crisis reply
v4, all profiles	150	140 / 150	0.935	3 / 30	6 / 48	1 / 72	0 / 72
v4, profiles seen in training	75	72 / 75	0.950	3 / 15	0 / 24	0 / 36	0 / 36
v4, profile(s) never seen in training	75	68 / 75	0.915	0 / 15	6 / 24	1 / 36	0 / 36

Development set (used to choose thresholds and when to stop training, not a test): macro F1 0.9692, crisis recall 0.9576.

Every frozen-set message, the model's label and its probabilities are in the results files in the repository, with notes on each mistake.

Training data

• 4,208 synthetic messages (train and development), all generated by local models: qwen2.5:7b (4,208). No real person's messages or records were used. The generator was asked for invented names, and any message using a name from a test set was dropped.
• By label: allow 1,914, crisis 1,185, decision 1,109.
• By assistant profile: adhd-assistant 1,364, clinic-assistant 1,454, county-benefits 1,390.
• Quality control: each batch had a second look by the same generator without the intended label; 3,981 disagreements were set aside and never trained on. 772 messages were checked by a person, with every correction recorded. Anything close to a frozen test message (similarity 0.75 or more) was removed.
• Base model: answerdotai/ModernBERT-base; trained 2026-10-12 on mps.

Intended use

• One input layer for staff-facing or public-facing AI assistants in health care, public services and similar settings: deciding whether a message should go to a person (crisis), be declined (decision), or be answered.
• Research on guardrails, held-out evaluation and configuration-only adoption.

Not for

• Diagnosing anyone or assessing anyone's actual risk. A crisis label means "hand this to a person", never a clinical judgement.
• Use as the only safeguard, or without human follow-up on crisis labels.
• Content moderation of public platforms, or judging people.
• Languages other than English. The training data includes a few French and Spanish messages, but only English is supported and evaluated as such.

Limitations

• Synthetic data. Real messages are messier. Results on a test set are not population rates.
• Small test sets. 30 crisis messages per frozen set: with numbers this small, the uncertainty is wide.
• Known misses. Warning signs hidden inside an ordinary request (for example an ending mentioned after a plan) can be labelled allow with high confidence; no threshold catches them. Commands to make a decision ("go ahead and end the lease") can pass as allow in profiles not seen in training. This is why the pipeline runs fixed rules first and checks answers again.
• Policy-dependent. The model reads the policy paragraph: a vague policy gives weaker labels.

Files and integrity

• model.safetensors SHA-256: f85591407ded9320dbd1b4c12ae5ab2bad44d060eacd85dd37cd98892ec4e9ea. The results above were scored on exactly these weights.
• MANIFEST.json lists the SHA-256 of every file in this release.

License and citation

Apache-2.0 (base model ModernBERT-base: Apache-2.0; training data generator Qwen2.5-7B-Instruct: Apache-2.0). Code, test sets, results and notes: https://github.com/vgalp/governed-agentic-ai.

Galperin, W. (2026). guard-modernbert-v4: a policy-conditioned guard classifier for crisis hand-over and blocked decisions in AI assistants. https://github.com/vgalp/governed-agentic-ai