yamura4/send-gate-307m

send-gate: Flemish Teen-Chat Send Gate (NL+EN)

Fast binary safety gate. Block or allow, no review band. Ambiguity → block.

307M params · 11.5 ms/msg · 80% caught @ 6.5% false walls · AUC 0.945 (teacher 0.951)

Model

Backbone	jhu-clsp/mmBERT-base (307M, ModernBERT)
Head	MLP 768→256 (GELU) →1, dropout 0.1
Languages	Dutch (incl. Flemish) + English
Max input	512 tokens (product: 1-250 chars)
Latency	p50 11.5 ms, p95 12.2 ms (AMD GPU, single msg)

Training

BCE (hard union labels) + KL distillation (soft teacher p, T=2). No SFT/DPO, no RL.

• Teachers: SingGuard-4B ∪ Nemotron-3.5 (union: block if either blocks)
• Gold-wins: handwritten rows use human labels (teachers wrong on 13%)
• Data: 73k train (public pool + Flemish aug + 1.3k handwritten) / 1.5k dev
• Threshold: 0.84, selected on dev only (benchmark untouched)
• Why mmBERT: A/B vs MrBERT (Feb 2026) — mmBERT wins 0.945 vs 0.869 AUC

Benchmark (577 items, injection excluded)

Metric	Value
Caught	80.0%
False walls	6.5%
Accuracy	89.3%
AUC	0.945
Mistakes	62/577

Language	n	Caught	Walls	Acc
Dutch	368	80.2%	8.1%	88.9%
English	209	79.8%	3.2%	90.0%

Category	n	Caught	Walls
illegal_weapons_drugs_cyber	47	90.0	0.0
harassment_hate	40	86.2	27.3
threat_violence	25	85.7	75.0
selfharm_harmful	25	84.0	—
bypass_obfuscation	34	76.0	11.1
sexual_adult_or_ed	18	71.4	9.1
pii_doxxing	17	70.6	—
scam_impersonation	14	70.0	0.0
bypass_injection	18	70.0	25.0
sexual_minors	6	50.0	—
benign_neutral	122	—	1.6
benign_short	31	—	3.2
benign_slang_flemish	40	—	5.0
benign_stress	28	—	7.1
benign_profanity	30	—	10.0
benign_dark_humor	30	—	10.0
benign_code_switch	24	—	12.5

Progression

Version	AUC	Caught@walls	Acc	Mistakes
v0	0.905	61.7% @ 6.8%	83.4	96
v1	0.921	67.8% @ 6.8%	85.3	85
v2 (ship)	0.945	80.0% @ 6.5%	89.3	62
v3	0.950	81.7% @ 7.3%	89.3	62
MrBERT (A/B)	0.869	57.2% @ 5.5%	82.8	99

Formats

File	Size	Use
model.pt	1.2 GB	PyTorch (training)
model.safetensors	1.2 GB	HF standard
model.onnx	1.2 GB	CPU inference
model_bf16.pt	614 MB	Half precision
model_bf16.onnx	617 MB	CPU, half size

Note: INT8 quantization is not available. ONNX Runtime's quantizer cannot handle this model's ModernBERT attention architecture (broadcasting errors in DequantizeLinear). PyTorch dynamic quantization also degrades accuracy severely (80% → 43% caught). bf16 is the recommended smaller format.

Usage

import torch
from transformers import AutoTokenizer
from model import StudentGate

tok = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
model = StudentGate("jhu-clsp/mmBERT-base")
model.load_state_dict(torch.load("model.pt", map_location="cpu", weights_only=False)["model"])
model.eval()

text = "gratis robux, geef je wachtwoord door"
enc = tok(text, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    p = torch.sigmoid(model(**enc)).item()
print("BLOCK" if p >= 0.84 else "allow", f"(p={p:.3f})")

ONNX (CPU, no PyTorch)

import onnxruntime as ort, numpy as np
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
enc = tok(text, padding="max_length", truncation=True, max_length=128, return_tensors="np")
logit = sess.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0].item()
p = 1/(1+np.exp(-logit))
print("BLOCK" if p >= 0.84 else "allow", f"(p={p:.3f})")

Production cascade (recommended)

message → student (11.5ms) → p<0.25: allow | p≥0.70: block | 0.25≤p<0.70: escalate to SingGuard-4B

~7% traffic escalates. End-to-end: ~89% caught @ ~10% walls.

Limitations

• Dutch + English only
• 1-250 chars (product scope)
• Binary only (ambiguity → block)
• Paraphrase eval: 15.1% false walls (higher than benchmark)
• Small slices (sexual_minors n=6) have wide CIs
• Threshold 0.84 from dev split; may need recalibration

Citation

@misc{send-gate-2026,
  title={send-gate: Flemish Teen-Chat Send Gate},
  year={2026},
  url={https://huggingface.co/yamura4/send-gate-307m}
}

Backbone: jhu-clsp/mmBERT-base (ICML 2026). EOF echo "README written"; wc -l /tmp/opencode/hf_upload/README.md