discord-moderation-minilm

A 23 MB INT8 ONNX text classifier for chat moderation, built for edge deployment — it runs on a Raspberry Pi, not just a GPU box.

This is the speed variant of a pair:

	size	latency	macro F1	false positives
this model (L6)	23 MB	~2 ms	0.982	2.7%
discord-moderation-minilm-l12	34 MB	~3.7 ms	0.987	2.1%

Use this one on a Pi or at very high volume. Use L12 otherwise — a bit more accurate, still tiny.

Classes	benign · abusive · spam · flood
Size	23 MB (INT8 ONNX) · 91 MB (fp32 safetensors)
Latency	~14 ms/message, desktop CPU, one message at a time
Base	sentence-transformers/all-MiniLM-L6-v2 (22M params)
Context	128 tokens

Why this exists

Most open toxicity classifiers have two problems for chat moderation:

1. No spam or flood class. They're toxicity-only, so a moderation pipeline has to bolt on a separate heuristic. This model predicts all four classes natively.
2. They over-flag competitive banter. "I'm gonna destroy you in Rocket League" is not a threat, but a model trained on Wikipedia edit-war comments often thinks it is. False positives punish innocent users, and they're the reason moderation bots get uninstalled.

Results

Held-out test set of 2,641 messages the model never trained on. Threshold 0.85.

metric	value
macro F1	0.982
false positives (innocent messages flagged)	32/1,167 = 2.7%
false negatives (abuse missed)	28/1,474 = 1.9%
per-class F1	benign 0.974 · abusive 0.978 · spam 1.000 · flood 0.976
out-of-distribution suite (39 hand-written cases: short messages, banter, threats)	39/39
latency	~2 ms/msg, INT8 ONNX on desktop CPU

Cross-domain results, measured honestly (external datasets the model never saw):

domain	result
Reddit comments removed by real human moderators (12k sampled)	99.0–99.8% caught
real game chat (CONDA valid): explicit toxicity	73.5% caught
real game chat: normal messages	89.0% left alone

The game-chat rows are the honest weakness. This model is trained on Wikipedia-register text and transfers well to Reddit-like registers — not to gaming chat. If you deploy in a gaming community, fine-tune on domain data first: in our experiments, ~9k rows of real game chat lifted implicit game-toxicity recall from 15% → 78%. Those rows are not in the published weights for licensing reasons (this model's data diet is strictly CC0 + owned synthetic).

For scale: general-purpose LLM judges prompted for moderation — the usual alternative — measured far worse on the same yardsticks at 40–100x the size (phi3:mini, 2.2 GB: F1 82 with ~27% false positives; llama-guard3:1b: missed 70% of interpersonal abuse).

Training-data quality note: 2,466 Jigsaw rows whose human label was confidently contradicted by an LLM second-annotator across the benign/abusive boundary were dropped before training. Removing disputed labels improved false positives measurably without hurting recall at matched operating points.

Usage

ONNX (recommended — this is the 23 MB path)

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

REPO = "strictlyinsecure/discord-moderation-minilm"
model = ORTModelForSequenceClassification.from_pretrained(REPO, subfolder="onnx", file_name="model.onnx")
tok = AutoTokenizer.from_pretrained(REPO)
clf = pipeline("text-classification", model=model, tokenizer=tok, top_k=None, device=-1)

clf("i'm gonna destroy you in rocket league tonight lmao")
# → benign  (0.992)
clf("FREE NITRO for everyone, claim here bit.ly/x9k2")
# → spam    (0.993)
clf("shut up you idiot nobody likes you")
# → abusive (0.991)

Requires pip install optimum[onnxruntime].

PyTorch (fp32)

from transformers import pipeline

clf = pipeline("text-classification", model="strictlyinsecure/discord-moderation-minilm", top_k=None)
clf("FREE NITRO click here bit.ly/x9k2")
# [{'label': 'spam', 'score': 0.99}, ...]

Thresholding

The model is bimodal — it outputs ~0.99 or ~0.003 and little in between. On the held-out set, the trade-off curve is nearly flat between 0.3 and 0.9, so the threshold is not a delicate choice:

threshold	abuse caught	false positives
0.30	98.8%	38 / 399
0.85	97.4%	26 / 399
0.95	95.6%	20 / 399

Training

• Base: all-MiniLM-L6-v2, 4-class sequence classification head.
• Data: 8,000 rows — 6,000 stratified from Jigsaw Toxic Comment Classification (CC0), length-filtered to chat-sized messages, plus 2,000 synthetic chat-register rows (evasion, sarcasm, spam, flood, and hard benign negatives).
• Class-weighted loss (inverse frequency). Without it the model collapses to predicting abusive for everything — an early run scored 100% recall on abuse with 8% recall on benign, i.e. it flagged 92% of innocent messages.
• 70/15/15 split, early stopping on validation macro F1, INT8 dynamic quantization for export.
• INT8 cost 0.008 macro F1 versus fp32 — effectively free.

toxicity and harassment are merged into one abusive class. On the labeled corpus, two annotators (Jigsaw's humans and an LLM judge) disagreed about which of those two buckets an abusive message belonged in 48% of the time. That boundary was training noise, not signal, and most moderation pipelines take the same action for both.

Limitations — please read before deploying

• The spam and flood scores are optimistic. Those classes were trained on templated synthetic data, and the evaluation set draws from the same template families. They score ~100% here. Real-world spam is far more varied. Read this as "learned these patterns", not "solved spam".
• Never trained on real chat logs. The training data is Wikipedia comments plus synthetic chat. The register gap is real and measurable — see ToxBuster, which found +57 points of recall from training on actual game chat instead of Wikipedia. Expect degradation on live traffic.
• English only. Slurs and abuse in other languages will likely be missed.
• Evasion is only lightly covered. Leetspeak, unicode homoglyphs and spaced-out letters appear in training, but only from a synthetic generator — a determined evader will get through.
• Evaluated on the author's own held-out split, not a public benchmark. Numbers are not directly comparable to models evaluated elsewhere.

Intended use

A fast first-pass filter in a moderation pipeline — cheap enough to run on every message, with a human or a stronger model handling escalation. It is not a substitute for human moderators, and it should not be the sole basis for automated bans.

License & attribution

Apache-2.0. See NOTICE for the attribution required on redistribution.

Trained on the Jigsaw Toxic Comment Classification dataset (CC0). The underlying comment text originates from Wikipedia and is licensed CC-BY-SA-3.0.

Base model sentence-transformers/all-MiniLM-L6-v2 is Apache-2.0.