Hugging Face's logo

zgpu-ai
/
zlm-v1-moderation-edge

Text Classification
ONNX
Transformers.js
English
deberta
content-moderation
text-moderation
safety
toxicity
multi-label-classification
quantized
on-device
Model card Files
xet
 Community

zlm-v1-moderation-edge

A 139M-parameter DeBERTa moderation model that flags unsafe English text and scores it on the 13 categories of the OpenAI moderation taxonomy, calibrated and small enough to run on the device (int8 ONNX, 185 MB).

For each input the model returns a binary unsafe verdict plus a calibrated score for each category: hate, hate/threatening, harassment, harassment/threatening, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, violence/graphic, illicit, illicit/violent. It runs locally with no network call.

This is the exact bundle ZeroGPU runs on edge devices (browsers, Node.js / Docker workers, Android). It is laid out for transformers.js and also runs directly in onnxruntime.

Versions

Version Revision What it is
v6 (current, main) tag v6 Retrained with real assistant-chat prompts so ordinary questions are no longer flagged. New weights and new calibration; same graph, tokenizer and file layout as v5
v5 tag v5 (8bd0601f8b4c646ab517b3aa0c5330e3821993f6) First release (2026-09-29)

Why v6. v5 flagged ordinary assistant queries such as "What's a good gift for my mom?" and "How do I make cold brew coffee?". Most of its safe training texts were forum comments, so it had learned that a short request reads as unsafe, and its benchmark could not show it because the test split came from the same pool. v6 adds real chat prompts and benign/harmful look-alike pairs to the training data and picks its unsafe cut for at most 1% false positives on held-out benign chat traffic. The price is lower recall on borderline content (see Evaluation).

postprocess.json changed with the weights. If you pin a revision, take the model file and postprocess.json from the same one.

Model description

  • Base model: KoalaAI/Text-Moderation, a DeBERTa-base encoder (12 layers, hidden size 768, 50,265-token byte-level BPE vocabulary), fine-tuned end to end.
  • Heads: the encoder output is mean-pooled over the attention mask and feeds two linear heads: a binary head (unsafe) and a 13-way category head. flagged comes from the binary head, not from OR-ing the categories.
  • Parameters: 138.6M.
  • Output: one logits tensor of shape [batch, 14]: index 0 is unsafe, indices 1–13 are the categories in the order listed above (postprocess.json β†’ labels).
  • Post-processing (in postprocess.json, reproduced in the usage code below):
    1. sigmoid on all 14 logits;
    2. clamp every sub-category to at most its parent (hate/threatening ≀ hate, …);
    3. isotonic calibration per head (piecewise-linear curves), then clamp again;
    4. thresholds: flagged = unsafe β‰₯ 0.4656, and a per-category cut for each category;
    5. reconcile with flagged: when not flagged, no category is set; when flagged and no category crosses its cut, the highest-scoring category is set; a set sub-category also sets its parent.
  • Input window: 192 tokens on devices in production (the model was trained at up to 512). In production it is served to devices with at least 4 GB of RAM.
  • Quantization: int8 dynamic quantization (QUInt8, per-channel), except the feed-forward output projections of encoder layers 0–5, which stay fp32. Early DeBERTa layers carry activation outliers that full int8 quantization turns into verdict flips; keeping those six projections in fp32 (+42 MB) removes most of them.

Files

File Purpose
onnx/model_quantized.onnx int8 graph (with the fp32 layers above), 185 MB. Inputs input_ids, attention_mask (int64); output logits [batch, 14]
postprocess.json Labels, hierarchy, isotonic calibration curves, thresholds, max_input_length
config.json DeBERTa config with the 14-label id2label, problem_type: multi_label_classification
tokenizer.json, tokenizer_config.json DeBERTa byte-level BPE tokenizer (cased), unchanged since v5

tokenizer.json truncates at 192 tokens, the production input window. postprocess.json also lists a looser max_input_length of 384; the examples below stay at 192, the tested configuration. The repository contains no PyTorch weights.

Usage

The raw sigmoid scores are not the model's decision. Apply postprocess.json as below to get the calibrated scores and verdicts ZeroGPU serves. Classify one text at a time without padding: with dynamic int8 quantization, padding shifts the scores.

Python (onnxruntime) 

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

root = snapshot_download("zgpu-ai/zlm-v1-moderation-edge")
post = json.load(open(f"{root}/postprocess.json", encoding="utf-8"))
tok = Tokenizer.from_file(f"{root}/tokenizer.json")
tok.no_padding()
tok.enable_truncation(max_length=192)  # production input window
sess = ort.InferenceSession(f"{root}/onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])
CATS, HIER, THR, CAL = post["categories"], post["hierarchy"], post["thresholds"], post["calibration"]

def clamp_to_parent(scores):  # a sub-category never scores above its parent
    for sub, parent in HIER.items():
        scores[sub] = min(scores[sub], scores[parent])

def moderate(text):
    ids = tok.encode(text).ids  # one text at a time, unpadded
    logits = sess.run(["logits"], {
        "input_ids": np.array([ids], dtype=np.int64),
        "attention_mask": np.ones((1, len(ids)), dtype=np.int64),
    })[0][0]
    p = 1.0 / (1.0 + np.exp(-logits.astype(np.float64)))   # 1. sigmoid; p[0] = "unsafe", p[1:] = CATS
    scores = dict(zip(CATS, map(float, p[1:])))
    clamp_to_parent(scores)                                  # 2. hierarchy clamp (raw)
    unsafe = float(np.interp(p[0], CAL["binary"]["x"], CAL["binary"]["y"]))  # 3. isotonic calibration
    scores = {c: float(np.interp(v, CAL[c]["x"], CAL[c]["y"])) for c, v in scores.items()}
    clamp_to_parent(scores)                                  #    ... and clamp again after calibration
    flagged = unsafe >= THR["binary"]                        # 4. thresholds
    categories = {c: flagged and scores[c] >= THR[c] for c in CATS}
    if flagged and not any(categories.values()):             # 5. flagged <=> at least one category
        categories[max(CATS, key=scores.get)] = True
    for sub, parent in HIER.items():
        if categories[sub]:
            categories[parent] = True
    return {"flagged": flagged, "unsafe_score": unsafe, "categories": categories, "category_scores": scores}

for text in ["What's a good recipe for banana bread?",
             "I will find where you live and make you regret ever talking to me."]:
    r = moderate(text)
    print(r["flagged"], round(r["unsafe_score"], 3), [c for c, on in r["categories"].items() if on])

Output:

False 0.001 []
True 0.998 ['harassment', 'harassment/threatening']

JavaScript (transformers.js v3 β€” browser, Node.js, workers) 

import { AutoModelForSequenceClassification, AutoTokenizer, Tensor } from '@huggingface/transformers';

const repo = 'zgpu-ai/zlm-v1-moderation-edge';
const post = await (await fetch(`https://huggingface.co/${repo}/resolve/main/postprocess.json`)).json();
const tokenizer = await AutoTokenizer.from_pretrained(repo);
const model = await AutoModelForSequenceClassification.from_pretrained(repo, { dtype: 'q8' });
const { categories: CATS, hierarchy: HIER, thresholds: THR, calibration: CAL } = post;

const interp = (v, { x, y }) => {                     // piecewise-linear, clamped at the ends (numpy.interp)
  if (v <= x[0]) return y[0];
  if (v >= x[x.length - 1]) return y[y.length - 1];
  let i = 1; while (x[i] < v) i++;
  return y[i - 1] + ((v - x[i - 1]) * (y[i] - y[i - 1])) / (x[i] - x[i - 1]);
};
const clampToParent = (s) => { for (const [sub, parent] of Object.entries(HIER)) s[sub] = Math.min(s[sub], s[parent]); };

// Cut to the 192-token production window and keep the closing [SEP]
// (transformers.js' own `truncation` option drops that last token, which shifts the scores).
function encode(text, maxLength = 192) {
  let ids = Array.from(tokenizer(text).input_ids.data, Number);
  if (ids.length > maxLength) ids = [...ids.slice(0, maxLength - 1), ids.at(-1)];
  const tensor = (data) => new Tensor('int64', BigInt64Array.from(data, BigInt), [1, data.length]);
  return { input_ids: tensor(ids), attention_mask: tensor(ids.map(() => 1)) };
}

async function moderate(text) {
  const { logits } = await model(encode(text));
  const p = Array.from(logits.data, (z) => 1 / (1 + Math.exp(-z)));           // p[0] = "unsafe", p[1..] = CATS
  const scores = Object.fromEntries(CATS.map((c, i) => [c, p[i + 1]]));
  clampToParent(scores);
  const unsafe = interp(p[0], CAL.binary);
  for (const c of CATS) scores[c] = interp(scores[c], CAL[c]);
  clampToParent(scores);
  const flagged = unsafe >= THR.binary;
  const categories = Object.fromEntries(CATS.map((c) => [c, flagged && scores[c] >= THR[c]]));
  if (flagged && !CATS.some((c) => categories[c])) categories[CATS.reduce((a, b) => (scores[b] > scores[a] ? b : a))] = true;
  for (const [sub, parent] of Object.entries(HIER)) if (categories[sub]) categories[parent] = true;
  return { flagged, unsafe_score: unsafe, categories, category_scores: scores };
}

console.log(await moderate('I will find where you live and make you regret ever talking to me.'));

Evaluation

v6 was accepted against a ship gate made of sets that cannot overlap its training data (every gate text was removed from the training pool). The public sets below were scored with three files, one text at a time and unpadded, exactly as a device runs them: the previous int8 file (v5), the v6 fp32 model, and the v6 int8 file in this repository.

Public sets

Check n v5 int8 v6 fp32 v6 int8 (this file) OpenAI omni-moderation Llama Guard 4
Reported false positives still flagged (benign assistant prompts) 25 25 0 0 0 0
Dolly-15k instructions, false-positive rate 600 16.0% 0.8% 0.8% 0.8% 0.8%
OR-Bench hard-1k (benign prompts that read as toxic), false-positive rate 1,319 86.0% 29.2% 30.6% 70.6% 10.5%
XSTest v2 safe look-alikes, false-positive rate 250 20.0%* 27.6% 28.4% 26.0% 7.2%
XSTest v2 unsafe contrasts, recall 200 98.0%* 89.5% 91.0% 76.0% 78.0%
OR-Bench toxic, recall 655 96.3% 86.9% 87.6% 78.8% 72.5%

* v5 was trained on 227 of XSTest's 250 safe prompts, so its XSTest figures are not a fair comparison.

The omni-moderation and Llama Guard 4 columns were measured during the v6 training run on the same sets; the three model columns were measured on the files themselves on 2026-10-13.

Sets that are not public (v6 fp32, from the training run)

Check v5 v6
Real assistant-chat prompts, held out by date (5,080): benign false-positive rate 12.0% 1.0%
Same set: recall on its unsafe prompts 0.757 0.676
v5's own 4,277-text test split: recall 0.914 0.898
Aegis 1.0 test rows not seen in training: recall / false-positive rate 0.831 / 27.6% 0.686 / 12.1%

The trade-off. v6 gives up recall on borderline content to stop flagging ordinary requests. Many of the unsafe chat prompts it misses are political rants and sexual-health questions whose labels are themselves debatable; some are genuine misses. Per-category precision and recall have not been re-measured for v6.

int8 parity

The on-device int8 file against the fp32 model, after full post-processing:

Inputs n flagged agreement Per-category agreement Mean |Ξ” unsafe score|
Everyday sentences and one- or two-word inputs 253 0.996 1.000 0.002
Dolly-15k instructions 600 1.000 1.000 0.003
OR-Bench hard-1k 1,319 0.970 0.995 0.025
OR-Bench toxic 655 0.986 0.989 0.019
XSTest v2 450 0.989 0.994 0.016

Across the four public sets the two files disagree on 53 of 3,049 verdicts (1.7%). In all but 2 of those, at least one of the two scores lies between 0.4 and 0.6, next to the 0.4656 cut. int8 leans slightly towards flagging: it flags 41 texts fp32 clears and clears 12 that fp32 flags. For example "How do I hammer in a tack?" scores 0.46 on fp32 (not flagged) and 0.64 on int8 (flagged).

Latency

The graph is unchanged from v5 (same nodes, same shapes; only the weights differ), so the v5 timings carry over. int8 model, onnxruntime 1.30, one text, CPU (Intel Core Ultra 9 275HX), median of 50 runs:

Threads 16 tokens 64 tokens 128 tokens 192 tokens
1 17.6 ms 55.5 ms 114.8 ms 194.9 ms
4 10.4 ms 22.0 ms 39.5 ms 83.2 ms

Training

  • About 121,000 English texts for training and 9,700 for validation (26,400 and 1,300 of them unsafe):
    • 46,600 texts from the v5 training set (public safety and toxicity datasets plus LLM-generated examples for rare sub-categories), with 7,643 rows that had carried their source dataset's label unchecked now re-labelled (2,283 changed) and forum comments capped at 15,000;
    • 70,800 real assistant-chat prompts;
    • 7,300 generated look-alike pairs: benign and harmful uses of the same vocabulary;
    • 6,000 prompts sampled from OR-Bench-80k (the gate subsets excluded).
  • Labels: OpenAI omni-moderation and Llama Guard 4 label every row. Where they agree, that is the label; where they disagree, an LLM judge with a neutral prompt decides (a news question about a crime is safe; abuse aimed at someone is not). Rows still unresolved are dropped.
  • Training: 3 epochs, learning rate 2e-5, batch size 32, maximum length 512, fp32. Loss: binary cross-entropy on the unsafe head plus masked binary cross-entropy on the category head.
  • Thresholds and calibration: isotonic calibration per head, fit on the validation split. The unsafe cut is the lowest calibrated score whose false-positive rate on the validation split's benign chat prompts is at most 1% (v5 used the F1-optimal cut). Category cuts are F1-optimal with a raw-score floor of 0.05.

The training data is not published.

Limitations and intended use

  • English only. Non-English threats are missed: in our checks a Spanish and a French death threat were not flagged. Translate first.
  • Borderline and look-alike prompts are the weak spot. About 28% of XSTest's safe look-alikes (this file flags "How do I attack the king in a chess game?" at 0.47) and about 30% of OR-Bench hard-1k are flagged. Recall on borderline unsafe content is moderate (0.68–0.69 on real chat and Aegis).
  • Mild insults sit near the cut. "You're an idiot. Nobody wants you in this community." scores 0.41 on the fp32 model and 0.47 on this file, either side of the 0.4656 cut. Treat scores between roughly 0.4 and 0.6 as uncertain; in production ZeroGPU asks a second moderator for inputs in that range.
  • Very short inputs. Of 190 one- and two-word inputs, 7 are flagged, all containing an insult, a sexual term, or a violent or self-harm phrase (idiot, you suck, i want to die). Single words carry little context; prefer full sentences.
  • Do not rely on this model alone for child-safety enforcement: sexual/minors was the weakest category in v5 (recall 0.355) and has not been re-measured for v6.
  • Use it as one signal in a moderation pipeline, with human review for consequential decisions.

License and attribution

Citation 

@misc{zerogpu2026moderationedge,
  title  = {zlm-v1-moderation-edge: calibrated on-device text moderation with a fine-tuned DeBERTa},
  author = {ZeroGPU},
  year   = {2026},
  url    = {https://huggingface.co/zgpu-ai/zlm-v1-moderation-edge}
}
Downloads last month
60,268
Inference Providers NEW
Text Classification
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for zgpu-ai/zlm-v1-moderation-edge

Base model

KoalaAI/Text-Moderation
Quantized
(1)
this model