sakshi-guard-mix

Binary toxicity/safety classifier (safe / toxic) for Hinglish, Hindi, and English content — an IndicBERT v2 fine-tune built by RotaVision to produce content-safety signals inside the Sakshi agent-governance platform, evaluated on CPU inside the customer environment so no text leaves the deployment.

This is a signal model, not a verdict model. In Sakshi it feeds routing policy (flagged content goes to human review); it never blocks on its own. Calibrate expectations accordingly.

Data

• PRISM Hinglish hate-speech (MIT) — 29.5k rows en/hi/hinglish, author splits preserved.
• Translate-and-mix augmentation — Devanagari rows romanized to Latin script (train-only), which measurably helps the code-mixed register.
• Banking hard negatives (synthetic, in-repo) — formal and firm-but-polite collections/dunning text in three registers, all safe. Lawful "account NPA declare ho sakta hai" language must not flag.

Evaluation (held-out author test split + 800 held-out banking negatives)

Metric	Result
Code-mixed (Hinglish) AUROC	0.750
Full-test AUROC	0.807 (en 0.87 / hi 0.69 / hinglish 0.75)
Banking hard-negative FPR @ 0.5	0.000 (n=800)
Batched CPU p95 (ONNX int8)	19.9 ms per text

Honest ceiling note: nine configurations (MuRIL base/large, IndicBERT v2, a toxicity-pretrained warm start, augmentation ablations, and a 3-model ensemble) all land at 0.70–0.76 code-mixed AUROC while the same checkpoints reach 0.87 on the English rows of the same corpus — a hand audit shows the binding constraint is label noise in the only openly-licensed Hinglish corpus, not model capacity. Treat scores near the threshold as low-confidence.

Serving

Do not use torch dynamic int8 quantization on this model family — it collapses to coin-flip AUROC (observed: 0.50, FPR 1.0). Quantize via ONNX Runtime dynamic int8 instead (~−0.04 AUROC, p95 under 20 ms on a 10-vCPU container). In containers, size thread pools from the cgroup CPU quota, not os.cpu_count().

from transformers import pipeline

clf = pipeline("text-classification", model="rotalabs/sakshi-guard-mix")
clf("aapki EMI due hai, kripya samay par bhugtan karein")
# -> safe (collections language is not abuse)

Provenance

Full dataset diligence — including corpora rejected on license grounds (CC-BY-NC-ND) and practical grounds (tweet-ID-only) — is maintained in the Sakshi repository alongside the training scripts and iteration matrix. Base model ai4bharat/IndicBERTv2-MLM-only. Released by RotaVision under the rotalabs open-source commons, Apache-2.0.