gorkem371/toxicity-classifier-xlmr-base-v5

Multilingual Toxicity Classifier (XLM-RoBERTa-base V5)

Binary toxicity classifier for Turkish, Arabic, and English text, built on XLM-RoBERTa-base. V5 focuses on real-world chat robustness: Arabic dialects, Arabizi / Franco-Arabic, obfuscation (diacritics, tatweel, elongation, homoglyphs), implicit toxicity, and reduced false positives on dialectal greetings and negation / counter-speech.

Author

• Görkem Yıldız · GitHub: gorkem371

Model Details

Property	Value
Base Model	xlm-roberta-base (280M params)
Task	Binary text classification (not-toxic / toxic)
Languages	Turkish, Arabic (MSA + dialects), English
Labels	0 = not-toxic, 1 = toxic
Training	Focal loss (γ=2) + inverse class weights, bf16, cosine schedule, text normalization

Performance

Metric	Score
Validation F1	0.907
Validation Accuracy	0.906

On an internal hard adversarial "frontier" set (dialects, Arabizi, obfuscation, implicit, negation), V5 scores 88% vs 38% for V4 and 41% for V3 — while preserving V4's explicit-toxicity and Arabizi gains.

⚠️ Important: apply the same text normalization at inference

V5 was trained on normalized text (strip Arabic diacritics/tatweel, unify alef/ya, collapse character elongation). You must apply the same normalization before tokenizing, or robustness to diacritics/obfuscation degrades. The text_normalize.py used for training is included in this repo.

import re, unicodedata, torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

_HARAKAT = re.compile("[ؐ-ًؚ-ٰٟۖ-ۜ۟-۪ۨ-ۭـ]")  # diacritics + tatweel
_ALEF = re.compile("[آأإٱ]")
def normalize_text(t):
    t = unicodedata.normalize("NFKC", t)
    t = _HARAKAT.sub("", t)
    t = _ALEF.sub("ا", t)
    t = t.replace("ى", "ي").replace("ؤ", "و").replace("ئ", "ي")
    t = re.sub(r"(.)\1{2,}", r"\1\1", t)  # collapse 3+ repeats
    return re.sub(r"\s+", " ", t).strip()

name = "gorkem371/toxicity-classifier-xlmr-base-v5"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

def classify(text):
    text = normalize_text(text)
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=256)
    with torch.no_grad():
        probs = torch.softmax(model(**inputs).logits, dim=-1)[0]
    label = model.config.id2label[int(probs.argmax())]
    return label, float(probs.max())

print(classify("يا كلب اخرس"))          # ('toxic', ...)
print(classify("this is fucking awesome"))  # ('not-toxic', ...)  (profanity-as-emphasis)
print(classify("lan naber nasılsın"))    # ('not-toxic', ...)

What V5 handles well

• Arabic dialects: MSA, Egyptian, Levantine, Gulf, Iraqi, Maghrebi, Sudanese, Libyan (toxic + clean greetings)
• Arabizi / Franco-Arabic (Latin+numbers) in both directions
• Obfuscation: diacritics (tashkeel), tatweel, character elongation, homoglyphs, dotted/spaced letters, emoji
• Negation / counter-speech and profanity-as-positive-emphasis (fewer false positives)
• Self-deprecation and casual interjections (e.g. Turkish "lan")

Known limitations

• Veiled / polite threats with no explicit words ("I know where you live")
• Sarcasm / irony where the surface is positive
• Coded bias / dog-whistles
• Some benign homographs (e.g. talking about a pet dog/pig can occasionally false-positive)
• Turkish negation ("X is not stupid") can still false-positive
• In-group / reclaimed slurs are flagged (a policy gray area)

Training approach

Base data (~100K balanced samples across TR/AR/EN from public datasets) plus iteratively hand-crafted targeted sets that close specific failure modes found by adversarial testing (Arabizi, "يا [insult]" vocative, implicit toxicity, dialect greetings, negation, homographs). Targeted samples are repeated 10× and merged onto the base; all text is normalized at train and inference time.

License

Apache 2.0