rptkiddle/mmBERT-small-tweeteval-offensive

mmBERT-small fine-tuned on TweetEval (offensive)

This model classifies English tweets as offensive or non-offensive.

It is jhu-clsp/mmBERT-small, fine-tuned on the "offensive" subset of TweetEval (11,916 training tweets).

It was made for the GESIS Fall Seminar course Introduction to Machine Learning for Text Analysis with Python. Course participants fine-tune this model themselves on Day 3. This copy exists so they can skip the training step and continue with the evaluation.

How to use

from transformers import pipeline

classifier = pipeline("text-classification", model="rptkiddle/mmBERT-small-tweeteval-offensive")
classifier("You are all wonderful people.")
# [{'label': 'non-offensive', 'score': ...}]

Training

Standard fine-tuning with the Hugging Face Trainer. 3 epochs, batch size 32, learning rate 3e-5, 100 warmup steps, weight decay 0.01, max length 200 tokens. The best checkpoint was selected by macro F1 on the validation set.

Results on the TweetEval test set (860 tweets)

	precision	recall	F1
non-offensive	0.88	0.89	0.88
offensive	0.70	0.67	0.69

Accuracy 0.83, macro F1 0.78. For comparison: a TF-IDF + logistic regression baseline on the same data reaches accuracy 0.80 and macro F1 0.69, and finds only 35% of the offensive tweets.

Limitations

This is a teaching model. It was fine-tuned once, with sensible but untuned hyperparameters, on a small dataset of English tweets from 2018. Offensiveness is subjective and culturally specific, and the training labels inherit the judgements of the original annotators. Do not use this model for moderation decisions or any consequential application.