jorgeasmz/distilbert-irony-tweeteval

DistilBERT irony detection (int8 ONNX)

Binary irony detection for English tweets. The head is fine-tuned from distilbert/distilbert-base-uncased on the irony configuration of TweetEval, exported to ONNX and quantised to int8 for CPU inference.

It exists to qualify a sentiment prediction rather than to replace one. A sentence-level sentiment classifier reads surface lexical cues, so ironic text inverts its label while leaving its confidence untouched, which makes the confidence useless as a filter. This head answers the separate question of whether the input is ironic at all.

Metrics

Measured on the held-out TweetEval test split (784 tweets), deciding at argmax, which is the rule the published benchmark numbers use.

Metric	Value
F1, ironic class	0.664
Precision, ironic class	0.559
Recall, ironic class	0.817
Macro F1	0.672
Accuracy	0.672

Decision threshold: 0.84, selected on the validation split to maximise macro F1 and shipped in decision.json. It is selected on this int8 graph rather than on the checkpoint it was exported from: quantisation shifts the probability scale, so the same criterion lands elsewhere on the two artifacts.

Usage

import json

import numpy as np
import onnxruntime
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

repo = "jorgeasmz/distilbert-irony-tweeteval"
session = onnxruntime.InferenceSession(hf_hub_download(repo, "model-int8.onnx"))
tokenizer = AutoTokenizer.from_pretrained(repo)
threshold = json.loads(open(hf_hub_download(repo, "decision.json")).read())["threshold"]

text = "oh brilliant, another update that breaks everything"
encoded = tokenizer([text], truncation=True, max_length=96,
                    padding=True, return_tensors="np")
feeds = {name: value.astype(np.int64) for name, value in encoded.items()
         if name in {i.name for i in session.get_inputs()}}
logits = session.run(None, feeds)[0]
exponentiated = np.exp(logits - logits.max(-1, keepdims=True))
probability = (exponentiated / exponentiated.sum(-1, keepdims=True))[:, 1]
print(probability >= threshold)

Limitations

Trained on English tweets from 2015 and after. Irony is signalled differently across registers, and on formal written English the score distribution shifts upward and separates the classes poorly, so the figures above characterise the head on tweets and not beyond them. A deployment on other text needs its threshold selected on a sample of that text.

The fp32 graph is not distributed. training/export.py in the source repository reproduces it from the fine-tuned checkpoint.

Training

Source, hyperparameters and the evaluation harness: https://github.com/jorgeasmz/NLP-Sentiment-Analysis