B ayes at F ig L ang 2022 Euphemism Detection shared task: Cost-Sensitive B ayesian Fine-tuning and V enn-Abers Predictors for Robust Training under Class Skewed Distributions (2022)

Abstract: Transformers have achieved a state of the art performance across most natural language processing tasks. However the performance of these models degrade when being trained on skewed class distributions (class imbalance) because training tends to be biased towards head classes with most of the data points . Classical methods that have been proposed to handle this problem (re-sampling and re-weighting) often suffer from unstable performance, poor applicability and poor calibration. In this paper, we propose to use Bayesian methods and Venn-Abers predictors for well calibrated and robust training against class imbalance. Our proposed approach improves f1-score of the baseline RoBERTa (A Robustly Optimized Bidirectional Embedding from Transformers Pretraining Approach) model by about 6 points (79.0% against 72.6%) when training with class imbalanced data.

## introduction the phenomena of skewed class distribution also known as class imbalance is ambiguous and common in most real-world datasets and natural language processing (nlp) tasks @xcite . instead of preserving an ideal uniform distribution over each category of labels, most large-scale datasets exhibit skewed class distributions with a long tail having some target distributions with significantly more observations than others @xcite . although transformer-based models @xcite have achieved a state of the art performance across several tasks in nlp, their performance tends to degrade when trained on long-tailed data. the main challenge lies in the sparsity of tail classes leading to estimation of the decision boundaries severely biased towards head classes (classes with more observations) @xcite . class imbalance problem can be tackled at either model training or model inference phases. approaches to handle class imbalance at training phase can be classified into re-weighting or resampling and those at model inference phase are mostly calibration techniques @xcite which adjusts a classifier's confidence scores without changing the internal weights or architectures @xcite of the underlying models. post-processing calibration techniques have been found to be efficient since they requires no further training of the model and are effective on multiple class imbalanced classification benchmarks in computer vision @xcite . inspired by the success of post-processing calibration techniques, we experiment with techniques that are theoretically known to produce well calibrated predictions; bayesian inference for neural networks @xcite @xcite and venn-abers predictors @xcite . we test these methods by participating in the shared task at the third workshop on figurative language processing 2022 at emnlp 2022 (conference on empirical methods in natural language processing). the training dataset exhibited a long tail distribution with 70% of the training texts containing euphemism @xcite . euphemisms are mild or indirect expressions that are used in place of more unpleasant or offensive ones common in social media data. they are used to show politeness when discussing sensitive topics or as a way to make unpleasant things sound better for example saying "laid to rest" instead of "buried" or "armed conflict" instead of "war" @xcite . with the need to curb inappropriate material on social media, people use these euphemisms to bypass media censoring software and thus automatically identifying texts containing these statements is a timely task. several computational techniques have been proposed for the euphemism task @xcite @xcite . to the best of our knowledge, this is the first attempt to combine bayesian transformers and venn-abers predictors for this task. the contributions of this work are: ## methodology the dataset @xmath0, y)} n @xmath1; d train for training the model, d validation for selecting the best models and calibration step, d test for testing our approaches. we fine-tune roberta @xcite with standard cross entropy loss and with label-distribution-aware margin loss (ldam) function @xcite . we first experiment with training our models in non-bayesian way using the standard maximum likelihood estimation and also in a bayesian way by applying bayesian layers in our neural network. the bayesian layers used for our experimentation are monte carlo dropout @xcite . to calibrate our predictions, we perform inference on the validation dataset d validation of size k with our trained model and obtained uncalibrated confidence scores denoted as {z 1 , ..., z k } for each test data point x. venn-abers predictors proceeds by fitting an isotonic regression on the set (z 1 , y 1 ), ...(z k , y k ), (z, 0) and the computing the score s(x i ) for each calibration data points (x i , y i ). let g be an increasing function on the set s(x 1 ), ...s(x k ) that maximizes the likelihood k @xmath2: the estimated label for a text data point x is the probability that minimizes the regret of the loss function calculated as in equation 6 . ## conclusion in this work, we have presented an approach for improving classification performance of transformer model when the data exhibits skewed class distributions. data exhibits skewed class distribution when majority of the data points belong to some classes while other classes have very few data points. the situation makes naive training of neural networks hard since they tend to biased towards head classes. our approach is based on cost sensitive bayesian learning with venn predictors for robust training against the class imbalance. experiments the euphemisms detection dataset which had class imbalance show that this method improves over traditional fine tuning by about 6% in terms of fscore (79.0% versus 72.6%). as future work, we would like to investigate how these finding extend beyond the euphemisms detection dataset.