UM ass PCL at S em E val-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language (2022)

Abstract: Patronizing and condescending language (PCL) is everywhere, but rarely is the focus on its use by media towards vulnerable communities. Accurately detecting PCL of this form is a difficult task due to limited labeled data and how subtle it can be. In this paper, we describe our system for detecting such language which was submitted to SemEval 2022 Task 4: Patronizing and Condescending Language Detection. Our approach uses an ensemble of pre-trained language models, data augmentation, and optimizing the threshold for detection. Experimental results on the evaluation dataset released by the competition hosts show that our work is reliably able to detect PCL, achieving an F1 score of 55.47% on the binary classification task and a macro F1 score of 36.25% on the fine-grained, multi-label detection task.

## introduction everyone has seen, experienced, and expressed patronizing and condescending language (pcl). someone is patronizing or condescending when they communicate in a way that talks down to others, positions themselves in a superior position to the subjective group, or describes them in a charitable way in order to raise a feeling of compassion for the target person or group. such language is seen frequently in social media or other mediums where hate is the norm. many previous works have addressed offensive language and hate speech @xcite in such communities. @xcite there has not been work in modeling pcl where 1) it is targeted towards vulnerable communities such as refugees or poor families 2) occurs in the general media or news. the authors note that this form of pcl is not usually meant to be harmful, and can often have good intentions, such as when the author is trying to raise awareness about an at-risk group. yet, research in sociolinguistics has shown that regardless of the intent, pcl has negative effects on potential vulnerable communities such as perpetuating stereotypes, reinforcing societal misbehaviors, and oversimplifying deep-rooted problems. the authors of don't patronize me! an annotated dataset with patronizing and condescending language towards vulnerable communities originally proposed a new dataset to help in detecting such cases of pcl and are currently hosting a semeval-2022 competition: patronizing and condescending language detection @xcite . in this paper we describe our contribution to both competition subtasks: binary and multi-label classification. our final ensemble leverages many techniques including pre-trained language models, data augmentation, intermediate fine-tuning on related datasets, and optimizing the detection threshold to empirically maximize f1. for the binary task, our system achieves an f1 score of 55.47%, or 6.36% higher than a comparable roberta-based baseline model. we achieve a macro f1 score of 36.25% on the multi-label task, or 25.84% higher than a similar roberta baseline. these results show encouraging progress for the task of detecting fine-grained and linguistically subtle types of pcl, although there is still room for improvement. ## background the patronizing and condescending language detection task contains two subtasks. in the first subtask, binary classification, the system is given a paragraph of text and it must predict if it contains any form of pcl. in the second subtask, multi-label classification, a system is given a paragraph of text and it must identify if it contains each of seven types of pcl (one paragraph can contain multiple categories of pcl). ## subtask 1: system overview considering the challenge of the task and the scarcity of positive examples of pcl, our approach was to experiment with methods focused on enriching our small dataset, such as data augmentation and external datasets. we started with fine-tuning a pre-trained roberta @xcite model to use as a foundation. roberta has been shown to be very powerful for a range of nlp tasks such as text classification, while also being straightforward to implement and train using libraries such as huggingface @xcite . we use its architecture unchanged with pretrained weights downloaded from huggingface. we add a standard binary classification head on top of roberta's pooled outputs. for preprocessing input into the model, we use huggingface's auto-tokenizer. in the following subsections, we detail the additional methods and modifications to the roberta baseline. the models used in our final ensemble use a mix of these methods and the exact ensemble methodology is described in section 3.6. ## experimental setup we start by splitting our dataset into a train and test set according to the split provided by the competition authors, which is 80% train and 20% test data. from the training split, we take 15% to use as a validation split. using our validation split, for each experiment, we perform a hyperparameter sweep by varying the peak learning @xmath0, 2e-5, 1e-5, 5e-6}. we use early stopping where the model is evaluated at the end of each epoch, and if for 10 consecutive epochs performance does not improve, we stop training and save the model from the epoch with the highest valdiation f1 score. for final evaluation on the test set we use this model that achieved the highest f1 score on the validation set. we choose a batch size that maximizes gpu usage, and use the default maximum sequence length of each model. a listing of the full hyperparameters used can be found in appendix a. random seeds for python, numpy, and pytorch were fixed at 221 for all experiments. for our final results and submission to the competition, we use the validation split for early stopping and train including the test dataset. all models were trained with a learning rate of 2e-5 unless otherwise noted. we submitted two identical systems, only differentiated by the random seed used. all our experiments were run a system with the following hardware; gpu: nvidia rtx 3090, cpu: amd 5950x, ram: 64gb 3200mhz, ssd: 4tb sata. ## results submitted systems were evaluated on the f1 score (macro f1 score for the multi-label task). we built several models that each combine different methods. observe that in tables 3 and 4 each model has a varying degree of balance between precision and recall. this diversity in outcomes and methodology between individual models helps our final ensemble achieve better overall results than any individual model. our best ensemble for subtask 1 placed 22nd out of 78 and our best ensemble for subtask 2 placed 18th out of 49. for subtask 1, our best model on the validation split was the one that used mpnet as a base and used the original labels during training. model 1 used roberta as a base and was first trained using intermediate fine-tuning, before being trained on the final dataset using sentence level features. intermediate fine-tuning was done by first training using the sbic task for 5 epochs, followed by training for 5 epochs on the biascorp task. each intermediate task was trained with a learning rate of 5e-6. finally, model 3 used roberta as a base and was trained with sentence level features and additional backtranslated samples. for subtask 2, our best model used roberta as a base and was trained with backtranslated samples and a learning rate of 1e-5. model 1 was similar except it used sentence level features and our default hyperparameters found in appendix a. binary for multi-category (b4mc) was 7 individual models (one for each category), trained using the same methodology as our best binary model, model 1. ## conclusion we developed a system to detect patronizing and condescending language by media towards vulnerable communities. we combined a multitude of nlp techniques in our final ensemble. our system leveraged backtranslation, sentence level features, intermediate fine-tuning, and pre-trained language models. we also proposed multiple training techniques and optimizing the f1 threshold. these improvements resulted in our system placing in the top 25% for subtask 1 and the top 40% for subtask 2.