L ast R esort at S em E val-2022 Task 4: Towards Patronizing and Condescending Language Detection using Pre-trained Transformer Based Models Ensembles (2022)

Abstract: This paper presents our solutions systems for Task4 at SemEval2022: Patronizing and Condescending Language Detection. This shared task contains two sub-tasks. The first sub-task is a binary classification task whose goal is to predict whether a given paragraph contains any form of patronising or condescending language(PCL). For the second sub-task, given a paragraph, we have to find which PCL categories express the condescension. Here we have a total of 7 overlapping sub-categories for PCL. Our proposed solution uses BERT based ensembled models with hard voting and techniques applied to take care of class imbalances. Our paper describes the system architecture of the submitted solution and other experiments that we conducted.

## introduction patronizing and condescending attitude in language generally denotes the writer's sense of superiority over others. if someone is patronizing or condescending, it means what they write/say is accompanied by a sense of pity or compassion. often, usage of pcl is relatively unconscious, and the intent of the writer is not to hurt a particular group or person they are referring to. so while being harmless in its intention. usage of pcl still poses a risk of harming vulnerable people or groups by stereotyping them or normalizing specific behaviour towards them. task4 at @xcite ), patronizing and condescending language detection provides two sub-tasks. the goal of sub-task1 is to identify if the given paragraph contains pcl. the goal of sub-task2 is to determine which subcategory of pcl expresses the condescension. the seven subcategories are unbalanced power relations, shallow solution, presupposition, authority voice, metaphor, compassion and the poorer, the merrier. a given paragraph can show instances of multiple subcategories. we experimented with multiple transformerbased models. we used focal loss and weighted random sampling to address the class imbalance; we also tried out ensembling models with hard voting, which improved the accuracy over the baseline models for both the sub-tasks. the paper is structured as follows: section 2: describes the dataset and related work. section 3: describes our system and model architecture. section 4 has information regarding the dataset size and splits with libraries used. section 5 discusses the findings from our experiments, and section 6 concludes our paper. ## background there has been work done to detect potentially harmful forms of language. liu et al. (2019a) used bert and lstm based models to detect offensive language in the dataset, offensive language identification dataset (olid) provided by zampieri et al. (2019) at semeval 2019. indurthi et al. (2019) used infersent (conneau et al. (2018)) semantic sentence representations to detect hate speech against immigrants and women in the dataset provided by basile et al. (2019) at semeval 2019. pcl's subtle and often unrealised nature makes its detection an arduous task for humans and artificial intelligence systems alike. there has been some recent work done when it comes to addressing pcl. @xcite presents a dataset of social media messages annotated for condescending acts in context. ## system-overview for our solution, we have relied on using pretrained transformer-based models like roberta (liu et al. (2019b)) which robustly optimizes the original bidirectional encoder representations from transformers(bert) devlin et al. (2019). it is pre-trained on much larger datasets, bigger batches and employs dynamic masking wherein a masking pattern is generated every time a sequence is fed to the model. we also experimented with the newer decoding-enhanced bert with disentangled attentionv3 ( he et al. (2021a) ), which is an improved version of the original deberta ( he et al. (2021b) ). it leverages electra style @xcite ) pretraining by replacing deberta's original mask language modelling (mlm) with a more sampleefficient pre-training task, replaced token detection (rtd), where the model is trained as a discriminator to predict whether a token in the corrupted input is either original or has been replaced by a generator. as the dataset is highly imbalanced, we use two different ways to deal with the imbalance, focal loss and weighted random sampling. ## results and discussion the results from all our experiments conducted for sub-task1 and sub-task2 can be seen in tables 4 and 3 , respectively. the models submitted in the evaluation phase are marked * in the tables, but we have shown results from all our experiments. we experimented with several models and techniques during the development and evaluation phases. we use the f1 score to judge our models, which is also the official metric. we ranked 23rd on the first task and 36th on the second task on our submitted models. we achieved better results on other models for both tasks, and the results are shown in their tables, respectively. we see that all methods, namely focal loss, weighted random sampling(wrs) and ensembling performed better than the baseline model. the 5-fold cross-validation, hard voting ensemble model with wrs achieves the best f1-score and recall score for sub-task1, more than the models where only wrs is applied. for the second task, we see the best average score from roberta model trained separately (sep) for each label with focal loss to achieve the best average f1 score. focal loss performs poorly on unbalanced power relations, which has the highest number of positive samples (716 out of 993) and performs better on imbalanced labels having a lower number of positive samples like metaphors and poorer the merrier having 197 and 40 positive samples out of 993 respectively. ## conclusion this paper presents and describes our solution system for the semeval2022 task4: towards patronizing and condescending language detection. we have applied bert based pre-trained language models roberta and deberta with hard voting ensembling techniques along with techniques to deal with imbalanced datasets like focal loss and weighted random sampling. our submitted solutions scored f1 scores of 0.5539 and 0.1456 for the two sub-tasks, respectively.