ASR trans at S em E val-2022 Task 4: Ensemble of Tuned Transformer-based Models for PCL Detection (2022)

Abstract: Patronizing behavior is a subtle form of bullying and when directed towards vulnerable communities, it can arise inequalities. This paper describes our system for Task 4 of SemEval-2022: Patronizing and Condescending Language Detection (PCL). We participated in both the sub-tasks and conducted extensive experiments to analyze the effects of data augmentation and loss functions used, to tackle the problem of class imbalance. We explore whether large transformer-based models can capture the intricacies associated with PCL detection. Our solution consists of an ensemble of the RoBERTa model which is further trained on external data and other language models such as XLNeT, Ernie-2.0, and BERT. We also present the results of several problem transformation techniques such as Classifier Chains, Label Powerset, and Binary relevance for multi-label classification.

## introduction in this paper, we discuss various linguistic techniques used for detecting patronizing and condescending language (pcl). this task particularly poses a new challenge in the field of nlp because of its subjectivity, subtle usage, and requirement of world knowledge. a person is said to be condescending or patronizing when he/she uses a superior tone to talk down to people or tries to raise pity by describing their situation. even though people often use pcl with good intentions, it encourages stereotyping, discrimination and leads to greater exclusion. therefore, it is important to devise methods that facilitate the automatic detection of pcl. semeval 2022 task 4 @xcite is the first attempt to detect the usage of pcl towards vulnerable communities. it has two subtasks: in sub-task a, we need to detect if the given paragraph contains pcl or not. sub-task b aims to classify pcl text further among potential intersecting categories. the two main challenges faced in this task are extreme class imbalance in the dataset and the subtle nature of pcl present in the text which makes it hard, even for humans to classify it correctly. also, the model needs to differentiate between actual news of extremely vulnerable situations from text containing pcl. pre-trained language models such as bert, xl-net, etc., have emerged as the state-of-the-art models for many nlp tasks such as text classification, machine translation, sequence tagging, etc. however, they are trained on typical day-to-day texts. pcl text is not trivial and classifying certain classes requires some level of world knowledge and commonsense reasoning. in this paper we conduct detailed experiments using the following models: roberta @xcite , xlnet @xcite , ernie-2.0 @xcite , bertlarge @xcite , label specific attention network (lsan) @xcite and their ensembles for pcl detection. additionally, we test the classifier chain approach for multi-label classification. we achieved significant improvement over the baseline roberta model in both the sub-tasks. we were ranked 52 nd with an f1 score of 0.4421 in sub-task a and 33 rd with an f1 score of 0.1889 in sub-task b among 81 teams. we release the code for models and experiments via github foot_0 the rest of the paper is organized as follows: section 2 describes the challenge, followed by a brief literature survey. section 3 explains the proposed approach in detail, while section 4 presents the experimental details required to reproduce the results. results and analysis are shown in section 5. finally, conclusions are drawn in section 7. ## experimental setup we used pytorch @xcite and hug-gingface library @xcite for training and inference. all the models are trained on google colab. adamw @xcite optimizer with a learning rate of 2e-5 is used for training all transformer-based models and adam is used for training all the other models. the maxi-mum length of the text is limited to 200. we chose a batch size of 32 for training all the models. ## results and analysis our model for sub-task a consists of an ensemble of tuned roberta (troberta), xlnet, and ernie-2.0 models trained with binary cross-entropy apart from these models, the results of other major transformer-based models are shown in data augmentation is widely used to generate slightly variant larger datasets from the existing smaller ones. since one recurring issue among all the models we trained is overfitting, three types of data augmentation techniques are tested to address this issue. we applied contextual augmentation for labeled sentences as proposed in @xcite . in this method, we replace the words in a sentence with words predicted by a bi-directional language model. we also tested the back-translation approach proposed in @xcite and easy data augmentation (eda) techniques described in @xcite . we observed that while backtranslation and contextual augmentation slightly improved the performance of the model, the use of eda degraded it. we conclude this is because of a contextual mismatch in the text generated with eda. since random deletion is one of the techniques used in eda, it may have led to the deletion of the keywords like 'them', 'us', 'poor' etc., resulting in bad performance. as the training dataset provided for this task is small and extremely imbalanced, we focused on analyzing the effects of data augmentation and choice approach f1 macro label powerset 0.479 classifier chain 0.506 binary relevance 0.492 ## conclusion and future work we have presented the results of various experiments using pre-trained language models such as roberta, xlnet, ernie-2.0, bert-large, and their ensembles for the detection of patronizing and condescending language. for the sake of comparison, we also presented experimental results of several rnn-based and traditional machine learning models with different features such as character ngrams, elmo embeddings, etc. we also conducted experiments with different problem transformation approaches like classifier chains for multi-label classification. since the dataset for this task is imbalanced, we also tested the effect of several data augmentation techniques and loss functions. we have achieved sizeable improvements over the baseline model by task-specific training and using techniques to mitigate class imbalance. in future work, we plan to explore other forms of classifier chains to more effectively model label dependence and hierarchy.