Tesla at S em E val-2022 Task 4: Patronizing and Condescending Language Detection using Transformer-based Models with Data Augmentation (2022)

Abstract: This paper describes our system for Task 4 of SemEval 2022: Patronizing and Condescending Language (PCL) Detection. For sub-task 1, where the objective is to classify a text as PCL or non-PCL, we use a T5 Model fine-tuned on the dataset. For sub-task 2, which is a multi-label classification problem, we use a RoBERTa model fine-tuned on the dataset. Given that the key challenge in this task is classification on an imbalanced dataset, our models rely on an augmented dataset that we generate using paraphrasing. We found that these two models yield the best results out of all the other approaches we tried.

## introduction detecting the presence of patronizing and condescending elements in text is an important task for nlp because of the social impact it has. pcl is characterised by a superior attitude towards others, or a manner of speech that seems to portray others in a pitying way. it is different from other problems in the field of text classification such as detection of hate speech or abusive comments because it is not necessarily done on purpose. having an automated system that is capable of understanding and classifying language that contains pcl elements would be the first step towards making people and entities, such as media publications, aware of the kind of language they use when talking about vulnerable communities, and as a result, prevent discrimination, stereotypes and harm that could potentially arise from the use of such language. the pcl detection task at @xcite aims to solve this problem by exploring different systems that are capable of detecting features that indicate and categorize pcl in the don't patronize me! dataset @xcite . sub-task 1 has been formulated as a binary classification problem, where the goal is to identify whether a given text falls under the category of pcl or not. sub-task 2 is a multi-label classification problem, where a given text is either free of pcl or belongs to one or more of the seven pcl categories described in the dataset provided. this paper describes the system developed by team tesla for semeval-2022 task 4. one of the key challenges of this task is the unequal class distribution across the dataset for both sub-tasks. in this paper, we introduce a system to detect pcl using i) a t5 @xcite model for subtask-1 and ii) a roberta @xcite model for sub-task 2. for our final submission, we use these two models finetuned on an augmented dataset, which we create by generating paraphrases of the sentences belonging to the minority classes in the dataset. our system ranked 51st out of 78 teams in sub-task 1 and 27th out of 49 teams in sub-task 2, as shown in the leaderboard foot_0 . our system for sub-task 2 outperforms the roberta baseline, obtaining an average f1-score of 0.2445 versus 0.1041 for the baseline. all of our code is made publicly available on github foot_1 . ## task description the pcl detection task provides participants with the don't patronize me! dataset (pérez-almendros et al., 2020), which consists of more than 10,000 paragraphs taken from english-language news stories across 20 different countries. each of these paragraphs has been annotated to indicate the presence of pcl, and the dataset for sub-task 2 is further annotated with a category label from different classes proposed. these classes are focused on pcl towards vulnerable communities. the datasets for both the sub-tasks are not balanced. the number of non-pcl examples is nearly 10 times more than the number of pcl instances in the dataset for sub-task 1. similarly, the seven classes are not equally represented in the dataset for sub-task 2, and the number of examples that do not belong to any category exceeds those that belong to at least one category by a significant amount. ## related work transformer-based approaches such as bert @xcite , roberta @xcite , al-bert @xcite , t5 @xcite , etc. have shown an impressive performance on a wide range of nlp tasks, including text classification. they have been found to yield good results on text classification that deal with problems such as detection of hate speech @xcite and offensive language @xcite . the problem of condescending language detection is explored in the talkdown dataset @xcite , where the authors released an annotated reddit corpus of condescending linguistic acts in context, along with a bert model finetuned on the dataset as the baseline. data augmentation has been widely used across various machine learning tasks, particularly in those tasks where data is scarce, such as computer vision problems in the medical domain @xcite , @xcite . these augmentation approaches allow for a larger and more diverse training dataset. with respect to text data augmentation, @xcite discusses various approaches, both in the data space and feature space. techniques such as synonym-replacement @xcite , embedding replacement @xcite , smote @xcite , generative methods @xcite , @xcite etc. have been studied for data augmentation. many recent works have also used back-translation @xcite and paraphrasing as a way to increase training data, such as data in the areas of dialogue-generation @xcite . ## models we fine-tune a range of pretrained models, described below, given their good performance on a wide range of nlp tasks. we use the transformer model implementations provided by simpletransformers foot_3 with the default hyperparameters. bert @xcite : we try out the bert base (uncased) model, which consists of 12 transformer layers, 12 self-attention heads per layer, and a hidden size of 768. roberta @xcite : we try out the roberta base model. similar to bert base , roberta base also consists of 12 transformer layers, 12 self-attention heads per layer, and a hidden size of 768. t5 @xcite : we use the text-to-text transfer transformer (t5) released by the authors. we use the t5 base model, which has about 220 million parameters, nearly twice the number of parameters in bert base . we test all three models for sub-task 1, and for sub-task 2 we only try out bert base and roberta base . ## experiments 6.1 implementation details for both the sub-tasks, we concatenate the paragraph text and the keyword associated with each sample, to explore if certain keywords have an effect on the sample being classified as pcl or not. we train each of the models with two different conditions -using a dataset without paraphrased instances (original dataset) and a dataset with the original and the paraphrased sentences (augmented dataset). we train them for a single epoch only, to avoid overfitting since the data available is less. we use an 80:20 train-dev split, preserving the class ratio. we use this to evaluate our model performance in sub-task 1. for sub-task 2 too, we use an 80:20 split for training and evaluating the model. for our final submissions, we train the models for both sub-task 1 and sub-task 2 on the entire dataset. ## results before submitting the final models, we evaluated each of the models' performances on validation sets. ## discussion and conclusion the t5 base models that we submitted for sub-task 1 do not perform well on the test set. one reason for this could be that the t5 base should have been trained for longer than one epoch. another reason could be that bert-based approaches (bert, roberta) might be better suited for this task. in addition to this, the results of t5 base fine-tuned on the augmented dataset are not very good either, which could be due to overfitting that results from seeing many paraphrased instances of the same sentence during training. a manual inspection of the generated paraphrases also reveals that their quality needs improvement, since many of the generated paraphrases do not differ much from each other, and at times the generation gets reduced to simple oversampling. the results of sub-task 2 are encouraging and we can see a significant improvement in f1-scores across almost all classes when we consider the roberta base model trained on the augmented dataset. this confirms that augmenting the dataset through paraphrasing does have a positive effect on model performance. ensuring dissimilarity in the generated paraphrases, choosing an ideal number of paraphrases to generate, and using other techniques to handle imbalanced data such as cost-sensitive learning or augmentation through other methods (or a combination of them), might yield better results. we thus, see that identifying pcl and categorizing its occurrences is feasible despite its subjective nature, and that transformer-based approaches are capable of doing this.