CS / NLP at S em E val-2022 Task 4: Effective Data Augmentation Methods for Patronizing Language Detection and Multi-label Classification with R o BERT a and GPT 3 (2022)

Abstract: This paper presents a combination of data augmentation methods to boost the performance of state-of-the-art transformer-based language models for Patronizing and Condescending Language (PCL) detection and multi-label PCL classification tasks. These tasks are inherently different from sentiment analysis because positive/negative hidden attitudes in the context will not necessarily be considered positive/negative for PCL tasks. The oblation study observes that the imbalance degree of PCL dataset is in the extreme range. This paper presents a modified version of the sentence paraphrasing deep learning model (PEGASUS) to tackle the limitation of maximum sequence length. The proposed algorithm has no specific maximum input length to paraphrase sequences. Our augmented underrepresented class of annotated data achieved competitive results among top-16 SemEval-2022 participants. This paper’s approaches rely on fine-tuning pretrained RoBERTa and GPT3 models such as Davinci and Curie engines with an extra-enriched PCL dataset. Furthermore, we discuss Few-Shot learning technique to overcome the limitation of low-resource NLP problems.

## introduction natural language understanding (nlu) and interpretation (nli) is a branch of natural language processing (nlp) in artificial intelligence (ai), which involves understanding and analyzing human language in-depth. recent advances in deep neural networks (dnns) have enabled nlp research scientists to achieve state-of-the-art results for tasks 1 our implementation is publicly available at https://github.com/daniel-saeedi/pcl_ detection_semeval2022 that were extremely difficult, if not impossible @xcite , @xcite . however, understanding human emotions, reactions, and uncovering hidden insights from unstructured text data such as news stories channel is still challenging. language attitudes and intentions extracting in response to the support for the marginalized and vulnerable communities is one of the emergent nlp applications. patronizing and condescending language (pcl) is a type of behavior that projects a sense of superiority to vulnerable populations @xcite . furthermore, biases and discrimination can result from patronizing attitudes, causing some people to feel unfairly treated, inadequate, unintelligent, and possibly infuriated @xcite . since raw text data extracting from web is a common data collection method, language models can learn different forms of harmful language @xcite . the pcl understanding is inherently different from sentiment analysis because positive/negative hidden attitudes in the context will not necessarily be considered positive/negative for pcl tasks. it is difficult due to the fair amount of world knowledge and commonsense reasoning required to understand this kind of language @xcite . the fine-grained idea of pcl detection towards vulnerable communities was presented by @xcite . they evaluated baseline results of nlp techniques to detect the presence of pcl and classify pcl types at the text span level. in this paper, we describe systems participating in the semeval-2022, pcl detection competition, multiple tasks of language interpretation. the competition is divided into binary classification and multi-label categorization tasks. data quality analysis led us to explore several nlp data augmentation techniques and state-of-the-art dnn architectures for these challenging tasks. our attempts to improve the performance of previous efforts ranked us 16 among 79 nlp research teams with very competitive results on the pcl detection task. our system's performance achieved 80% and 58% f1-score on the training and test datasets, respectively. in comparison, the winning system's f1-score was 65%. also, the in-depth dataset analysis revealed multi-label classification techniques commonly confused in the pcl categorization task. this paper is organized as follows. in section 2, we introduce two pcl tasks, an in-depth analysis of their datasets, and the challenges of these tasks. in section 3, we describe our different strategies to tackle discovered challenges of data quality. next, we explored text augmentation methods to fine-tune the transformer-based model for each individual task. in section 4, we discuss our applied models, the experimental setup for fine-tuning models, and their performance. finally, we conclude the paper in section 5. ## tasks definition and dataset analysis as discussed, pcl competition consists of two classification tasks, each focused on the different objectives of pcl towards underprivileged communities. figure 1 shows samples of data, their salient features, and annotated labels in training set for both tasks. the first task aims to classify a paragraph that contains pcl as an act of appearing kind or helpful but internally feeling superior to others. the second task is the investigation of the text cat-egorization problem, where each pcl-containing paragraph may belong to several pcl categories foot_0 . ## tackling data imbalance taken together, these challenges led us to approach skewed class proportion problems in the pcl dataset with various data augmentation (da) techniques in nlp @xcite . like many other nlp techniques, da is not an exact science, and understanding both dataset and task is essential. we conducted an ablation study to measure the impact of da on the performance of the system. we aimed to enhance the size of dataset to reduce the side effect of data imbalance. before trying text augmentation methods, we preprocessed the data by removing html-tags and non-alphabetic characters. then, we expanded english language contractions, e.g., from "you've" to "you have." the following subsections explain our da methods. ## model description our system is based on pre-trained transformers models on the augmented pcl dataset. we focused on exploiting superior performance of roberta and gpt3 models. ## conclusion this paper presented a system description for pcl detection and multi-label categorization tasks. our exploratory data analysis revealed annotated pcl dataset is highly imbalanced. we enhanced data quality with a combination of data augmentation methods. we presented a modified version of sentence paraphrasing deep learning model, multisentence pegasus, to tackle the limitation of maximum sequence length. the proposed algorithm has no specific maximum input length to paraphrase sequences. we evaluated the performance of the large pre-trained roberta model on the extra enriched pcl dataset. we boosted the baseline performance and achieved competitive results among the top-16 semeval-2022 participants. furthermore, we tried two models of gpt3, davinci and curie with few-shot learning technique. our investigation showed both models perform well without hyper-parameter tuning and on just a few examples of pcl. we believe these tasks have many potentials and challenges to further improve current results.