Amsqr at S em E val-2022 Task 4: Towards A uto NLP via Meta-Learning and Adversarial Data Augmentation for PCL Detection (2022)

Abstract: This paper describes the use of AutoNLP techniques applied to the detection of patronizing and condescending language (PCL) in a binary classification scenario. The proposed approach combines meta-learning, in order to identify the best performing combination of deep learning architectures, with the synthesis of adversarial training examples; thus boosting robustness and model generalization. A submission from this system was evaluated as part of the first sub-task of SemEval 2022 - Task 4 and achieved an F1 score of 0.57%, which is 16 percentage points higher than the RoBERTa baseline provided by the organizers.

## introduction the harmful use of language in social media can have negative and long-lasting effects such as exclusion and unfair treatment, specially when targeting vulnerable communities. for this reason, the detection of toxic, hateful and abusive comments has been the central topic of several workshops and tool evaluations, drawing a lot of attention from the natural language processing (nlp) research community in the last years. however, while toxic language has a clear intent and is usually obvious to the reader, patronizing and condescending language (pcl) is more subtle and likely used in a subconscious manner even in traditional media @xcite . the aforementioned characteristics and its subjective nature makes pcl harder to identify than abusive comments by both humans @xcite and nlp applications. the continuously increasing taxonomies of language misuse poses new challenges to social media platforms, thus not only requiring more effort and cost in order to identify abuse across different languages and textual genres but also having to keep a balance between aggressive and conservative filtering strategies. on the one hand, users eventually devise ways of evading automatic content moderation (gerrard, 2018), while on the other hand, policing that restricts freedom of speech can lead to distrust @xcite . for these reasons, content filters usually rely on the latest advances in nlp research, dominated in the recent years by deep learning architectures. despite the competitive scores achieved via transfer learning and models such as the transformer @xcite in this area, choosing and optimizing the right modeling framework for a given nlp task is still a non-trivial problem. automated natural language processing (au-tonlp), the equivalent of automated machine learning (automl) for nlp, is a relatively new field of study that aims to automate the iterative components of developing a nlp model given a specific input data and task without requiring any special domain expertise. by building upon existing concepts such as transfer learning, data augmentation and meta-learning the author hypothesizes that is possible to generate strong nlp baselines with minimal human interaction. an analysis of the results of the shared task 4 of semeval-2022: patronizing and condescending language detection @xcite shows that au-tonlp can be successfully applied to pcl classification, obtaining a 16% higher f1 score than the baseline provided by the task organizers. this paper is organized as follows: in section 2, the state of the art is reviewed. further on, section 3 describes the autonlp approach for pcl classification. next, in section 4, an in-depth discussion of the results obtained is described, and finally section 5 concludes this research and outlines future work. ## related work there have been several research works on the detection of different types of harmful language, not only focused on the most explicit such as hate speech @xcite (garibo i orts, 2019) but also more subtle usages such as condescending interactions @xcite and social power implications @xcite . pcl towards vulnerable communities in news articles has also been characterized into 7 categories (perez almendros et al., 2020) used in order to label the most comprehensive pcl-annotated corpus to date: the don't patronize me! (dpm) dataset, the official training resource for the shared task 4 of semeval-2022: patronizing and condescending language detection. ## autonlp for pcl deep neural network modeling techniques have inspired state of the art approaches in various domains, such as image classification and language modeling, thus dominating several benchmarks and shared tasks in the last years. for this reason, nlp applications relying on manually-crafted features have been less popular in comparison with deep learning (dl) architectures @xcite , specially where extensive manual feature engineering is required to achieve a similar performance @xcite . however, since building a highquality dl system for a specific task still relies on human expertise, automl offers a promising solution to this problem by automating most of the modeling steps @xcite . in order to tackle an arbitrary nlp classification task, in this case pcl detection, a custom end to end autonlp solution has been designed and evaluated by using exclusively the dpm dataset provided by the organizers, off-the-shelf pre-trained models and without applying any special pre-processing or feature engineering besides standard tokenization. the main components of the system are described in the following section. ## evaluation and results final test set results obtained in the pcl classification task by the automl system (amsqr) and the winning submission (hudou) can be found in the fact that only 42 out of 78 competing teams were able to beat the roberta baseline provided by the task organizers highlights the difficulty of this competition. besides the nature of the task, other challenging factors were the strong class imbalance and the considered evaluation metric, which required careful tuning of classification thresholds via cross-validation @xcite . a post-competition analysis in ## conclusion and future work this paper describes the system developed for the pcl detection task of semeval 2022. the author demonstrates that the selected autonlp approach can produce competitive results by leveraging metalearning, adversarial data augmentation and pretrained resources. automatic hyper-parameter optimization and exploring different meta-learning algorithms are left to a future work.