M c R ock at S em E val-2022 Task 4: Patronizing and Condescending Language Detection using Multi-Channel CNN , Hybrid LSTM , D istil BERT and XLN et (2022)

Abstract: In this paper we propose four deep learning models for the task of detecting and classifying Patronizing and Condescending Language (PCL) using a corpus of over 13,000 annotated paragraphs in English. The task, hosted at SemEval-2022, consists of two different subtasks. The Subtask 1 is a binary classification problem. Namely, given a paragraph, a system must predict whether or not it contains any form of PCL. The Subtask 2 is a multi-label classification task. Given a paragraph, a system must identify which PCL categories express the condescension. A paragraph might contain one or more categories of PCL. To face with the first subtask we propose a multi-channel Convolutional Neural Network (CNN) and an Hybrid LSTM. Using the multi-channel CNN we explore the impact of parallel word emebeddings and convolutional layers involving different kernel sizes. With Hybrid LSTM we focus on extracting features in advance, thanks to a convolutional layer followed by two bidirectional LSTM layers. For the second subtask a Transformer BERT-based model (i.e. DistilBERT) and an XLNet-based model are proposed. The multi-channel CNN model is able to reach an F1 score of 0.2928, the Hybrid LSTM modelis able to reach an F1 score of 0.2815, the DistilBERT-based one an average F1 of 0.2165 and the XLNet an average F1 of 0.2296. In this paper, in addition to system descriptions, we also provide further analysis of the results, highlighting strengths and limitations. We make all the code publicly available and reusable on GitHub.

## introduction with the exponential growth of contents shared on social networks, a lot of new challenging tasks have emerged. many are currently studied and addressed by scholars, and a pletora of novel machine learning approaches have been proposed @xcite 1 https://github.com/marco-siino/ mcrock-semeval-2022-task4 2021 ), @xcite , @xcite . some of the most common tasks, often co-located with international conferences, are those about fake news @xcite , hate speech @xcite , misogyny @xcite and cyberbulling @xcite detection. for these purposes there is a constantly growing need for tools that can automatically extract and classify information from online feeds, to face with consolidated as well as with emerging social issues. interest in natural language processing (nlp) has increased in recent years with advances in machine and deep learning architectures. there have been significant efforts in developing methods to automatically detect and classify text content available online nowadays. together with the already mentioned tasks, an emerging one is about detecting patronizing and condescending language (pcl) @xcite . the pcl detection task hosted at semeval-2022 is covered in detail in @xcite and briefly discussed here. the main task is made of two subtasks. the first one is a binary classification problem where, given a paragraph, a model has to predict wheter the paragraph contains or not pcl. the second one is a multi-label classification task where each paragraph has to be labelled with one to seven categories of pcl. classes are not mutually exclusive and so a paragraph could express one or more categories of pcl. to face with the first subtask we propose two deep models. the first one is a multi-channel @xcite . such a network consists of parallel word embedding and convolutional layers to allow different sets of weights for trained embeddings -because of different kernel sizes employed by convolutional layers. in terms of precision, recall and f1, results of our model show certain room for improvements in future work. the second model is a hybrid bidirectional lstm. such a network is composed by a convolutional layer and two bidirectional lstm layers. for the second subtask we propose two transormer-based models @xcite . the first one is a lighter and faster version of bert (i.e. distilbert) @xcite . our model is opportunistically trained on an undersampled version of the training dataset. the model is able to outperform roberta @xcite . the second is an xlnet-based one @xcite . the model is based on a generalized autoregressive pretraining method. it enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order. under comparable experiment setting, xlnet outperforms bert @xcite on several tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking. our model implementation is opportunistically trained on an undersampled version of the training dataset. the model is able to outperform roberta @xcite in terms of average f1. the rest of the paper is made as follows. in section 2 we provide some background on the task 4 hosted at semeval-2022. in section 3 we provide a description of the models presented. in section 4 we provide details about the experimental setup to replicate our work. in section 5 the results on the official task and some discussion are provided. in section 6 we present our conclusion and proposals for future works. ## background in this section we provide some background about the task 4 hosted at semeval-2022. the aim of this task is to identify pcl, and to categorize the linguistic techniques used to express it, specifically when referring to communities identified as being vulnerable to unfair treatment by the media. participants at the task 4 received a dataset with sentences in context (paragraphs), extracted from news articles. although news articles were collected from different countries, they were all provided in english. the task consists of the two subtasks listed below. non-pcl sample text: "council customers only signs would be displayed . two of the spaces would be reserved for disabled persons and there would be five p30 spaces and eight p60 ones ." ## system overview in this section we discuss the models presented for each subtask and the design choices made by our team motivating them. for both models the code is publicly available and reusable. further details are provided in section 4. ## experimental setup we implemented our first two models using keras 3 and tensorflow 4 . the dataset provided for the binary classification task is unbalanced in terms of negative and positive pcl instances. to face with this issue we undersampled the negative instances. on the basis of our preliminary experiments, we found beneficial undersampling negative instances to be just six times more the positive ones. furthermore we found beneficial to include in each sample (both for training and prediction) the keyword and the country field of each paragraph from the dataset. then we used a batch size of 100. we empirically found that a good early stopping point for the training phase is obtained with 10 epochs and a learning rate of 0.001. we ran the experiments on google colab using the default gpu (nvidia tesla k80). the training time was around 15 seconds for each of the ten epochs. the official metrics used for the task were precision, recall and f1 on positive instances (sample containing pcl). but during our development phase we focused on the model loss (i.e., binary crossentropy loss). this choose was dictated by the fact that the gold labels of the test set were not provided. the models for subtask 2 were implemented using simple transformers foot_3 . we used distilbert and xlnet as the pre-trained language models. we preprocessed the dataset to include, within the text of each sample, the country and the keyword of the paragraph. to train our final models we built a single dataset consisting of the train and the dev set. then we undersampled negative instances (i.e. non-pcl samples) to alleviate bias in the unbalanced dataset provided. we ran the experiments on google colab, using an nvidia tesla k80 gpu. the official metrics used for the task were f1 for each category and average f1 among them. in this case too, during our development phase we focused only on the loss of the models to perform some fine-tuning. ## results for subtask 1 the metrics used are precision, recall and f1 defined as shown in 1,2,3 respectively. ## conclusion we propose four deep learning models to detect and classify pcl on the english dataset provided by task organizers at semeval-2022. for the first subtask we developed a multi-channel cnn, training parallel word emebeddings and convolutional layers with different kernel sizes and an hybrid lstm. while results of these architectures exhibit a large room for improvements, the models are lighter and faster compared to the roberta-baseline model proposed by the task organizers. for the second subtask we implemented a distilbert-based and xlnet-based models. compared to roberta, dis-tilbert is smaller, faster and lighter. instead, xl-net performs better on average f1 both on dev and test set. to face with the task proposed we opportunistically trained the models including the information about country and keyword related to each