JUST - DEEP at S em E val-2022 Task 4: Using Deep Learning Techniques to Reveal Patronizing and Condescending Language (2022)

Abstract: Classification of language that favors or condones vulnerable communities (e.g., refugees, homeless, widows) has been considered a challenging task and a critical step in NLP applications. Moreover, the spread of this language among people and on social media harms society and harms the people concerned. Therefore, the classification of this language is considered a significant challenge for researchers in the world. In this paper, we propose JUST-DEEP architecture to classify a text and determine if it contains any form of patronizing and condescending language (Task 4- Subtask 1). The architecture uses state-of-art pre-trained models and empowers ensembling techniques that outperform the baseline (RoBERTa) in the SemEval-2022 task4 with a 0.502 F1 score.

## introduction the language used when talking about other people significantly impacts society and individuals. talking about others and using caring and sympathetic language to express them causes them to be concerned, regardless of the author's intention, which is often to help others by raising awareness of their cause. unfair treatment of vulnerable groups on social media increases exclusion and inequality @xcite . several researchers study modeling language that intentionally undermines others, such as offensive language or hate speech @xcite . pcl modeling is still a relatively new topic of research in nlp. for illustration, the use of pcl in the media is frequently unconscious, subtler, and more subjective than the sorts of discourse that are typically addressed in nlp. to the best of our knowledge, a particular focus on pcl for vulnerable communities has not yet been considered. through a broader context, some work on pcl had been studied on communication between two people, such as in social media interactions, where others patronize an individual. for example, the authors in @xcite published the talk down corpus for detecting condescension in reddit comment-reply pairs. finding or creating a high-quality dataset that covers all or most cases of pcl is very difficult and requires significant effort from specialized researchers. as researchers in nlp, we use it to investigate pcl in vulnerable communities @xcite . in this paper, we describe our model on task 4 of semeval 2022 sub task1 @xcite to identify pcl and categorize the linguistic techniques used to express it. specifically when referring to communities identified as being vulnerable to unfair treatment in the media. we compare the outcomes of multiple pre-trained models (bert, roberta, and ensemble on them). we furthermore show the effect of tuning their hyperparameters values (batch-size and epoch) on model prediction. also, we illustrate our proposed architecture where we feed the data to an ensemble model with the stacking of 2 bert models and -in parallel-to another ensemble model with the stacking of 2 roberta models. finally, the results of the two ensembling models are fed to a max voting ensemble, which predicts the outcome with achieving the best f1 score at 0.502. the rest of the paper is organized as follows; section 2 presents a related work, and section 3 offers methodology. results are discussed in section 4. finally, section 5 presents conclusion. ## related work many researchers applied machine learning models to classify text @xcite . @xcite classified text using a recurrent convolutional neural network. they captured the key components in texts using the max-pooling layer from word representations. then they grabbed contextual information from it. to check the efficacy of the proposed method, they conducted several experiments on four commonly used data sets. as a result, they found that their suggested method outperforms the latest methods in many data sets. gunal et al. @xcite ) studied text classification algorithms on several sides. like limitations of each technique and its application in real-world situations are discussed. they found some steps useful in decreasing the time complexness and memory complexity of existing text classification algorithms like central component analysis and incidental projection, etc. using deep neural networks with lstm modules, semberecki et al. @xcite ) classified text documents. they tried to construct feature vectors, which represent the documents to be categorized: each feature vector represents the sequence of words that are included in the documents. first, they convert the terms into vector representations and then use the sequences of these vector representations as features of the documents. they evaluated the feasibility of this approach to text categorization utilizing a set of wikipedia articles. they show that the lstm network-based approach with documents represented as vectors achieves an accuracy of 86%. ## methodology our approach methodology can be summarized as follows: we begin by describing the dataset for this task. then, the preprocessing step is described. our final section describes how just-deep can identify the patronizing language in the text. ## result we performed several experiments to determine which is the best suitable model for this task and which model produces the highest value of f1 score. first, we experimented with bert and roberta pre-trained models and tuned their hyperparameters (batch_size and epoch) to achieve the maximum possible f1 score. as a first step, we explored bert and roberta's best hyperparameters, such as batch_size, epochs, and max sequence length. we use five different models: bert model, roberta model, stacking of 2 bert, stacking of 2 roberta, and just-deep model as described in the architecture section. as shown in the ta-ble1 model, just-deep achieves the best results compared to the rest of the models we tested, as it reached an f1 score value equal to 0.502. as we explained earlier, in the just-deep model, we trained the dataset with augmentation. then we applied the stacking ensemble technique separately on 2 bert models and 2 roberta models. then, we took the max voting between the two models as a result; therefore, this model produces the best results because it combines the two models that we used, bert and roberta. the other models show less favorable results; the application of the stacking ensemble technique to the 2 bert models gives the highest results after the just-deep, where the f1 result is close to 46%, followed by the model resulting from the application of the stacking ensemble technique to the 2 roberta, where it achieved results close to 0.44. then, roberta's model achieved the f1 with a score of 0.43. the worst model was bert achieved 0.42. ## conclusion deep learning helped in the development of many aspects these days, and with in-text classification, we become to have the ability to make a lot of applications related to it and so on. in this paper, we describe our just-deep model solving the semeval-2022 task 4 subtask 1 to check if the text contains any form of pcl. the just-deep model obtained an f1 score of 0.5 using ensembling of pre-trained language models bert and roberta. our strategy depends on training data set with augmentation then applying stacking ensemble technique to 2 bert and 2 roberta and take max voting between the models to achieve f1 score higher than the other experiments we conduct where we use plain bert and roberta models with different hyperparameters.