NLP - LTU at S em E val-2023 Task 10: The Impact of Data Augmentation and Semi-Supervised Learning Techniques on Text Classification Performance on an Imbalanced Dataset (2023)

Abstract: In this paper, we propose a methodology fortask 10 of SemEval23, focusing on detectingand classifying online sexism in social me-dia posts. The task is tackling a serious is-sue, as detecting harmful content on socialmedia platforms is crucial for mitigating theharm of these posts on users. Our solutionfor this task is based on an ensemble of fine-tuned transformer-based models (BERTweet,RoBERTa, and DeBERTa). To alleviate prob-lems related to class imbalance, and to improvethe generalization capability of our model, wealso experiment with data augmentation andsemi-supervised learning. In particular, fordata augmentation, we use back-translation, ei-ther on all classes, or on the underrepresentedclasses only. We analyze the impact of thesestrategies on the overall performance of thepipeline through extensive experiments. whilefor semi-supervised learning, we found thatwith a substantial amount of unlabelled, in-domain data available, semi-supervised learn-ing can enhance the performance of certainmodels. Our proposed method (for which thesource code is available on Github12) attainsan F 1-score of 0.8613 for sub-taskA, whichranked us 10th in the competition.

## introduction remarkable technological advancements have made it simpler for people from diverse backgrounds to interact through social media using posts and comments written in natural language. these opportunities, however, come with their own challenges. hateful content on the internet increased to such levels that manual moderation cannot possibly deal with it @xcite . thus, precise identification of harmful content on social media is vital for ensuring that such content can be discovered and dealt with, minimizing the risk of victim harm and making online platforms safer and more inclusive. detecting online sexism on social media remains a challenge in natural language processing (nlp), and the explainable detection of online sexism (edos) shared task on semeval23 @xcite addresses this problem. the task has three main sub-tasks: (i) task a; binary sexism detection, in which we determine whether a given sentence contains sexist content, (ii) task b; sexism classification, which places sexist sentences into four categories: threats, derogation, animosity, and prejudiced discussions, and (iii) task c; fine-grained vector of sexism, an eleven-class categorization for sexist posts in which systems must predict one of 11 fine-grained vectors. one major challenge of this task is the imbalanced class distribution. for instance, sub-task a consists of only 3398 sexist posts, and 10602 nonsexist ones. using an imbalanced dataset to train models can result in prediction bias towards the majority class @xcite . in this paper, we (team nlp-ltu) present the automatic sexism detection system developed and submitted to semeval23 task 10; edos. the objective of this study is (i) to examine how different state-of-the-art pre-trained language models (plm) perform in sexism detection and classification tasks, and (ii) to contribute towards answering the following research question (rq): to what extent can data augmentation improve the results and address the data imbalance problem? the core of our approach is a voting-based ensemble model consisting of three pre-trained language models: bertweet-large @xcite , deberta-v3-large @xcite , and roberta-large @xcite . additionally, in order to address the issue of data imbalance and to expand our dataset, our system's pipeline employed techniques such as data augmentation and semi-supervised learning. we achieved competi-tive results, ranking us in the top ten for task a. 3 our results suggest that (i) using plms trained on domain-specific data (e.g. bertweet-large) leads to better results than using plms pre-trained on other sources (ii) in most cases extending all classes via augmentation leads to higher classification scores than using augmentation on the minority classes only to completely balance the class distribution. however, drawing conclusive inferences would require further experiments with multiple data augmentation methods and datasets. (iii) with a substantial amount of unlabelled, in-domain data available, semi-supervised learning can enhance the performance of certain models. the rest of the paper is organised as follows: in section 2, we present prior related work; in section 3, we discuss the proposed system. then, we describe the experiments in section 4. section 5, presents results and error analysis. finally, we conclude the work in section 6 and describe what further has to be done. ## related work in the following section we discuss already existing efforts on the detection of sexism, and efforts directed at data augmentation. ## system overview this section outlines the system pipeline employed in our study, as depicted in figure 1 . the proposed approach entails two main stages, generating additional training samples (module 1), and classification (module 2). each is described in its own subsection below. ## experimental setup all experiments have been implemented using the pytorch and huggingface libraries @xcite on an 8 32gb nvidia v100 gpu-equipped dgx-1 cluster. the server contains 80 cpu cores with the ubuntu 18 operating system. when evaluating solutions, the macro-averaged f 1-score was the primary metric. ## conclusion this paper presents our solution to the shared task on explainable detection of online sexism at semeval23. our approach involved employing ensemble voting techniques with previously finetuned language models, specifically bertweetlarge, roberta-large, and deberta-v3-large, which resulted in the best performance for both task a and b. additionally, we discovered that fine-tuning roberta-large was the most effective approach for addressing task c, outperforming the ensemble voting method. these findings address the first objective of examining how different state-of-the-art transformer-based models perform in sexism detection and classification. to address our research question; (rq): to what extent can data augmentation improve the results and address the data imbalance problem, we employed a task agnostic data augmentation method, specifically back-translation, in two scenarios: one to double the dataset and the other to augment the underrepresented class. our results showed that augmenting all classes was more effective than balancing the dataset by augmenting only the underrepresented class, which motivates further exploration of the effects of data augmentation on text classification with unbalanced datasets. in future research, we plan to explore alternative data augmentation techniques to produce more diverse sentences, such as utilizing generative models like gpt-2, to balance and double the dataset's size, and compare the results with the back-translation method. moreover, we plan to investigate why augmenting all classes sometimes was more effective than augmenting only the underrepresented class and balancing the dataset.