A dam R at S em E val-2023 Task 10: Solving the Class Imbalance Problem in Sexism Detection with Ensemble Learning (2023)

Abstract: The Explainable Detection of Online Sexism task presents the problem of explainable sexism detection through fine-grained categorisation of sexist cases with three subtasks. Our team experimented with different ways to combat class imbalance throughout the tasks using data augmentation and loss alteration techniques. We tackled the challenge by utilising ensembles of Transformer models trained on different datasets, which are tested to find the balance between performance and interpretability. This solution ranked us in the top 40% of teams for each of the tracks.

## introduction sexism and other forms of harassment can be commonly observed on the internet. it can not only discourage attacked groups from interacting on the internet but also pose a serious threat @xcite . especially with the growing popularity of social media platforms, the need for automation in detecting such occurrences arises. such tools already exist and are being constantly developed but still face multiple issues @xcite . one of the present problems in such systems is often the lack of reasoning and interpretation behind the prediction itself. presenting an explanation why the system has predicted a text to be sexist enhances trust in such systems and enables them to be more widely used @xcite . the explainable detection of online sexism (edos) task presents a problem of detecting sexism in english texts gathered from popular social media platforms: reddit and gab @xcite . the detection problem is proceeded by two subtasks: firstly, classifying the sexist entries into one of four categories, and finally, marking it as one of the eleven fine-grained vectors. in this work, we propose systems to tackle each of the subtasks which share the architecture and data pre-processing methods. for each problem, an ensemble of transformer-based models is built and fine-tuned on english texts concerning the subtask. models within the ensemble are trained on different sets of data. the provided edos datasets have been extended by data points from various sources concerning past research on the topic. the texts have been pre-processed and normalized to fit the schema provided with the train data to enhance the results. the performance metric chosen for the task is the macro-averaged f1 score, which treats each class equally and promotes balanced classification. our contributions consist of providing ensemblebased solutions to all subtasks ranking us in the top 40% for each of them-36th place for sexism detection, 29th for category of sexism, and 25th for fine-grained vector of sexism tracks. we also test different methods of data augmentation and architecture designs to tackle class imbalance problems. a comparison of different ensembles is presented, weighting its advantages and disadvantages over using a single model. ## background the task of explainable sexism detection consists of three subtasks. task a is the binary classification of sexism. task b considers only sexist texts and classifies them into one of four distinct categories: threats, derogation, animosity, and prejudiced discussions. task c breaks down the categories into an even more fine-grained vector of eleven possible classes. the provided training dataset consists of 14000 data points, each containing an english text and three labels for each subtask. the texts have been scraped from gab and reddit in equal proportions and labelled by three annotators with supervision from two domain experts. classes for task a are imbalanced and skewed towards the 'nonsexist' class, with 10602 entries. this translates to just 24.3% of labels belonging to the minority class. a similar challenge is present in both task b-the smallest class contains just over 9% of en-tries and task c-only 1.4% of entries belong to the 'condescending explanations or unwelcome advice' class. the texts have already been pre-processed in some regard, as the usernames and urls have been replaced with a special token. the problem of automated sexism detection in social media has already been tackled by multiple researchers in the past years as it gained traction as a crucial issue along with the development of natural language processing (nlp) techniques allowing to overcome it more efficiently. sexism identification in social networks (exist) is a shared task focused on sexism detection proposed by the organisers for the iberlef 2021 campaign @xcite . with the first edition's success, it was followed by a similar challenge in the following year which provided an overview of leading techniques for tackling such problems @xcite . the relevance of those tasks is even more considerable for the problem presented in edos, as exist also introduced a more fine-grained categorisation of sexism into five categories. the leading solutions for the latest instalment of the challenge were based on the transformer architecture, a deep neural network architecture based exclusively on attention @xcite . the authors concluded that transformer models adapted to the social media domain improve the accuracy of sexism detection. the best-performing models proved to be majorityvote ensembles of multiple models with varying hyperparameters. superior results for the task were achieved using only the text embeddings without additional features. however, waseem and hovy showed on a similar task of hate speech detection on twitter that demographic information can provide an improvement on the model @xcite . ## system overview the system proposed in this work to tackle each task consists of five mutual steps: data collection, data analysis and preprocessing, model training and development, hyperparameter tuning and model selection, and final ensemble implementation. while the utilized data, models, and implementation varies between subtasks, the methodology remains the same. starting from data collection, where additional fitting data is obtained apart from the provided resources, which is then analyzed, processed and normalized. on the combined data, multiple models with different architectures are trained or fine-tuned and undergo hyperparameter tuning on the validation dataset. based on the results on selected test sets, a subset of models is selected, and an ensemble is created. ## experimental setup and final system implementation for final experiments and implementation, the datasets introduced in section 3.2 have been extended with provided development data (2000 data points) and pre-processed with the help of the cleantext foot_4 package. after preliminary experiments have been completed, selected models have been retrained on the extended datasets using the transformers foot_5 package pipelines along with pytorch foot_6 . for validation purposes, 25% of the entries in development datasets have been selected to achieve the same class split percentage as the primary dataset. for task a, it amounted to 500 validation data points; for tasks b and c, it was 100. the similarity for task b has been recalculated on the full dataset using methods described in section 3.2 and the sentence transformers foot_7 package. ## results task c the performance of the final solution for task c was bound to the predictive capabilities of the previous model due to the architecture. nevertheless, it managed to score a macro-f1 of 0.457, ranking us as the 25th best team out of 63 submissions. in this case, the ensemble choice was fitting, outperforming the remainder of possible combinations while achieving better results than chosen models on their own. ## conclusion in this paper, we describe a system for each of the three tracks for the edos task based on soft voting ensembles of multiple transformer-based models. the solutions scored top 40% on all three subtasks highlighting the consistency. we compare different methods of data augmentation and loss function adaptation to combat class imbalance. the impact of including additional balancing data from external sources on the performance of the models is explored. it is shown that increasing the number of minority class entries can increase efficiency in that class, but the overall performance might degrade. a comparison of performance on different test sets and the variation of ensemble results are discussed in detail, weighting its benefits and drawbacks. we also explore the benefits of using field-specific pre-trained models in comparison to the base counterparts. plans for future work involve exploring and explaining the models to focus on the interpretability part of sexism detection, along with further comparisons between single-model architecture and the use of ensembles in that regard. mitigating the negative impact of augmenting the dataset with minority class entries from external sources on majority class performance would be another development step in this study.