JUST _ ONE at S em E val-2023 Task 10: Explainable Detection of Online Sexism ( EDOS ) (2023)

Abstract: The problem of online sexism, which refers to offensive content targeting women based on their gender or the intersection of their gender with one or more additional identity characteristics, such as race or religion, has become a widespread phenomenon on social media. This can include sexist comments and memes. To address this issue, the SemEval-2023 international workshop introduced the “Explainable Detection of Online Sexism Challenge”, which aims to explain the classifications given by AI models for detecting sexism. In this paper, we present the contributions of our team, JUSTONE, to all three sub-tasks of the challenge: subtask A, a binary classification task; subtask B, a four-class classification task; and subtask C, a fine-grained classification task. To accomplish this, we utilized pre-trained language models, specifically BERT and RoBERTa from Hugging Face, and a selective ensemble method in task 10 of the SemEval 2023 competition. As a result, our team achieved the following rankings and scores in different tasks: 19th out of 84 with a Macro-F1 score of 0.8538 in task A, 22nd out of 69 with a Macro-F1 score of 0.6417 in task B, and 14th out of 63 with a Macro-F1 score of 0.4774 in task C.

## introduction social media has facilitated the freedom of expression, but it has also enabled the spread of hate speech against specific communities, such as races, sexes, and ethnicities @xcite @xcite . sexism is a particularly challenging form of hate speech to address because it can be difficult to identify and significantly impacts women's lives. women are often subjected to abusive and discriminatory behaviors based on gender, religion, or other identity factors @xcite @xcite . given the seriousness of the problem, researchers in natural language processing (nlp) have been inspired to develop novel methods for sexism in text detection and classification @xcite . this paper discusses the just_one team's contribution to semeval 2023-task10. the task aimed to detect and identify various types of sexist content using gab and reddit data @xcite . we used the pre-trained language models: bert @xcite , and roberta / unlabeled roberta @xcite from hugging face, alongside the ensemble method (selective ensemble), and optimized the models' hyperparameters for each of the three subtasks (a, b, c). it is worth mentioning that our team ranked 19th out of 84 in task a, with a macro-f1 score of 0.8538, 22nd out of 69 in task b with a macro-f1 score of 0.6417, and 14th out of 63 in task c, with a macro-f1 score of 0.4774. the rest of the paper is structured as follows: a background of task setup and data description is covered in section 2. section 3 provides insights into the related work. a system overview is provided in section 4. the experiments are described in section 5; section 6 provides our results. finally, we conclude this work in section 7. ## related work the field of text classification has seen significant advancements thanks to machine learning, with a particular focus on identifying instances of sexism in text. researchers have explored various approaches in this area, including collecting sexist data and employing different techniques to detect the presence of a sexual agenda in a piece of variable name description rewire_id unique identifier for each entry text input text (post) label_sexist the label is sexist, or not label_category category of each sexist post label_vector sub-category of each sexist post text. for example, in a recent study by the authors of @xcite , three transformers were used to detect sexism and classify text into multiple categories in response to the 2022 iber-lef exist challenge. these included a monolingual t5 model for english, a multilingual bert, and the xlm-roberta model, which achieved a macro f1-score of 0.7496 for binary classification and 0.4675 for multi-class classification. the researchers further improved performance by using unsupervised pre-training with additional data and data augmentation. the authors of @xcite developed a unimodal and multimodal classifier for detecting instances of written and visual sexism in social media memes using various machine learning models. the researchers collected a benchmark dataset of 800 memes from various social media platforms and extracted visual and textual features. using these features, they developed unimodal classifiers for detecting sexism in memes based on visual or textual features and a multimodal classifier created through early and late fusion. the best binary classification performance was achieved using a support vector machine (svm) classifier with a recall of 0.81 and 0.83 for the neural network classifier. shibly and colleagues employed multiple ma-chine learning algorithms, including decision tree, logistic regression, random forest, long short-term memory, and a voting classifier, to automatically detect instances of hate speech targeting women, as detailed in @xcite . they used the metoo open-source dataset for their study and found that the voting classifier model achieved the highest f1-measure value of 0.9. in @xcite , two different strategies were suggested for creating a thematic training package that integrates manual and machine translations of arabic materials related to gender inequality and violence against women. in the semieval 2020 competition focused on identifying the offensive language in social media across multiple languages (offenseval 2: multilingual offensive language identification in social media), das and their team participated and explained their methodology in their publication cited as @xcite . the team utilized a checkpoint ensemble-based transformer model composed of ensembled gpt-2, ensembled roberta, and ensembled distilroberta to address the competition's three subtasks. additionally, the attention mask dropout technique was employed to mitigate the issue of poor composition in social media texts. the team's performance in the english edition of the competition was notable, with macro f1 scores of 0.909, 0.551, and 0.616 achieved for the three subtasks across five languages. the focus of @xcite was on hate speech in social media and the concerns around censorship, with a particular emphasis on human rights. the authors proposed new methods to identify and address discrimination while preserving freedom of expression more effectively. they introduced a novel clustering method that utilizes a neural network approach to improve the classification of hate speech. to evaluate its performance, the method was tested on the twitter hate speech dataset and the common sentiment dataset using a publicly available inclusion model. the results demonstrated an improvement of around 5 points on an f1 scale compared to the original work on a publicly available hate speech assessment dataset. the paper also acknowledges the challenges of reproducing deep learning methods and comparing the results of other studies. ## system overview the study outlines the approach for designing the system, which is tailored for each sub-task based on factors such as the task's difficulty level and dataset size. multiple experiments were conducted to ensure the design met the system's needs. this flexible approach allows for adaptation to the unique requirements of the system. the details of the work on each sub-task are presented separately as follows: ## experimental setup the labeled dataset comprises 10,000 samples from gab and 10,000 from reddit, split into 14,000 entries for training, 2,000 for development, and 4,000 for testing. additionally, 2 million unlabeled data entries augment the training stage. we evaluated multiple pre-trained models by training them on the labeled training set and testing their performance on a validation set with an 80:20 ratio. the models were trained for 3 or 4 epochs, depending on the size of the dataset, and we selected the best model checkpoint based on its macro f1 score on the development set. we took steps to prevent overfitting during this process. transfer learning was employed in our approach by reusing fine-tuned embedding parameters. this is done by saving the checkpoint of fine-tuned model embeddings from the earlier stage and utilizing it as the initialization embeddings for the next model stage. the illustration of transfer learning is shown in figure 3 . the methodology employed in this study involved running our code on the kaggle environment using python 3.9 and pytorch. for all experiments, we utilized the implementation by hugging face to import pre-trained models. we also used the focal loss to deal with the imbalanced dataset. we performed multiple experiments to fine-tune various pre-trained models, determine which ones were best suited for our sub-tasks, and identify the most appropriate hyperparameters. ## results our team performed satisfactorily for the three subtasks of semeval 2023 -task 10: explainable detection of online sexism. the roberta large model produced promising results for task a, achieving a macro-f1 score of 0.8538 and ranking 19 out of 84 teams. for task b, the selective ensemble model resulted in a macro-f1 score of 0.641 and an order of 22 out of 69, while for task c, it achieved a macro-f1 score of 0.4774 and a ranking of 14 out of 63. our participation highlights the importance and urgency of detecting online sexism. we used different approaches for each task to adapt to them. in task a, we used fine-tuned pre-trained model to get the best performance we reached. while we realized that using the ensemble approach in tasks b and c gave better results than fine-tuning, this is because we assume that ensemble approaches are more effective when there is variance in the data and avoid overfitting when fine-tuning is used with more complex data. ## conclusion this paper outlines our approach to participating in semeval 2023 -task 10: explainable detection of online sexism (edos), a crucial and relevant issue that negatively impacts women. the task attracted numerous teams this year and comprised three subtasks. we utilized several models based on pre-trained roberta and bert architecture and a selective ensemble approach. for the binary classification subtask (a), we fine-tuned the roberta model, which resulted in a macro f1 score of 0.8538. our selective ensemble model achieved a macro f1 score of 0.6417 for subtask b and 0.4774 for subtask c. in summary, our future work includes investigating alternative ensemble strategies like weighting average and majority voting to improve the performance of our edos system in multi-class subtasks (b) and (c). we also plan to explore other pre-training techniques, such