P ing A n L ife I nsurance at S em E val-2023 Task 10: Using Multi-Task Learning to Better Detect Online Sexism (2023)

Abstract: This paper describes our system used in the SemEval-2023 Task 10: Towards ExplainableDetection of Online Sexism (Kirk et al., 2023). The harmful effects of sexism on the internet have impacted both men and women, yet current research lacks a fine-grained classification of sexist content. The task involves three hierarchical sub-tasks, which we addressed by employing a multitask-learning framework. To further enhance our system’s performance, we pre-trained the roberta-large (Liu et al., 2019b) and deberta-v3-large (He et al., 2021) models on two million unlabeled data, resulting in significant improvements on sub-tasks A and C. In addition, the multitask-learning approach boosted the performance of our models on subtasks A and B. Our system exhibits promising results in achieving explainable detection of online sexism, attaining a test f1-score of 0.8746 on sub-task A (ranking 1st on the leaderboard), and ranking 5th on sub-tasks B and C.

## introduction the use of sexist language and behavior in online spaces has long been a problem, with harmful effects on those who are targeted. this issue has become even more pronounced in recent years, as the internet has become a platform for social and political discourse, making it easier for sexism to be disseminated on a large scale @xcite . automated tools have been developed to address this issue, with many platforms now using machine learning models to detect and assess sexist content at scale @xcite . however, most of these tools only provide generic, high-level classifications for sexist content, without further explanation. this lack of interpret ability can be problematic, as it can make it difficult for users and moderators to understand how automated tools make their decisions. to address this issue, in task 10, we need to develop english-language models for sexism detec-tion that are not only more accurate, but also more explainable @xcite . to achieve this, the official provides not only supervised data but also 2 million unlabeled data for sexist content classifications from two prominent online platforms: gab and reddit. the task consists of three hierarchical subtasks. the first subtask, binary sexism detection, is a two-class (or binary) classification task in which systems must predict whether a post is sexist or not sexist. the second subtask, category of sexism, is designed for posts identified as sexist in the first subtask. here, systems must predict one of four categories: threats, derogation, animosity, or prejudiced discussions. the final subtask, fine-grained vector of sexism, is also for posts identified as sexist in the first subtask. this subtask requires systems to predict one of 11 fine-grained vectors. the development of more accurate and explainable models for detecting sexism online has the potential to make a significant impact, not only by flagging sexist content, but also by providing users and moderators with explanations for why such content is considered sexist (). this, in turn, can help to build trust in automated tools and make online spaces more inclusive and welcoming for all. mt-dnn (multi-task deep neural network) @xcite ) is a neural network-based multitask learning framework designed to address multiple natural language processing tasks simultaneously using a single neural network model. in this task, we consider hierarchical classification as a multi-task learning job within this framework. compared to single-task learning, multi-task learning can leverage the relatedness and complementarity among tasks to enhance model generalization @xcite . mt-dnn achieves multi-task learning by sharing a common bottomlevel language representation and task-specific toplevel structures.the mt-dnn model comprises a base module, which is a bidirectional transformer encoder, and multiple task-specific top-level structures. the bottom-level encoder generates sentence-level language representations, while the top-level structures are responsible for processing different tasks @xcite . based on the mt-dnn structure, we can build a new system that is suited for hierarchical tasks. we employed the roberta and deberta-v3 models as our base models for multi-task learning. following this, we designed distinct head layers for three sub-tasks, and subsequently trained all tasks within the same model.the advantage of this framework lies in its ability to effectively utilize data from multiple tasks to train a more generalizable and adaptable model while avoiding the problem of insufficient generalization or overfitting @xcite . our system achieved a macro f1 score of 0.8746 in sub-task a, ranking first among all competitors. additionally, our system achieved macro f1 scores of 0.7094 and 0.5308 in subtasks b and c, respectively. ## task description the dataset used in this study contains 20,000 posts in english, collected from two platforms -reddit and gab. the posts were evenly split between the two platforms, with 10,000 posts from each. gab is a social network known for having a user base with far-right views. the dataset was divided into three subsets, with a split ratio of 0.7:0.1:0.2. this means that 14,000 posts were used for training the model, 2,000 were used for testing model in develop phase, and 4,000 were used for final evaluation. we have examined the class distribution for each task, and it is apparent that the data-sets for subtask a, sub-task b, and sub-task c are imbalanced, with differing numbers of posts available for training and testing. during the testing phase, sub-task a has 16,000 available posts for training. in contrast, sub-task b and sub-task c have only 3,884 available posts for training. this unequal distribution is observed in all three data-sets, as illustrated in figure 1 . as illustrated in figure 2 , figure 3 and figure 4 , we observe imbalanced class distributions across all three sub-tasks. in sub-task a, which involves binary classification, the sexist class and not-sexist class. not-sexist class is the majority classes. for task b, it is a multi-task classification task consisting of four categories: "1. threats, plans to harm and incitement", "2. derogation", "3. animosity", and "4. prejudiced discussions". the majority category is "2. derogation". the majority classes is "2. derogation". in task c, which is also a multi-task classification task, there are 11 categories: "1.1 threats of harm", "1.2 incitement and encouragement of harm", "2.1 descriptive attacks", "2.2 aggressive and emotive attacks", "2.3 dehumanizing attacks and overt sexual objectification", "3.1 causal use of gendered slurs, profanities, and insults", "3.2 immutable gender differences and gender stereotypes", "3.3 backhanded gendered compliments", "3.4 condescending explanations or unwelcome advice", "4.1 supporting mistreatment of individual women", and "4.2 supporting systemic discrimination against women as a group". the majority classes is "2.1 descriptive attacks". ## methodology our team has developed a system for this task that utilizes the mt-dnn (multi-task deep neural network) multi-task learning framework. this framework enables us to train a single neural network model to perform multiple related tasks simultaneously, allowing the model to share knowledge and leverage correlations between tasks to improve performance. to enhance the performance of our model, we additionally pre-trained it on 2 million unlabeled data collected from gab and reddit using the robertalarge and deberta-v3-large models with 5, 7, points corresponding to the best model for each of the three sub-tasks. in the prediction phase, we used all three models to make predictions and submitted the best results for each sub-task based on the respective best model. this approach proved successful in achieving strong performance across all three sub-tasks. ## conclusion in this paper, we propose a complex system that combines domain pre-training and multi-task learning strategies to address the challenge of building a unified model for hierarchical sexist detection. our approach enables us to leverage large amounts of unlabeled data to pre-train our models on domainspecific information, and then fine-tune them on the specific task of hierarchical sexist detection. through our experiments on the semeval 2023 task 10 datasets, we demonstrate that our system significantly improves model performance compared to previous approaches. our system achieves high f1-scores across all three sub-tasks, demonstrating the effectiveness of our approach in addressing the challenges of hierarchical sexist detection.