LCT -1 at S em E val-2023 Task 10: Pre-training and Multi-task Learning for Sexism Detection and Classification (2023)

Abstract: Misogyny and sexism are growing problems in social media. Advances have been made in online sexism detection but the systems are often uninterpretable. SemEval-2023 Task 10 on Explainable Detection of Online Sexism aims at increasing explainability of the sexism detection, and our team participated in all the proposed subtasks. Our system is based on further domain-adaptive pre-training. Building on the Transformer-based models with the domain adaptation, we compare fine-tuning with multi-task learning and show that each subtask requires a different system configuration. In our experiments, multi-task learning performs on par with standard fine-tuning for sexism detection and noticeably better for coarse-grained sexism classification, while fine-tuning is preferable for fine-grained classification.

## introduction sexism has been appearing frequently in online spaces in recent years, which not only makes online spaces unfriendly but also exacerbates social prejudice and causes serious harm to targeted groups. in order to control and mitigate it, considerable efforts have been made to detect online sexism @xcite @xcite . semeval-2023 task 10 on explainable detection of online sexism (edos) @xcite aims to improve interpretability via flagging sexist content (task a) and further deciding on a particular type of sexism in the text, including 4-category (task b) and 11-category (task c) classification systems. in this paper, we present our participation in all edos subtasks. given the complexity of obtaining annotated data, the research on using additional data via dif-ferent training techniques is quite intensive. @xcite illustrated the benefits of further pre-training and introduced two methods for it: only using the task data or only using the domain data. these approaches, used individually and in combination, are shown to perform well in domain-specific tasks with relatively low costs of computing resources. safi @xcite developed a unified end-to-end neural model using a multi-task learning (mtl) approach to address the tasks of aggression and misogyny detection. @xcite pre-trained a bert-based model on 1 billion comments and fine-tuned the model with multilingual toxicity data before fine-tuning it on the target dataset. they both demonstrated that using data from similar tasks and fine-tuning the model with it in a multi-task way could improve the model's performance. inspired by the above studies, we explore the impact of further pre-training as well as multi-task learning on the edos tasks in our work. specifically, we collect several datasets that have been developed for the related tasks (section 2.1) and include them with various annotation schemes, e.g., binary labeling of hate speech (hs), categorization of target groups, fine-grained misogyny and sexism classification. first, we run experiments to choose the most suitable preprocessing for our models (section 2.2) including emoji normalization @xcite , hashtag segmentation @xcite , and masks for contents like usernames and links @xcite . next, we conduct further pre-training on the 2 million texts provided by the organizers and the other hate speech data (section 2.3). we experiment with both domain-adaptive and task-adaptive pre-training strategies proposed by @xcite . finally, we investigate whether the model can benefit from additional in-domain data (section 2.4) using the mtl approach. the contributions of this paper are as follows: ## system overview we treat all tasks as classification problems. specifically, we use pre-trained transformer-based @xcite ) models and focus on three approaches to enhance the model performance: data collection and preprocessing, additional pretraining, and mtl. ## experiments we conduct a series of fine-grained experiments on preprocessing, further pre-training and multi-task learning. we compare several preprocessing components, and the main focus of pre-training and mtl experiments is the input data. we do experiments sequentially, applying the findings from the previous step to the next ones. ## conclusion explainable models for online sexism detection are important for building a safe environment and mitigating interpretability problems. however, our results show that high performance becomes more difficult to achieve for such a complicated task as the labels become detailed and the task becomes fine-grained. we found that domain-adaptive further pretraining of a language model improves its performance on a downstream task. built on the domainadapted models, mtl and standard fine-tuning behave differently depending on the task, which means that the task formulation has a heavy impact on the model selection even in the case of in-domain data.