CL - UZH at S em E val-2023 Task 10: Sexism Detection through Incremental Fine-Tuning and Multi-Task Learning with Label Descriptions (2023)

Abstract: The widespread popularity of social media has led to an increase in hateful, abusive, and sexist language, motivating methods for the automatic detection of such phenomena. The goal of the SemEval shared task Towards Explainable Detection of Online Sexism (EDOS 2023) is to detect sexism in English social media posts (subtask A), and to categorize such posts into four coarse-grained sexism categories (subtask B), and eleven fine-grained subcategories (subtask C). In this paper, we present our submitted systems for all three subtasks, based on a multi-task model that has been fine-tuned on a range of related tasks and datasets before being fine-tuned on the specific EDOS subtasks. We implement multi-task learning by formulating each task as binary pairwise text classification, where the dataset and label descriptions are given along with the input text. The results show clear improvements over a fine-tuned DeBERTa-V3 serving as a baseline leading to F1-scores of 85.9% in subtask A (rank 13/84), 64.8% in subtask B (rank 19/69), and 44.9% in subtask C (26/63).

## introduction with social media's expanding influence, there has been a rising emphasis on addressing the widespread issue of harmful language, especially sexist language @xcite @xcite . automatic content moderation and monitoring methods have become indispensable due to the sheer amount of posts and comments on social media platforms. however, the deployment of automatic methods has led to a new problem: current approaches to sexism detection rely on transformer-based language models whose inner workings, in spite of model interpretability research, generally remain opaque @xcite . this stands in contrast with the need for explainable and transparent decision processes in content moderation. the edos 2023 shared task @xcite ) focuses on the detection (subtask a), and coarse-(subtask b) and fine-grained (subtask c) categorization of sexism. the purpose of sexism categorization is to aid the explainability of sexism detection models, where categorization can serve as additional information for why a post was classified as sexist. in this paper, we present our approach for all three subtasks. the annotated data for detecting sexism is scarce compared to other natural language processing tasks and is often not publicly available. in response to this, we adopt a multi-task learning approach, where we first train a general model for the detection of hateful and abusive language, and incrementally adapt it to the target task. we implement multi-task learning via manipulation of the input, concretely by adding label descriptions, and dataset identifiers. this means that the model is presented with a pairwise text classification task where it gets a label description and a dataset identifier as a first sequence and the text to classify as the second sequence. the model then learns to predict if the label description presented in the first sequence, in the context of a dataset identifier, applies to the input text of the second sequence. figure 1 demonstrates the approach. we collect data for a range of related tasks, including hate speech detection, offensive language detection, emotion detection, stance detection on feminism and abortion, and target group detection, leading to an auxiliary training set of over 560,000 annotated examples. our method involves a three-stage training process. as a first step, we train a general abusive language detection model using all available training data. in the second step, we further fine-tune this model on all three edos subtasks, and finally, in the third step, we fine-tune the model only on the target subtask. our models obtain strong results for subtask a, a macro-f 1 score of 0.859 achieving place 13 out of 84, but rank lower in subtasks b and c, indicating the proposed approach works comparatively well with few classes during inference time, but decreases in performance, relative to other approaches, with a higher number of classes. our ablation study demonstrates that multi-task learning with label descriptions leads to clear performance improvements over a baseline consisting of deberta-v3 @xcite fine-tuned on each subtask. however, it remains unclear if there is a positive contribution from the additionally proposed dataset identifier. ## system description we formulate each edos subtask as a binary pairwise classification task where the model predicts if a given label applies to the input text. this allows us to simultaneously train on multiple datasets with different labeling schemes and a different number of distinct labels without adjusting the model architecture or having to use multiple model heads. formally, our model receives as input (1) the concatenation of a dataset identifier d @xmath0, and (2) the input text t. it predicts the probability distribution @xmath1, l j ), t)) where @xmath2. y 1 then denotes the probability that l j , given the context of d i , does apply to t. ## experiments and results ## conclusion in this paper, we presented our approaches and results for all three subtasks of the shared task towards explainable sexism detection. we developed and evaluated a multi-task learning model that is trained in three phases: (1) training a general multi-task abusive language detection model, (2) fine-tuning the model on all three edos subtasks, thus specializing it in sexism detection, and (3) fine-tuning the model only on the target subtask. we implemented the multi-task capabilities only via input manipulation, i.e., label descriptions and dataset identifiers, without modifying the model architecture or using multiple model heads. in the official shared task evaluation, our approach led to strong results on subtask a and moderately good results on subtask b and c, indicating that the method decreases more in performance with a higher number of classes than other approaches. our ablation tests demonstrate that multitask learning via label descriptions led to significant performance improvements on subtask a and large performance improvements on subtasks b and c. it remains unclear if the dataset identifier has any positive effect. overall the results show that our model for binary sexism detection is reliable, but that there is still much room for improvement in sexism categorization.