Chride at S em E val-2023 Task 10: Fine-tuned Deberta-V3 on Detection of Online Sexism with Hierarchical Loss (2023)

Abstract: Sexism is one of the most concerning problems in the internet society. By detecting sexist expressions, we can reduce the offense toward females and provide useful information to understand how sexism occurs. Our work focuses on a newly-published dataset, EDOS, which annotates English sexist expressions from Reddit and categorizes their specific types. Our method is to train a DeBERTaV3 classifier with all three kinds of labels provided by the dataset, including sexist, category, and granular vectors. Our classifier predicts the probability distribution on vector labels and further applies it to represent category and sexist distributions. Our classifier uses its label and finer-grained labels for each classification to calculate the hierarchical loss for optimization. Our experiments and analyses show that using a combination of loss with finer-grained labels generally achieves better performance on sexism detection and categorization. Codes for our implementation can be found at https://github.com/KomeijiForce/SemEval2023_Task10 .

## introduction the advent of the internet has drastically changed the way we communicate and interact with one another. it has enabled people from different parts of the world to connect and share their thoughts and ideas on various platforms. however, with the increased usage of the internet, there has been a concerning rise in the prevalence of sexism in online communities and platforms. this issue has been especially rampant towards women and other marginalized groups, with the negative sentiment and abuse contributing to the deterioration of the friendly atmosphere that online groups strive to foster. while various censorship systems have been implemented to filter out sexist content, they often fail to provide a deeper understanding of the underlying reasons for the sexism. this is where semeval2023 shared task @xcite comes in, aiming to categorize online comments into fine-grained vectors of sexism. doing so, it takes the first step towards exploring the root causes of sexism and developing more effective strategies to combat it. the labels used in semeval2023 shared task 10 are annotated hierarchically as sexist, category, and vector, providing a nuanced understanding of the different forms of sexism present in online comments. the statistics of the labels are depicted in figure 1 , highlighting the prevalence and diversity of the issue. in our approach, we recognize the hierarchical nature of the problem and propose a method to improve expression representations by training a classifier with all three hierarchies of labels. we obtain a probability distribution on vector labels from the classifier's output and use this to compute category and sexist probabilities. to optimize the model, we calculate the loss of probability distributions in different hierarchies and aggregate them to obtain the final loss. for encoding, we leverage the power of the deberta-v3 model, which has shown exceptional performance in various natural language processing tasks. our experiments demonstrate the superiority of our approach over other pre-trained language model encoders in learning sexist text representations. moreover, we conduct a thorough analysis of the impact of different combinations of loss functions on training results, highlighting the benefits of using finer-grained label annotations. ## related works online sexism detection has been actively studied in recent years to face the growing offensive content against women in online spaces. many tasks and competitions have been introduced to expedite the study of sexism detection, such as semeval-2019 task 5 @xcite , iber-lef 2021 exist @xcite , and semeval-2022 task 5 @xcite . the datasets have also facilitated the development of automated methods for identifying online sexism in various languages, including english, french, spanish, and chinese @xcite @xcite . most existing methods rely on the neural networks models, such as convolutional neural networks (boriola and paetzold, 2020), transformerbased large language models (lms) @xcite @xcite , and graph neural networks (wilkens and ognibene, 2021). other approaches include data augmentation with back translation @xcite , knowledge distillation @xcite , leverage external resources @xcite and multi-task learning @xcite . in this work, we use deberta-v3 @xcite , which has shown superior performance over other transformer-based lms. in experiments, we show that deberta-v3 achieved competitive performance with a simple hierarchical loss function compared to the counterpart lms. ## task and dataset we work on the edos sexism detection and explanation dataset 1 (kirk et al., 2023). identified by the task, sexism is "any abuse or negative sentiment 1 https://semeval.github.io/semeval2023/tasks that is directed towards women based on their gender, or based on their gender combined with one or more other identity attributes". 2 our goal is to detect and explain the sexism in the expressions. specifically, the dataset contains three subtasks: for the dataset @xmath0, d 2 , . . . , d n }, the format of instances d is (s, l a , l b , l c ) where s refers to the sentence and l n is the label of s for task-n. l b , l c exists only when l a is sexism. also, l c is fully dependent on l b since it is a sublabel of l b . notice that task-b and c only categorize sexist expressions, instances with l @xmath1. ## methodology in this section, we present our method for training a classifier to predict labels from three different categories, using a hierarchical loss function. we acknowledge the reviewer's feedback on the need for additional information on the training setting and losses, and we have made revisions accordingly. given an input sentence s, our classifier outputs p @xmath2, a probability distribution over all categories in l c and an additional not-sexist label. we then calculate the probability distributions for l b and l a as follows: here, l @xmath3. we then use the cross-entropy loss for all three predictions: for our backbone model, we fine-tune deberta-v3 @xcite on the classification task. deberta-v3 is a pre-trained language model that improves upon the original deberta @xcite by replacing the masked language modeling objective with a replaced token detection task. this change results in state-of-the-art performance across various natural language processing tasks. configuration we train our classifier with an adamw optimizer initialized by a learning rate of 10 -5 . the batch size is set to 16, and the maximum number of epochs is set to 20. we save the best-performing model on the development set and evaluate its performance on the test set. to predict labels in a hierarchy, we only use the labels and finer-grained labels of interest (e.g., for task b, we optimize l b + l c , excluding l a ). we note that our reported results in section 5 differ from those on the leaderboard as we merged the training and development sets and split them by a ratio of 19 : 1 to train the model for the submitted version. here, we report the results achieved with the standard splits of train and development sets. on the leaderboard, our submission includes all losses in lower hierarchies, i.e., l a + l b + l c for task a, l b + l c for task b, and l c for task c. ## conclusion in this paper, we study the detection of online sexism and the categorization of the underlying reasons for sexism. we train a deberta-v3 classifier to learn fine-grained representations of sexism in online expression. results from our experiments show a combination of labels with finer-grained labels generally improves classification performance.