shefnlp at S em E val-2023 Task 10: Compute-Efficient Category Adapters (2023)

Abstract: As social media platforms grow, so too does the volume of hate speech and negative sentiment expressed towards particular social groups. In this paper, we describe our approach to SemEval-2023 Task 10, involving the detection and classification of online sexism (abuse directed towards women), with fine-grained categorisations intended to facilitate the development of a more nuanced understanding of the ideologies and processes through which online sexism is expressed. We experiment with several approaches involving language model finetuning, class-specific adapters, and pseudo-labelling. Our best-performing models involve the training of adapters specific to each subtask category (combined via fusion layers) using a weighted loss function, in addition to performing naive pseudo-labelling on a large quantity of unlabelled data. We successfully outperform the baseline models on all 3 subtasks, placing 56th (of 84) on Task A, 43rd (of 69) on Task B,and 37th (of 63) on Task C.

## introduction sexism, as defined within this task @xcite , refers to any abuse or negative sentiment that is directed towards women based on their gender, or based on their gender combined with one or more other identity attributes. whilst sexism may, in the general case, be gender agnostic (i.e. also applies to discrimination towards men or any other identified gender), it remains true that this form of discrimination disproportionately targets women. as the userbases of social media platforms grow, and disputes regarding the censorship and moderation of particular opinions and ideologies increase, extreme negative opinions towards particular groups are allowed to proliferate. consequently, much recent research in nlp has been on the detection of various forms of online hate, including racism, homophobia, and sexism, to combat its growing prevalence @xcite . within semeval2023 task @xcite , the 3 component subtasks relate to increasing levels of granularity in the classification of sexism in short english texts sourced from reddit and gab. task a presents a binary detection task to determine whether social media posts express sexism, whilst task b splits this further into a 4way multi-class classification task regarding broad categories of sexist content (threats, derogation, animosity, and prejudiced discussions). finally, task c presents the most fine-grained level of classification, presenting an 11-class taxonomy (described as "vectors of sexism") which further splits the 4 classes from task b into specific constituents. ## background recent years have witnessed many works in the area of hate speech and offensive language detection. as the field develops, research is beginning to diverge from simple offence detection into more fine-grained classification into areas such as racism, homophobia, and sexism, thereby getting closer to identifying not only whether language is hateful, but also why @xcite . in pursuit of this goal, there have been a range of previous shared tasks focusing on such detection, including most recently trac-2 @xcite and mami @xcite . in becoming more precise in identifying types of hate speech, problems have been shown in regard to annotator agreement concerning offensive language, with the addition of more categories consequently increasing uncertainty and levels of disagreement across human raters, demonstrating the increasing difficulty of the 3 subtasks in this current semeval 2023 challenge @xcite . the winning systems from such tasks frequently employ ensemble learning involving the training of multiple separate models @xcite . in contrast to this, we focus on approaches that are more compute-efficient, primarily via the use of adapters @xcite @xcite and weighted loss to handle class imbalances. ## system overview as described in @xcite , the task data for each task were divided into training, development and test sets in a 70:10:20 ratio. we further subdivided the training data, reserving 10% for model validation; within each experiment, model checkpoints were saved following each training epoch and their performance on this validation set calculated. the best-performing checkpoint (highest macro-f1) was then restored. a further 10% of the training data was reserved for comparison between models trained under different experimental conditions. for clarity, performance results throughout this paper are reported for this selection set (sel, 7% of the original task data), the organisers' development set (dev, 10%) and, for our final submitted systems, the task test set (final, 20% of the original task data). most of our experiments (and all of our submitted systems) made use of adapters, which are lightweight, modular extensions to pretrained transformer language models introduced by @xcite . adapters consist of small modules inserted between the layers of a transformer model. the pre-trained network weights are frozen and the adapter weights initialised close to unity before they are trained, effectively enabling tuning of the information flow to suit a particular task. this configuration reduces the number of parameters which must be trained when compared with full model fine-tuning, and can be useful in a multi-task learning set-up, with distinct adapters suited to each task sharing identical pre-trained model weights. adapters may be combined in a number of structures, including parallel and sequential configurations @xcite . our experiments made use of the adapterhub framework @xcite to configure and train adapters. adapterhub is a fork of the huggingface transformers library @xcite with adapter-specific functionality, and also offers an online platform for sharing of trained adapter modules. ## conclusion in this paper, we demonstrate the methods used in our submission towards the 3 subtasks in the edos 2023 semeval challenge. specifically, we demon-strate the effectiveness of class-specific adapters and fusion layers for improving classification performance in setups where some data classes are imbalanced, in addition to the use of weighted loss for further handling these imbalances. we hope that the importance of detecting content such as sexism is appreciated by the wider community, and that further shared tasks are developed to allow more rapid progress to facilitate content moderation. we additionally hope that the approaches demonstrated here promote further use of more compute-efficient methods in shared task submissions. while they did not yield the highest performance on the competition development set, our adapter stacking experiments showed promise, and we suspect that there may be value in further refining this approach for hierarchical classification problems in the future. in recent years, some research has criticised the use of gold standard labels for subjective tasks like sexism detection, hate speech and offensive language detection, sarcasm detection, etc. annotator disagreements in such tasks often occur due to people having differing, valid opinions. the removal of these variations in opinion to establish a single 'gold standard' label could introduce bias into these labelled datasets. in the future, we will explore the effect of inter-annotator disagreement on bias in sexism detection.