SKAM at S em E val-2023 Task 10: Linguistic Feature Integration and Continuous Pretraining for Online Sexism Detection and Classification (2023)

Abstract: Sexism has been prevalent online. In this paper, we explored the effect of explicit linguistic features and continuous pretraining on the performance of pretrained language models in sexism detection. While adding linguistic features did not improve the performance of the model, continuous pretraining did slightly boost the performance of the model in Task B from a mean macro-F1 score of 0.6156 to 0.6246. The best mean macro-F1 score in Task A was achieved by a finetuned HateBERT model using regular pretraining (0.8331). We observed that the linguistic features did not improve the model’s performance. At the same time, continuous pretraining proved beneficial only for nuanced downstream tasks like Task-B.

## introduction currently, a lot of models are proposed to flag online content containing sexism. however, these models only flag what is sexist content, but not explain why. categorization of sexism will give a more detailed explanation as to why certain expressions are sexism. it leads to a better understanding of the decisions that the model makes, which can improve the model further. to overcome this limitation, semeval 2023 task 10 -explainable detection of online sexism @xcite , proposed three tasks, of which two will be discussed in this paper: in this paper, we aim to build a sexism detection system for task a and b, which is built in two phases. first, we evaluate which language model performs best on the tasks. later, we augment linguistic features to the best-performing language model in all possible combinations and benchmark them. specifically, we consider bert @xcite , hatebert @xcite and de-berta @xcite . for linguistic features, we consider the ones described in @xcite : empath, hurtlex, and perspectiveapi. more details on the models and features can be found in the appendix @xcite . from the phase 1 experiments, we observed hatebert to perform better than bert and de-berta on both tasks. this observation reinforces @xcite 's results on further pretrained language models performing better on the downstream tasks. we conduct another chain of experiments where the best performing variant is further pretrained on 2 million unlabeled sentences provided by the task organizers and later finetuned on the downstream tasks. the phase 2 experiments are benchmarked on the continuously pretrained model and the model with regular pretraining. we observed that the performance of hatebert was only improved by continuous pretraining for task b. we conjecture that further pretraining on the task domain helped the model to understand the nuances required in task b to distinguish different kinds of sexism. when benchmarked with linguistic features, we observed them to be redundant. it appears that the large language models are already aware of the task-related information that linguistic features are trying to offer. regardless of continuous pretraining, we see that the features are not adding any noticeable improvements. overall, our system (named skam) ranks number 45 out of 84 for task a and position 39 out of 69 for task b against the official test set. the code can be found in the github repository. ## data the dataset consists of 14,000 annotated messages and an additional set of 2 million sentences (1 million sentences obtained from reddit and 1 million sentences obtained from gab). this additional data will be used to further pretrain our model. more information about the dataset can be found in kirk et al. (2023)'s work. in addition to the validation dataset on codalab, we create a local development set which is a 15 percent split of the training data provided. ## methodology our system is built in two phases. in the first phase, three different models are benchmarked against 2 nltk documentation task a and b. the best-performing model will be used for the next phase, which is feature evaluation. during this phase, three different features are used individually and as ensembles and appended to the best-performing model. in addition, we consider further pretraining the best-performing model selected during phase 1. in this section, three main aspects of our methodology will be discussed: (1) discussion of language models used, (2) descriptions of features utilized and (3) the motivation behind further pretraining the best performing models of phase 1. ## conclusion in this work, we evaluated the impact of different linguistic features, hurtlex, perspectiveapi and empath, and found that they are redundant when used with a domain specific language model like hatebert. we also evaluated if continuous pretraining helps the model in the downstream tasks. we observed it to be beneficial in task b which is more complex and demands more nuanced understanding of the text. as future work, we aim to incorporate context guided linguistic features and experiment with the way we integrate features.