M ila NLP at S em E val-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection (2023)

Abstract: We present the system proposed by the MilaNLP team for the Explainable Detection of Online Sexism (EDOS) shared task. We propose an ensemble modeling approach to combine different classifiers trained with domain adaptation objectives and standard fine-tuning. Our results show that the ensemble is more robust than individual models and that regularized models generate more “conservative” predictions, mitigating the effects of lexical overfitting.However, our error analysis also finds that many of the misclassified instances are debatable, raising questions about the objective annotatability of hate speech data.

## introduction sexism and misogyny are ever-present in online spaces, creating a hostile environment which can prevent women from enjoying the benefits brought about by the internet @xcite . because of its ubiquity, tackling sexism manually has proven ineffective, redirecting efforts towards automatic detection of this phenomenon using natural language processing. @xcite presented a novel corpus for the classification of sexist content from gab and reddit in english using fine-grained labels. the hierarchical classification of sexism permits the development of classifiers that are both more accurate and explicable. in the explainable detection of online sexism (edos) shared task, systems are required to detect sexist content. the task is divided into three sub-tasks that refer to different levels of granularity (see §2). in this paper, we propose an ensemble modeling approach to combine different fine-tuned language models. although ensemble modeling has been shown to be beneficial for various tasks in nlp @xcite @xcite , its potential for hate speech detection has only been explored in a small number of papers @xcite @xcite . our results confirm that ensemble learning improves robustness and classification performance and we find that regularized fine-tuning leads to higher uncertainty in sexism prediction. the resulting "conservative" models reduce both false positives and false negatives when included in the ensemble. our system ranked 9th for the binary sexism detection task, with a macro f1-score of 0.8616 (-0.013 compared to the winning team). moreover, through a deeper analysis on test instances, we find that many examples in the aggregated dataset are likely mislabelled -showing how to best annotate sexism (and other subjective language phenomena) remains an open question. ## task overview the edos shared task @xcite requires participants to build and test automatic models for sexism detection for social media posts. the challenge is framed as a classification task and divided into three hierarchical sub-tasks: a binary sexism detection: the goal of the task is to identify whether the post is sexist or not; b multiclass sexism categorization: systems are required to assign every sexist post from phase one of four categories: threats, derogation, animosity, prejudiced discussions; c fine-grained sexism categorization: similarly to b, systems are required to identify categories, but now among 11 distinct vectors of sexism, e.g., threat of harms, immutable gender stereotypes, or supporting system discrimination. this paper presents a system trained to solve task a. although we did not address tasks b and c specifically, we provide insights into how errors are distributed across their fine-grained categories ( §4.1). ## methodology our proposed method is an ensemble of four finetuned classifiers. we built each classifier under different training regimes, e.g., using a simple finetuned model or running domain adaptation, or enabling regularisation. see §4 for further discussion on the performance of individual models and the benefits of ensembling them. to produce a single classification label, we average every model's prediction and use a .5 threshold to assign the sexist label. ## results we compare the results of each of the models on the provided test set. in addition, we experiment with a series of ensemble models: rob, rob r3f , and deb mlm in combination with rob ear as well as an ensemble of all the above models. overall, our ensemble achieves an f1 score of .862. full results per model are shown in although rob ear was the worst-performing model, it offers modest gains in performance in combination with other models. the best performing model was the ensemble of deb mlm + rob ear . ## related work hate speech detection has received a lot of interest in natural language processing in recent years but despite the large number of works in the field, only a small subset focuses on the problem of sexism and misogyny detection. @xcite demonstrates that hate speech detection techniques do not transfer across diverse hate speech targets, highlighting the necessity for ad-hoc sexism and misogyny detection methodologies and datasets. indeed, the corpora that were made accessible as a part of shared tasks @xcite @xcite @xcite made it possible for a range of nlp techniques to be taken towards the challenge of automatically detecting instances of sexism and misogyny on twitter messages @xcite @xcite @xcite . ## conclusion we addressed task a of the edos shared task (kirk et al., 2023) on sexism detection. we compare domain adaptation to a fine-tuned baseline and tested two regularization techniques. we leveraged the capabilities of each model in an ensemble model example rob fp f*ck off you annoying pinkoid mayocel and go get yourself an indonesian bride. damn ugly she-ape! i wish men would stop asking girls out altogether just so they can get an appreciation of their privilege. unfortunately, most men value banging a girl over their own dignity. fn probably from 4 different dudes too. you have to be a f*cking pathetic cuck to get into anything with a single mother. there's a reason for gender roles. men don't lust over butch girls, and women don't want soft, metrosexual hipsters. i totally agree. either you worship women or you're a misogynist. rob r3f fp if she loves rapefugees so much let her go spend a week alone in a camp with some of them. after they remove the coke bottles from her orifices they should ask her again how much she loves refugees absolutely. you hit the nail on the head, stop validating women that are not even invested. she probaby has 5 beta orbitters that like every f*cking bs she posts on social media. if you are bleeding from your vagina, you are a woman...period. [url] fn the little nazi twerp deserved to get his girl ass fucked while he was away being a tool of imperialism. would have been extra funny if it was a black dude banging her. women who live with the baby-daddy are *always* f*cking the babydaddy or he would have kicked her to the curb already. feminine traits in men can be exploited for gain, comfort, being in control; what kind of sexual benefit (the kind that men are generally interested in) can be gained from an androgynous woman? deb mlm fp the mormans have female cucks. we literally outlawed it because it's bad for scociety. don't support her or pay her way or you be come beta bux the argument "i can't change my height" is a bad one. you are assuming women gaf about what is in your control. when a 7/10 has a plethora of guys to choose from, some will be tall, and she will choose accordingly. supply and demand fn even femcels+the hambeats on it admitt that they could easily get sex if they wanted. theres nothing you have to do to get sex as female. you dont even have to be female, even transwomen are enough in 2k18 reeeeee wimminz are ivel white knighting got him torched. hope his burns have taught him something important about female nature. but then again, manginas and simps rarely learn. based on our findings, future work should experiment with how best to combine adaptation and regularization for robustness, and continue to investi-gate how best to collect, annotate and model sexism by e.g., modelling annotator disagreement based on unaggregated labels, following the guidelines set out in the perspectivist data manifesto. 5 additionally, interpretability tools must be developed and integrated into models to improve transparency, accountability, and fairness @xcite .