Nozza@ LT - EDI - ACL 2022: Ensemble Modeling for Homophobia and Transphobia Detection (2022)

Abstract: In this paper, we describe our approach for the task of homophobia and transphobia detection in English social media comments. The dataset consists of YouTube comments, and it has been released for the shared task on Homophobia/Transphobia Detection in social media comments. Given the high class imbalance, we propose a solution based on data augmentation and ensemble modeling. We fine-tuned different large language models (BERT, RoBERTa, and HateBERT) and used the weighted majority vote on their predictions. Our proposed model obtained 0.48 and 0.94 for macro and weighted F1-score, respectively, ranking at the third position.

## introduction despite the progress on lgbt+ rights, internet still remains a hostile environment for lgbt+ people. the growing number, intensity, and complexity of online hate cases is also reflected in the real world: anti-lgbt+ hate crimes increased dramatically in the last three years. 1 in 2020, the uk's lgbt+ anti-violence charity (galop) presented a report about online hate crimes regarding homophobia, biphobia, and transphobia. 2 they surveyed 700 lgbt+ people distributed through online community networks of lgbt+ activists and individuals. the results are worrisome: 8 out of 10 people experienced online hate speech in the last five years, and 1 out of 5 said they had been victims of online abuse at least 100 times. transgender people experience online harassment at a higher rate (93%) than cisgender ones (70%). it is also alarming that 18% of people claimed that online abuse was linked with offline incidents. these statistics show a worrying picture of the everyday experience that lgbt+ people are living. natural language processing (nlp) has emerged as a significant field of research for combating online hate speech because of its ability to automate the process at scale while, at the same time, decreasing the labor and emotional stress on online moderators @xcite . despite the interest of the nlp community in creating datasets and models for the task of hate speech detection, no research effort has been made to cover homophobia and transphobia specifically. this is a problem because @xcite has demonstrated that hate speech detection models do not transfer to different hate speech target types. the shared task of homophobia and transphobia detection @xcite enabled researchers to investigate solutions for this problem with the introduction of a novel dataset. the dataset comprises around 5k youtube comments manually annotated with respect to the presence of homophobia and transphobia. the corpus shows a high imbalance with respect to the non-hateful class, which covers 95% of the dataset. in this paper, we propose an approach designed to overcome the problem of class imbalance. we use ensemble modeling to combine different fine-tuned large language models. we also perform data augmentation from an external dataset to include more homophobic and transphobic instances. however, data augmentation results in lower performance, and we did not use it for the submission. our system ranked third for the english track with a macro f1-score of 0.48 and a weighted f1score of 0.94. ## data the shared task on homophobia and transphobia detection in social comments released three different datasets in english, tamil, and code-mixed tamil-english @xcite . the dataset the numbers clearly show a strong imbalance of the dataset distribution. on average, the class nonanti-lgbt+ content covers 94% of the instances, while there are only 6% of homophobic instances and 0.3% of transphobic ones. ## experimental results finally, we tested the performance of the data augmentation approach ( ## related work in the last years, many shared tasks have been organized with the aim of detecting hate speech on social media comments @xcite @xcite . while the majority of them focus on english, some efforts have been made to include other languages (e.g., italian, arabic) @xcite @xcite @xcite . @xcite proposed a one-of-a-kind shared task for homophobia and transphobia detection on social comments for three languages (english, tamil, and code-mixed tamil-english). several nlp approaches have been proposed for the task of hate speech detection @xcite @xcite @xcite @xcite . while ensemble modeling has been proven to be effective for several tasks in nlp @xcite @xcite , a limited number of research work have investigated its potentiality for hate speech detection @xcite @xcite . only recently, researchers have focused on detecting and measuring harmfulness against lgbtqia+ community members in nlp. some research work investigated bias in co-reference resolution @xcite , conversational language models @xcite , and llms @xcite . in a similar spirit, @xcite discussed the harms of treating gender as binary in english language technologies, and pointed to the complexity of gender representation. focusing on the notion of referential gender, @xcite presented an overview on phenomena relating to 3rd person pronouns and discussed how nlp can and should model pronouns. ## conclusion this article describes our approach for the shared task of homophobia and transphobia on social media comments. we propose to couple ensemble learning and data augmentation to address the problem of class imbalance of the dataset. we found that augmenting the dataset with a corpus from a different domain was ineffective. our submitted model consists of the weighted majority vote of different fine-tuned llms (bert, roberta, and hatebert) ranked at the third position out of 13 submissions. in the future, we aim to explore how fine-tuned llms are biased towards members of the lgbt+ community and propose a bias mitigation solution following @xcite @xcite .