ME n T r@ LT - EDI -2024: Multilingual Ensemble of Transformer Models for Homophobia/Transphobia Detection (2024)

Abstract: Detection of Homophobia and Transphobia in social media comments serves as an important step in the overall development of Equality, Diversity and Inclusion (EDI). In this research, we describe the system we formulated while participating in the shared task of Homophobia/ Transphobia detection as a part of the Fourth Workshop On Language Technology For Equality, Diversity, Inclusion (LT-EDI- 2024) at EACL 2024. We used an ensemble of three state-of-the-art multilingual transformer models, namely Multilingual BERT (mBERT), Multilingual Representations for Indic Languages (MuRIL) and XLM-RoBERTa to detect the presence of Homophobia or Transphobia in YouTube comments. The task comprised of datasets in ten languages - Hindi, English, Telugu, Tamil, Malayalam, Kannada, Gujarati, Marathi, Spanish and Tulu. Our system achieved rank 1 for the Spanish and Tulu tasks, 2 for Telugu, 3 for Marathi and Gujarati, 4 for Tamil, 5 for Hindi and Kannada, 6 for English and 8 for Malayalam. These results speak for the efficacy of our ensemble model as well as the data augmentation strategy we adopted for the detection of anti-LGBT+ language in social media data.

## introduction homophobia is defined as intentional discrimination against those who identify as a part of the lgbt+ community. it can be demonstrated in many ways, which can include abuse or social ignorance. transphobia, on the other hand, refers to the targeted hatred towards transgender individuals whose current gender identity and the one assigned to them during birth differ. both of these forms of hate speech have negative repercussions on the mental health as well as the overall well-being of people who are a part of the lgbt+ community 1 https://sites.google.com/view/lt-edi-2024/ @xcite . this highlights a critical need to build systems that identify this form of prejudice and bigotry. pre-trained language models (plms) like bert @xcite and @xcite , built on transformer architectures have gained recognition for their ability to interpret languages in a manner similar to humans by displaying stateof-the-art results in many nlp tasks such as document classification and language modelling. plms undergo unsupervised training on a large corpus of text data which can then be fine-tuned on domain and task-specific corpora for downstream tasks, such as the shared task on homophobia and transphobia detection by lt-edi@eacl 2024. bert, specifically, introduced the concept of bidirectional context understanding which considers both the succeeding and preceding word for a particular word to capture a more elaborate and nuanced meaning within the language. for our system, we propose an ensemble consisting of three such popular bert-based transformer architectures, namely multilingual bert (mbert) @xcite , multilingual representations for indic languages (muril) @xcite and xlm-roberta @xcite . ## conclusions and future work our submission for the shared task on homophobia and transphobia detection in social media comments demonstrates how pre-trained language models (plms) specifically those built on a bertbased architecture can be effectively used in the case of text classification. our ensemble model consisting of mbert, muril and xlmroberta, has shown consistent results by achieving the top three ranks for 5 language tasks, ranking first for spanish and the under-resourced language tulu. we have been able to achieve average macro f1 scores of 0.707, 0.582, 0.960, 0.960, 0.488, 0.746, 0.325, 0.935, 0.407 and 0.744 for tulu, spanish, telugu, gujarati, marathi, tamil, hindi, kannada, english and malayalam respectively. in the future, we would like to experiment with the following aspects in further detail :