M ason T igers@ LT - EDI -2024: An Ensemble Approach Towards Detecting Homophobia and Transphobia in Social Media Comments (2024)

Abstract: In this paper, we describe our approaches and results for Task 2 of the LT-EDI 2024 Workshop, aimed at detecting homophobia and/or transphobia across ten languages. Our methodologies include monolingual transformers and ensemble methods, capitalizing on the strengths of each to enhance the performance of the models. The ensemble models worked well, placing our team, MasonTigers, in the top five for eight of the ten languages, as measured by the macro F1 score. Our work emphasizes the efficacy of ensemble methods in multilingual scenarios, addressing the complexities of language-specific tasks.

## introduction in this current era dominated by social media platforms, people heavily rely on online content for communication, learning, knowledge-sharing, and staying abreast of new technologies. the comment sections, intended for constructive feedback, unfortunately, sometimes become grounds for hate speech, offensive comments, and discrimination, including targeting a specific community. such behaviors cause trauma, fear, anxiety, depressive symptoms and discomforts among lgbtq+ individuals @xcite , hindering them from freely expressing their thoughts and feedback. to ensure the safety and comfort of users on online platforms, it becomes imperative to identify and address hate speech and offensive comments. although there are existing policies aimed at protecting communities from such misconduct, violations may lead to the removal of offending comments foot_1 foot_2 . however, the identification process ne-cessitates the application of nlp and ai techniques due to the diverse nature of hate speech, which can manifest in both direct and passive forms. surprisingly, there is a higher focus on researching this topic in english and other high-resource languages like hindi, with ample resources. for instance, couple of shared tasks have been organized previously e.g. @xcite , @xcite , @xcite , accompanied by substantial datasets e.g. @xcite , @xcite . however, there has been a lack when it comes to identifying hate speech in low-resource and under-resource languages. this shared task @xcite aims to identify hate speech contents, specifically homophobia, transphobia, and non-anti-lgbt+ sentiments, directed at lgbtq+ individuals in 10 different languages, including 7 low-resource languages. to tackle the linguistic diversity, we conduct separate experiments for each language, leveraging different transformer-based models with proficiency in distinct languages. notably, for tulu, an under-resourced language, we employ a prompting approach. alongside these experiments, various techniques are explored, and the most effective one during the evaluation phase is implemented in the test phase for comprehensive validation. ## related works as smart devices, mobile apps, and social media platforms become more widely utilized, there are also more negative effects linked to them, such as cyberbullying, hurtful comments, and rumors have increased. these platforms have also become a space for negative behaviors like sexism, homophobia @xcite ), misogyny @xcite ), racism @xcite ) and transphobia @xcite ). internet trolling, where people say mean things online, has become a global problem. to deal with this, researchers are looking into automated methods since checking every message manually is impossible. in this section, we provide a brief summary of the research attempts that focused on identifying homophobia, transphobia, and non-anti-lgbt+ content from youtube comments. to motivate researchers to tackle the issue of identifying toxic language directed at the lgbtq community, in the last few years, several shared tasks have been released. in one such shared task related to @xcite , researchers submitted systems to deal with homophobic and transphobic comments. @xcite gives an overview of the models submitted. three subtasks for the languages tamil, english, and tamil-english (code-mixed) were the emphasis of this shared task. apart from this, several studies have been conducted. the author of @xcite used the m-bert model, which shows that it is capable of accurately identifying homophobic or other abusive language in turkish social media comments. similarly, another shared task related to this topic has been organized showing xlm-r performing best with spatio-temporal data in 5 different languages @xcite . @xcite presents a mexican-spanish annotated corpus along in which beto-cased (spanish bert) outperforms the other models. for several text classification tasks, transformerbased ensemble approaches perform very well, like the works by @xcite ; @xcite . also, prompting large language models like gpt3.5 @xcite is another popular approach in recent classification tasks @xcite for the past year. efforts to identify homophobic and transphobic comments have primarily focused on a maximum of five languages to date. however, in this shared task, a total of 10 languages have been chosen, and ongoing efforts now include telugu, tulu, and marathi. ## datasets the dataset provided for the shared task contains 10 languages -tamil, english, malayalam, marathi, spanish, hindi, telugu, kannada, gujarati, and tulu. it is compiled using five separate research works. the previous iteration of the workshop @xcite the current dataset combines all these works to build a comprehensive dataset for the task of homophobia and/or transphobia detection in 10 languages @xcite . the detailed dataset demonstration and labelwise data percentage for all the languages are available in ## experiments among the 10 languages (except tulu) we use xlm-r, m-bert and language specific berts: roberta @xcite , hindibert (nick doiron, 2023), tamilbert @xcite , malayalam-bert @xcite , marathibert @xcite , spanishbert @xcite , telugubert @xcite , kannadabert @xcite , gu-jaratibert @xcite for hindi, tamil, malayalam, marathi, spanish, telugu, kannada, gujarati respectively. while using marathibert and hindibert, we pad the sentence length upto 512 tokens, because of the limitation of the aforementioned berts. training parameters are mostly kept the same across all models, mentioned in after that we perform weighted ensemble ap-role: "you are a helpful ai assistant. you are given the task of detecting homophobia and transphobia in a given text. definition: homophobia and transphobia detection is the process of identifying expressions of hatred or discrimination against lgbtq+ individuals in communication.'. examples: an example of homophobic/transphobic comment: <example1>. an example of non-homophobic/transphobic comment: <@xmath0. task: generate the label [yes/no] for this "text" in the following format: <@xmath1. thanks." proach of the above models (xlm-r, m-bert, language specific berts) and use the macro f1 scores of the models on the dev data as the weight along with the confidence score to get the ensemble macro f-1 score of the test phase. after the test labels get published, we use the f1 score in the rank list as the weight along with the corresponding confidence score for the additional experiment of weighted ensemble approach. as tulu is very close to kannada and tulu doesn't have any language specific fine-tuned model, we used kannadabert on tulu. in south karnataka, individuals who speak tulu are typically fluent in both tulu and kannada. due to the long-standing interaction between tulu and kannada, it can be anticipated that codeswitching between these two languages is a probable outcome @xcite . moreover, we implement few shot learning using gpt3.5 for the tulu dataset (see figure 1 ). such prompting is very widely used in recent works on text classification @xcite . we got the same result as the ensemble approach in the few shot prompting technique. we specifically used 8 -shot prompting for tulu language. this process is inspired by @xcite . we improve the macro f1 in 4 out of 10 cases in this additional experiments and rest of the 6 cases we get the same macro f1 as the test phase. ## results we employ an ensemble-based methodology for the tasks, since in text classification tasks this can further improve the results. for for all the ensemble approaches, we use xlm-r, mbert, and a bertbased model fine-tuned for that specific langauge as we mentioned in the previous section. in the testing phase, we ensemble the confidence score of the three models and then calculate the weighted average. the weight in this context is the macro f1 score of the corresponding models on dev data. for tulu language, we use gpt 3.5 for few shot prompting. we use few instances of each of the two labels with specific prompt and and then get the test labels predicted by gpt 3.5. after the end of testing phase when the test labels get published, we further run the ensemble ## conclusion the task of detecting abusive speech targeting sexual and gender minorities has become increasingly important given the rise of social media and its potential to propagate harmful stereotypes that further marginalize vulnerable populations. this paper presents our efforts to address online transphobia and homophobia in multilingual contexts, which remains an under-studied area in abusive language detection. we employ an ensemble approach combining multiple individual models to identify abusive speech across datasets in ten languages. our findings demonstrate that while no individual system consistently achieves superior performance across all data, monolingual languagespecific bert models fine-tuned on our abusive speech data unsurprisingly emerge as strong approaches for this classification problem. however, our ensemble framework leveraging voting across multiple bert variants along with other models surpasses any individual system, indicating the value of model diversity even when one base technique manifests strengths. we hypothesize that the inconsistencies across models and languages result partly from imbalanced, sparse instances of actual abusive samples in our data. hence, future work should prioritize constructing larger, more balanced benchmark datasets for abusive language detection encompassing underrepresented identities. nonetheless, this research presents a starting point for identifying multilingual, multidirectional abuse in online spaces through ensemble natural language systems.