Overview of Third Shared Task on Homophobia and Transphobia Detection in Social Media Comments (2024)

Abstract: This paper provides a comprehensive summary of the “Homophobia and Transphobia Detection in Social Media Comments” shared task, which was held at the LT-EDI@EACL 2024. The objective of this task was to develop systems capable of identifying instances of homophobia and transphobia within social media comments. This challenge was extended across ten languages: English, Tamil, Malayalam, Telugu, Kannada, Gujarati, Hindi, Marathi, Spanish, and Tulu. Each comment in the dataset was annotated into three categories. The shared task attracted significant interest, with over 60 teams participating through the CodaLab platform. The submission of prediction from the participants was evaluated with the macro F1 score.

## introduction the growth of the internet has given rise to the widespread use of social media, and numerous other online spaces @xcite . the use of social media in particular has seen a significant increase in communication across various languages around the world @xcite . these platforms enable users to post, share content and freely express their opinions on any subject at any time @xcite . however, the liberty of expression found on the internet also comes with downsides. it allows individuals, who might otherwise feel powerless, to impact and even harm others' lives @xcite . this is often facilitated by the anonymity and emotional detachment that online interactions provide @xcite . the rapid increase in online content has raised significant concerns within digital communities @xcite . this issue is particularly acute for individuals identifying as lesbian, gay, bisexual, transgender, and other lgbtq+ identities, who often face heightened vulnerability @xcite . members of the lgbtq+ community are frequently targets of harassment, discrimination, violence, and in extreme cases, even death, due to their appearance, who they love, or their gender identity @xcite . sexual orientation and gender identity are fundamental aspects of personal identity and should be respected rather than used as grounds for discrimination @xcite . in several regions, being identified as lgbtq+ can be life-threatening. consequently, many seek support and connection through social media, hoping to find others with similar experiences and form supportive communities (chakravarthi et al., 2022c) @xcite . the task at hand involves utilizing a newly established gold standard dataset designed for identifying instances of homophobia and transphobia. this shared task uses a new gold standard dataset in dravidian and indo-arian languages tamil, malayalam, telugu, kannada, gujarati, tamil-english (code-mixed), tulu, hindi, spanish, and english languages. in this overview, we conducted a shared task on homophobia and transphobia at lt-edi 1 in ten lan-guages which were annotated in 3 labels. in the upcoming section, we will describe the task description, dataset statistics, and participant-provided experiment analysis to investigate homophobia and transphobia detection from the youtube comments on dravidian languages. ## related work in the realm of natural language processing and computational linguistics, recent studies have made significant strides in understanding and analyzing the nuances of language as it pertains to social issues. a notable example is the work of @xcite , who compiled a corpus to examine the linguistic behaviors of homosexual individuals in china, shedding light on cultural and linguistic patterns. similarly, chakravarthi et al. (2022a) developed a fine-grained taxonomy specifically for homophobia and transphobia in english and tamil languages, providing a structured framework for analyzing such content @xcite . expanding on this, chakravarthi et al. (2022c) spearheaded a shared task focused on the identification of homophobia, transphobia, and nonanti-lgbt+ content in tamil, english, and tamil-english (code-mixed) languages @xcite . this initiative was crucial in understanding the subtleties and variations of discriminatory language across different linguistic contexts. complementing this, chinnaudayar navaneethakrishnan et al. (2022) conducted a study on sentiment analysis and homophobia detection in code-mixed dravidian language youtube comments, covering tamil, malayalam, and english. this research was pivotal in exploring the intersection of sentiment analysis and social bias detection in multilingual online spaces @xcite . in a related vein, manikandan et al. (2022) employed transformer-based model methodologies like bert and xlmroberta to identify transphobic and homophobic insults in social media comments. their work highlighted the efficacy of advanced computational models in detecting subtle and explicit forms of hate speech. further, the growing prevalence of social media and its impact on communication and relationship building has been explored in depth by researchers like @xcite and @xcite . their studies delved into the dynamics of social networking sites like youtube, where user interactions through comments, likes, and shares can significantly influence public discourse and perception. however, this increased interaction on social platforms also brings challenges, as highlighted by @xcite , who explored the prevalence of antisocial behaviors like misogyny, sexism, homophobia, and transphobia. @xcite further contributed to this discussion by examining the occurrence of racism and other forms of bias in online spaces. these studies underscore the importance of developing robust computational methods to detect and analyze such harmful content. the field has seen a surge in research focusing on text-based algorithms for identifying abusive language @xcite and hate speech, as demonstrated by the work on youtube comment mining and the analysis of social media data for detecting discriminatory language. building on this foundation, a notable study, conducted in 2021, delved into homophobia and transphobia identification, providing valuable insights and methodologies for future research in this area. this body of work collectively emphasizes the crucial role of computational linguistics in addressing social issues and fostering more inclusive and respectful online environments. ## task description this task marks the third year we have conducted a shared task focused on homophobia and transphobia detection 2 . we present a diverse dataset sourced from youtube comments and posts in ten different languages: english, tamil, malayalam, telugu, kannada, gujarati, hindi, marathi, spanish, and tulu. this dataset is thoughtfully annotated with three distinct labels: homophobia, transphobia, and non-anti-lgbt+ content (a category designated for content that does not exhibit either of these prejudiced behaviors). participants in this task are provided with extensive training, development, and testing datasets. the primary objective for participants is to devise robust algorithms capable of accurately categorizing these comments and posts. their systems must discern whether the text under scrutiny contains instances of homophobia, transphobia, or falls into the non-anti-lgbt+ category. this challenge not only addresses the pressing issue of online hate speech but also contributes to inclusive language detection in a global context, promoting safer online spaces for all. ## dataset social media platforms like twitter, facebook, and youtube significantly influence public opinion through user-generated content, impacting reputations. recognizing this, there's an increasing need for tools to extract emotions and identify irrelevant content online, especially on platforms like youtube, where user comments are rapidly growing. this is particularly relevant for the lgbtq+ community, who engage with such platforms and share their thoughts on various topics. focusing on youtube, we collected comments from videos related to lgbtq+ themes. we avoided personal stories from lgbtq+ individuals to maintain privacy. using the youtube comment scraper tool foot_2 , we gathered comments and manually annotated them with three labels: 'homophobic', 'transphobic', and 'non-anti-lgbt+ content'. our dataset expanded to include ten languages: english, tamil, malayalam, telugu, kannada, gujarati, hindi, marathi, spanish, and tulu. this diverse dataset was compiled following the annotation guidelines provided in the dataset research paper @xcite . ## participants methodology in our shared task, we had a total of 61 participants registered, 12 teams who submitted results in various languages. various teams employed innovative methodologies to tackle the challenge of detecting homophobia and transphobia in social media comments. the "dkit_nlp" @xcite team utilized a bert (bert-base-uncased) @xcite model, combining training and development sets and fine-tuning it with specific hyperparameters for optimal performance. "mucs" approached the task with voting classifiers, employing techniques like syllable tf-idf, oversampling, and transformer-based bert models, alongside mvlearn. "scalar_sys1" utilized adaboost, integrating multiple classification models and focusing on hyper-parameter tuning to enhance the performance of their ensemble model. the "hypnotize" team analyzed deep learning and transformer-based models across eight languages, focusing on data ## results there was a total of 61 participants from the 12 teams submitted their results. for english 11 teams, tamil 7 teams, spanish 4 teams, hindi 7 teams, gujarati 6 teams, telugu 8 teams, kannada 8 teams, malayalam 9 teams, marathi 6 teams, and tulu 4 teams submitted the final results of all languages. ## conclusion we presented the third shared task overview on homophobia and transphobia detection in social media comments on ten different language datasets. we expect this task to have a long-term impact on the nlp domain because we received a variety of submissions with various methodologies. the most successful system was achieved by synthesizing advanced machine learning techniques, custom data preprocessing, and strategic model finetuning, effectively addressing the complex challenge of detecting homophobia and transphobia in multilingual social media content. the prediction evaluation was evaluated with a macro f1 score. the increased number of participants and improved system performance indicate a growing interest in dravidian nlp.