Overview of The Shared Task on Homophobia and Transphobia Detection in Social Media Comments (2022)

Abstract: Homophobia and Transphobia Detection is the task of identifying homophobia, transphobia, and non-anti-LGBT+ content from the given corpus. Homophobia and transphobia are both toxic languages directed at LGBTQ+ individuals that are described as hate speech. This paper summarizes our findings on the “Homophobia and Transphobia Detection in social media comments” shared task held at LT-EDI 2022 - ACL 2022 1. This shared taskfocused on three sub-tasks for Tamil, English, and Tamil-English (code-mixed) languages. It received 10 systems for Tamil, 13 systems for English, and 11 systems for Tamil-English. The best systems for Tamil, English, and Tamil-English scored 0.570, 0.870, and 0.610, respectively, on average macro F1-score.

## introduction violence is becoming more common on social media platforms, negatively influencing internet users. social media plays an essential role in online communication in the digital era, allowing users to freely upload and share content and express their opinions and thoughts. the use of social media platforms for online communication has grown across all languages worldwide. these platforms allow users to post and exchange content and express their opinions on any topic at any moment @xcite . it has become a big concern for online communities due to the proliferation of online material @xcite . it's considerably worse for lesbians, gays, bisexuals, transgender people, and other (lgbtq+) vulnerable people @xcite . lgbtq+ individuals are subjected to abuse, inequality, torture, and even execution worldwide because of how they look, whom they love, or who they are @xcite . sexual orientation and gender identity are crucial elements of our identities that should never be misused or discriminated against @xcite . in many countries, however, being lgbtq+ can lead to death; therefore, a vulnerable person may turn to social media for assistance or share their tales in the hopes of meeting others who share their experiences @xcite . this shared task uses a new gold standard dataset for homophobia and transphobia identification in dravidian tamil, english, and tamil-english (codemixed) languages. tamil (iso 639-3: tam) is one of the dravidian languages and a primary language of tamil nadu, pondicherry, sri lanka, and singapore, as well as a recognized minority language in malaysia and south africa with 75 million speakers @xcite . tamil is one of the world's longest-surviving classical languages. the earliest old tamil documents are small inscriptions in adichanallur dating from 905 bc to 696 bc. tamil uses agglutinative grammar, which uses suffixes to indicate noun class, number, case, verb tense, and other grammatical categories. tamil is the standard metalinguistic terminology and scholarly vocabulary, as opposed to sanskrit, which is the norm for most aryan languages @xcite . tamil words are made up of a lexical root and one or more affixes. the majority of tamil affixes are suffixes. tamil suffixes are either derivational suffixes, which modify the part of speech or meaning of the word, or inflectional suffixes, which designate categories like as person, number, mood, tense, and so on. there is no ultimate limit to the length and scope of agglutination, which might result in large words with several suffixes, requiring many words or a sentence in english @xcite @xcite . there are 12 vowels, 18 consonants, and one unique character called the aytam in the current tamil script. the vowels and consonants combine to make 216 compound characters, bringing the total number of characters to 247 @xcite @xcite @xcite . however, social media users frequently utilize it because it is easier to type other languages has the roman script. as a result, the maximum of the information for these under-resourced languages available on social media is code-mixed. this shared task aims to aid research on detecting homophobic and transphobic content in tamil, english, and tamil-english (code-mixed) languages from social media. participants were provided with the training, development, and test set for this task. the task description, data description, task and evaluation settings, participant's methodology, results and discussion, and conclusion are all summarized in the upcoming section. ## related work as social media applications are used worldwide, information and communication technology, mainly social media, has changed the way individuals communicate and develop connections. for instance, youtube is a popular social networking site where users can create their profiles, submit videos, and make comments. thanks to " liking " and " sharing " methods, it has a broad audience as thousands of people may watch each video or comment, thanks to "liking" and "sharing" methods @xcite @xcite @xcite . these comments permit cyberbullies to share unflattering or undesirable information about their victims easily. unfortunately, this opens the door for antisocial behaviors such as misogyny @xcite , sexism, homophobia @xcite , transphobia @xcite , and racism @xcite to flourish. when it involves crawling social media data, there are several efforts on youtube mining, largely focusing on exploiting user comments. computer scientists began to research text-based algorithms for spotting abusive languages and hate speech by mining social media data. the use of social media has proliferated. a previous study on homophobia and transphobia identification was conducted in 2021 on the dataset paper @xcite in which tamil, english, and tamil-english codemixed datasets were built. the dataset comprises 15,141 comments: tamil -4946, english -4161, tamil-english -6034, collected from youtube. the dataset was classified at various levels of offensiveness, namely," homophobic," " transphobic," "counter speech," "hope speech," and " non-anti-lgbt+ content," by many annotators, trained volunteers from the lgbtq+ community who identify as lgbtq+ or lgbtq+ allies. ## task description the primary goal of this venture is to detect homophobic and transphobic statements in a dataset collected from social media in tamil, english, and tamil-english. this task is a comment/post-level classification task. systems must classify a comment as homophobia or transphobia or non-anti-lgbtq+ content. although a comment/post in the dataset may contain more than one sentences, the corpus' average sentence length is one. the corpus includes annotations at the comment/post level. the participants were given development, training, and test datasets in tamil, english, and tamil-english. ## data description twitter, facebook, and youtube are social media sites that include unintentionally converting information provided by millions of consumers, which may impact a person's or company's reputation. there is a growth in call for the importance of emotion extraction software systems and identifying irrelevant words in online social media. the datasets are based on users' comments on popular videos, review products, etc., increasing on youtube nowadays. thus, it allows extra usergenerated content material in languages with constrained resources. likewise, it is equal for vulnerable lgbtq+ people who watch similar motion pictures and remark approximately the video they join. we chose to acquire statistics from social media feedback on youtube since it is the most substantially used medium with-inside the world for expressing an opinion approximately a specific video. homophobia and transphobia are not given much attention. recently @xcite created an expert annotated dataset for detecting online misogyny. we collected our dataset inspired by their work. we collected comments from the youtube videos that explain lgbtq+ instead of collecting statements from lgbtq+ people's personal coming out stories because they contained confidential information. these comments were collected with the help of the youtube comment scraper tool foot_1 and were manually annotated with three labels, namely 'homophobic,' 'transphobic' and 'nonanti-lgbt+ content.' we collected the dataset in 3 language settings: tamil, english, and tamil-english. the complete details about the dataset can be gathered from @xcite 5 task setting and evaluation setting all of the datasets have an unbalanced distribution of homophobia and transphobia classes. the majority of comments in the tamil-english code-mixed dataset belong to the non-anti-lgbtq+ content (5,385) class, indicating a class imbalance seen in the table. in the tamil and english dataset, the majority class is non-anti-lgbtq+ content (3,205 and 4,657) compared to the other two categories. this disparity was rectified by selecting the macroaveraged f1-score (f) official evaluation metric task significant variance number of instances in different classes. macro-averaging gives the same weight to all classes, irrespective of their size. we utilized a scikit learn classification report tool foot_2 . participants were able to submit up to five test runs, with one of them serving as official runs that would be scored and shown on the leader board. if no official runs were specified, the most recent contributions from each team were assumed to be official. in their papers, we allowed groups to explore the distinctions between their systems. the goal is for teams to compare the effectiveness of various setups on the test set. ## participants methodology in this competition, a total of 98 participants registered. from this, we received a total of 10, 13, and 11 submissions for tamil, english, and tamil-english languages, respectively. the techniques and outcomes of these tasks have been described. for more critical information, refers to their papers, which are stated below: ablimet (maimaitituoheti and abulimiti, 2022) has used a fine-tuning approach to the pretrained language model. this model processes the target data and normalizes its output by a layer normalization module, followed by two fully connected layers. the pre-trained language model they used is the roberta-base model for the english subtask, tamil-roberta for tamil, and tamil-english subtasks. ssncse_nlp @xcite has used a combination of word embeddings and classifiers, as well as some transformers for experiments with the code mixed datasets. they executed the feature extractions using tf-idf and count vectorizer with some models, namely svm, mlp, random forest, k-nearest neighbors, and simple transformers like labse, tamillion, and indicbert. sammaan @xcite : this team used an ensemble of transformer-based models to build the classifier. they got 2nd rank for english, 8th rank for tamil, and 10th rank for tamil-english. they experimented with models bert, roberta, hatebert, indicbert, xgboost, random forest classifier, and bayesian optimization. umuteam @xcite : this team used neural networks that combine several features sets, including linguistic components extracted from a self-developed tool and contextual and non-contextual sentence embeddings. this team got 7th, 3rd, and 2nd ranks in english, tamil, and tamil-english. ## results and discussion there was a total of 98 people who registered for this shared task. for the tamil, english, and tamil-english datasets, 14 teams submitted final findings. in the ## conclusion this paper describes the first collaborative effort for detecting homophobia and transphobia in social media on the tamil, english, and tamil-english (code-mixed) dataset to classify youtube comments. the most successful system used xlm roberta pre-trained language models for zeroshot learning to deal with data imbalance and multilingualism. for tamil, english, and tamil-english datasets, their method received macro f1 scores of 0.87, 0.43, and 0.61. the findings show that all three languages, tamil, english, and tamil-english, have the opportunity for improvement. the increased number of participants and improved system performance indicates a growing interest in dravidian nlp. we intend to expand the effort in the future to include more dravidian languages such as malayalam, kannada, and telugu. to make the system more real-time, we also planned to add mixed script data.