MUCS @ LT - EDI -2024: Learning Approaches to Empower Homophobic/Transphobic Comment Identification (2024)

Abstract: Homophobic/Transphobic (H/T) content includes hatred and discriminatory comments directed at Lesbian, Gay, Bisexual, Transgender, Queer (LGBTQ) individuals on social media platforms. As this unfavourable perception towards LGBTQ individuals may affect them physically and mentally, it is necessary to detect H/T content on social media. This demands automated tools to identify and address H/T content. In view of this, in this paper, we - team MUCS describe the learning models submitted to “Homophobia/Transphobia Detection in social media comments:LT-EDI@EACL 2024” shared task at European Chapter of the Association for Computational Linguistics (EACL) 2024. The learning models: i) Homo_Ensemble - an ensemble of Machine Learning (ML) algorithms trained with Term Frequency-Inverse Document Frequency (TFIDF) of syllable n-grams in the range (1, 3), ii) Homo_TL - a model based on Transfer Learning (TL) approach with Bidirectional Encoder Representations from Transformers (BERT) models, iii) Homo_probfuse - an ensemble of ML classifiers with soft voting trained using sentence embeddings (except for Hindi), and iv) Homo_FSL - Few-Shot Learning (FSL) models using Sentence Transformer (ST) (only for Tulu), are proposed to detect H/T content in the given languages. Among the models submitted to the shared task, the models that performed better for each language include: i) Homo_Ensemble model obtained macro F1 score of 0.95 securing 4th rank for Telugu language, ii) Homo_TL model obtained macro F1 scores of 0.49, 0.53, 0.45, 0.94, and 0.95 securing 2nd, 2nd, 1st, 1st, and 4th ranks for English, Marathi, Hindi, Kannada, and Gujarathi languages, respectively, iii) Homo_probfuse model obtained macro F1 scores of 0.86, 0.87, and 0.53 securing 2nd, 6th, and 2nd ranks for Tamil, Malayalam, and Spanish languages respectively, and iv) Homo_FSL model obtained a macro F1 score of 0.62 securing 2nd rank for Tulu dataset.

## introduction homophobia and transphobia are the two terms that refer to negative attitude towards the homosexual and transsexual people like lgbtq. these attitudes are expressed in terms of h/t comments, insults, and discriminatory language on social media platforms @xcite . this unfavourable perceptions towards homosexual and transsexual people can have a very negative effect which can exacerbate mental health issues and give them a sense of helplessness and fear @xcite . hence, there is a need to develop automated tools to detect h/t content to maintain healthy social media platforms. in a multilingual country like india, people prefer to blend english words or sub-words with their native language creating code-mixed texts @xcite . the intricate nature of code-mixed text introduces additional complexities, where words or sub-words from different languages may be combined in different ways lacking grammatical rules, making it challenging to establish consistent patterns for classification. the h/t content available on social media may also be in code-mixed form @xcite . to address the challenges of h/t content detection in social media text, in this paper, weteam mucs, describe the models submitted to "homophobia/transphobia detection in social media comments:lt-edi@eacl 2024" shared task foot_0 @xcite . while the shared task is modeled as a multi-class text classification problem for h/t content detection in english, hindi, tamil, telugu, kannada, gujarathi, malayalam, marathi, and spanish languages, by employing ml and tl approaches, h/t content detection in tulu is modeled as binary text classification problem using fsl approach. the rest of the paper is structured as follows: section 2 contains related works and section 3 explains the methodology. section 4 describes the experiments and results and the paper concludes in section 5 with future work. researchers have explored different approaches to detect h/t content on social media platforms. description of few research works that are carried out to perform similar tasks are given below: ashraf et al. (2022) presented ml models (support vector machine (svm), random forest (rf), passive aggressive classifier, gaussian naïve bayes (gnb), multi-layer perceptron) trained with tf-idf of word bigrams, for h/t content detection in english, tamil and tamil-english. out of their proposed models, svm classifier outperformed the other classifiers with weighted f1 scores of 0.91, 0.92, and 0.88 for english, tamil, and tamil-english respectively. singh and motlicek (2022) fine-tuned cross lingual language models robustly optimised bert (xlm-roberta) model in the zero-shot learning framework for detecting h/t contents in english, tamil, and tamil-english texts. their proposed methodology obtained weighted f1 scores of 0.92, 0.94, and 0.89 for english, tamil, and tamil-english languages respectively. pranith et al. (2022) presented tl based approach with two different bert variants (in-dicbert and labse (language-agnostic bert sentence embedding)) for h/t content detection in english, malayalam, tamil-english, and tamil languages. their proposed labse model obtained weighted f1 score of 0.46 for english language and indicbert model obtained weighted f1 scores of 0.54, 0.39, and 0.28 for malayalam, tamil-english, and tamil languages respectively. chanda et al. (2022) fine-tuned multilingual bert (mbert) model for detecting sentiment and homophobia content in malayalam and kannada code-mixed texts and obtained macro f1 scores of 0.72 and 0.66 for malayalam and kannada code-mixed texts respectively. nozza et al. (2022) fine-tuned different large language models (llms) (bert, roberta, hatebert) and ensemble modeling with majority voting to combine different finetuned llms. to handle the class imbalance they performed data augmentation by collecting external dataset to include more h/t instances for the detection of h/t content in english dataset. their proposed ensemble model outperformed other models with a weighted f1 score of 0.94 for english dataset. though there are several models to identify h/t content in social media text, there is still scope for developing models for h/t content detection in low-resource languages like tamil, malayalam, telugu, tulu, etc., as these languages are not much explored in the realm of code-mixed content. ## methodology the proposed methodology includes implementation of a wide range of learning models including ml, tl, and fsl approaches for identifying h/t content in the datasets provided by the organizers of the shared task. pre-processing techniques are applied commonly to all the datasets. as the datasets are imbalanced, resampling techniques are used to balance the datasets in some of the learning models. pre-processing and resampling steps are explained below: ## experiments and results statistics of the dataset provided by the organizers @xcite @xcite of the shared task for the detection of h/t contents in social media text for english, hindi, tamil, telugu, kannada, gujarathi, malayalam, marathi, and spanish are shown in ## conclusion and future work this paper describes the models submitted by our team -mucs, to the shared task "homophobia/transphobia detection in social media comments: lt-edi@eacl 2024" shared task at eacl 2024. four distinct models: i) homo_ensemble