VEL @ LT - EDI : Detecting Homophobia and Transphobia in Code-Mixed S panish Social Media Comments (2023)

Abstract: Our research aims to address the task of detecting homophobia and transphobia in social media code-mixed comments written in Spanish. Code-mixed text in social media often violates strict grammar rules and incorporates non-native scripts, posing challenges for identification. To tackle this problem, we perform pre-processing by removing unnecessary content and establishing a baseline for detecting homophobia and transphobia. Furthermore, we explore the effectiveness of various traditional machine-learning models with feature extraction and pre-trained transformer model techniques. Our best configurations achieve macro F1 scores of 0.84 on the test set and 0.82 on the development set for Spanish, demonstrating promising results in detecting instances of homophobia and transphobia in code-mixed comments.

## introduction hate speech is speech that is directly or indirectly against a person or group and contains animosity because of something inherent to that person or group @xcite @xcite . locating hate speech on the internet is a difficult task that even the most advanced models struggle to complete @xcite . as a result of the rapid development in user-generated online content, which has not only resulted in a vast increase in the accessibility of information but has also delivered a vast increase in the accessibility of information @xcite , individuals have been provided with a simple platform on which to express their opinions and communicate with others in a public forum @xcite . this has resulted in some undesirable uses of online spaces, such as the dissemination of hate speech, which is regrettable. the use of abusive language frequently accompanies the dissemination of hate speech in everyday life, especially on social media @xcite @xcite . in studies conducted under the headings of hate speech, offensive language, and aggressive language, the examination of homophobic language is typically grouped with analyses of other forms of hostility @xcite @xcite . "emotional disgust towards individuals who do not conform to society's gender expectations" is one definition of transphobia @xcite . homophobia is the unreasonable fear, loathing, and intolerance of homosexual men and women in close proximity @xcite . typically, hate speech detection models are evaluated by measuring how well they perform on data set aside for testing. most of the evaluation, accuracy, and f1 score are used as metrics @xcite @xcite . the overview paper @xcite , described the overall descriptions of the participants participated and the dataset of the shared task on homophobia and transphobia detection in so-cial media comments. we participated in task a for spanish, which focused on the detection of homophobia and transphobia in social media comments. the task was organized by lt-edi@ranlp-2023. with the provided dataset, we developed machine learning models using feature extraction as baselines, as well as the muril transformer model. among our models, the muril model yielded the best results, achieving a weighted f1 score of 0.86. these scores indicate the effectiveness of our approach in accurately identifying instances of homophobia and transphobia in spanish social media comments. our participation in this shared task has provided valuable insights into the detection and understanding of discriminatory behavior in online platforms. ## related work researchers examined the linguistic behaviors of homosexual individuals in china by compiling a corpus of their texts @xcite . @xcite ) created fine-grained taxonomy for homophobia and transphobia for english and tamil languages. @xcite conducted a shared task to the identification of homophobia, transphobia, and non-anti-lgbt+ content from the given corpus. this task was centered on three subtasks for the tamil, english, and tamil-english (code-mixed) languages. it received 10 tamil systems, 13 english systems, and 11 tamil-english systems. the average macro f1-score for the top systems for tamil, english, and tamil-english was 0.570, 0.877, and 0.610, respectively. transphobic and homophobic insults directed at lgbtqi+ persons for the shared task have been identified using transformer-based model methodologies such as bert and xlmroberta models by @xcite . bert offers 91%, while xlm-roberta offers 93%. the content was predicted using the indicbert and labse machine learning models. the following were the results: indicbert was utilized to train tamil, malayalam, and tamil-english, whereas labse was utilized to predict the english content. the weighted average f1 scores for english, malayalam, tamil-english, and tamil were 0.46, 0.54, 0.39, and 0.28, respectively by @xcite . @xcite participated in both sentiment analysis and homophobia detection tasks. under the feature extraction techniques of count vectorizer and tf-idf, pre-trained models such as bert, xlm, and mpnet were used alongside classifiers such as svm, mlp, and random forest. the rankings for sentiment analysis assignment are rank 1 in the tamil dataset, rank 6 in the malayalam dataset, and rank 7 in the kannada dataset. the sentiment analysis task in the malayalam dataset yielded the highest f1 score of 0.63, while the homophobia detection task yielded 0.95. various machine learning algorithms are contrasted with the proposed system's performance @xcite . we discuss the existing research in the field of text classification, particularly focusing on the specific context of indian languages. we implemented the muril (multilingual representations for indian languages) @xcite pre-trained transformer model, which we utilize in our study. muril, available through the hugging face model repository, is specifically designed to handle the linguistic complexities and nuances of indian languages, enabling effective text classification tasks. while previous works have explored various approaches for text classification in indian languages, our paper distinguishes itself by leveraging the muril model and fine-tuning it on a diverse range of downstream tasks. by highlighting the advantages of muril and showcasing the results of our fine-tuning experiments, we contribute to the growing body of research focused on enhancing the performance of text classification in indian languages. ganized by lt-edi@ranlp-2023. the objective of this task was to identify instances of homophobia and transphobia in social media comments across multiple languages, including english, hindi, tamil, spanish, and malayalam. our focus was specifically on task a, which involved detecting these forms of discrimination in spanish comments using a 3-class classification approach. the dataset provided for this task consisted of 1586 social media comments, each annotated as homophobic, transphobic, or none. we divided this dataset into a training set of 850 comments, a development set of 236 comments, and a test set of 500 comments. the class distribution for each set is presented in ## results and discussion in this section, we evaluated the results of various models used to detect homophobia and transphobia in the spanish language. the performance of these models was assessed using metrics such as accuracy (acc), macro precision (mp), macro recall (mr), macro f1 (mf1), weighted precision (wp), weighted recall (wr), and weighted f1 (wf1) scores. we experimented with five machine learning models, including nb, svm, lr, dt, and rf utilizing countvectorizer feature extraction. the weighted f1 scores obtained for these models were 0.60, 0.78, 0.82, 0.79, and 0.77, respectively. next, we explored the performance of a large language model, specifically the muril large cased model, which was originally pre-trained on indian languages. however, we adapted it for detecting homophobia and transphobia in the spanish language. comparing the results in ## conclusion in this study, we examined the detection of homophobia and transphobia in the spanish language using machine learning models and a pretrained transformer model. among the traditional models with countvectorizer feature extraction, the muril pre-trained transformer model outperformed them with a weighted f1 score of 0.84 on the test set. the transformer model's superior performance demonstrates its effectiveness in classifying instances of homophobia and transphobia. the confusion matrix visualization further supported the model's ability to correctly identify such instances. consequently, we conclude that the pretrained transformer model is a suitable choice for this task, offering the potential for addressing social issues and promoting inclusivity in online spaces. future research can explore fine-tuning techniques and larger datasets to enhance the model's performance.