cantnlp@ LT - EDI -2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models (2023)

Abstract: This paper describes our multiclass classification system developed as part of the LT-EDI@RANLP-2023 shared task. We used a BERT-based language model to detect homophobic and transphobic content in social media comments across five language conditions: English, Spanish, Hindi, Malayalam, and Tamil. We retrained a transformer-based cross-language pretrained language model, XLM-RoBERTa, with spatially and temporally relevant social media language data. We found the inclusion of this spatio-temporal data improved the classification performance for all language and task conditions when compared with the baseline. We also retrained a subset of models with simulated script-mixed social media language data with varied performance. The results from the current study suggests that transformer-based language classification systems are sensitive to register-specific and language-specific retraining.

## introduction the purpose of this shared task was to develop a classification system to predict whether samples of social media comments contained forms of homophobia or transphobia across different language conditions. there were no restrictions on language models or data pre-processing methods. the five language conditions: english, spanish, hindi, malayalam, and tamil. in addition to the language conditions, participants were tasked with developing a system for a three-class and seven-class classification system defining different forms of homophobic and transphobic hate speech @xcite . the main contribution of our proposed system outlined in this paper included spatio-temporal relevant social media language data to retrain a transformer-based language model to increase the sensitivity of the pretrained language model (plm). we have also created simulated samples of scriptmixed social media language data which was used as part of the retraining process. ## methodology in this section, we provide a system overview of our transformer-based language model. we also provide details on our retraining and fine-tuning procedures. ## results the results of task a are shown in we have highlighted the performance metric in bold in terms of the optimal classification models submitted to the organisers for evaluation in ta-ble 4. hindi and tamil ranked third out of seven, malayalam ranked fourth out of seven, and english ranked seventh out of eleven. due to issues with the labels, the submission for the spanish condition in task a was invalid. however, the macro averaged f1 metric is provided in brackets for reference. in task b, malayalam performed the best of the baseline classification models with a macro averaged f1 score of 0.86 while english performed the worst with a macro averaged f1 score of 0.15. the performance increased once we fine-tuned the classification model with the retrained xlm-roberta with the macro averaged f1 score for english improving from 0.15 to 0.54 and for tamil improving from 0.77 to 0.90. the performance remained stable between the baseline and retrained models for malayalam. when we introduced the script-mixed models for malayalam and tamil, we saw varying levels of performance. the macro averaged f1 score for malayalam increased from 0.86 to 0.88. this suggests an increase in performance accuracy. counterintuitively, the macro averaged score f1 for tamil decreased from 0.90 to 0.80 which was on par with the baseline model. this suggests a decrease in performance accuracy. our malayalam classification system fine-tuned on the script-mixed social media language data ranked first out of six, while the tamil classification system fine-tuned on the script-mixed social media language data ranked fourth out of seven despite the decrease in performance from the retrained language model. our english classification system fine-tuned on the retrained language model 107 ranked sixth out of nine. ## discussion this paper addresses intrinsic issues related to hate speech detection in written social media data. this was mirrored in our training, validation, and testing data which reflects the multifaceted challenges of real-world scenarios. hate speech is not confined to any single language or geographic region and its instances are often buried within the vast array of existing textual data, particularly in the context of social media. the employment of the xlm-roberta model has demonstrated to be an effective system in detecting homophobia and transphobia in social media comments, particularly when the plm has been retrained with spatio-temporal data for the english and tamil language conditions. these findings underline the potential of integrating geographic language data into models as a means of enhancing their performance not only on highly represented languages, but also on lower underrepresented languages, thus offering a robust solution. in the preceding sections of this paper, we have outlined and highlighted in to alleviate this issue, the utilisation of synthetic data through data augmentation techniques could prove to be a promising approach. data augmentation, as a broad concept, involves expanding the existing data sets to enhance their diversity, and therefore, the generalisability of the models trained on them @xcite . the generation of synthetic data has been demonstrated to be an effective mechanism in addressing biased data sets, but it also presents a desirable practice particularly suited for hate speech detection given the prevailing concerns over text obfuscation of such instances @xcite . to help facilitate a system that can account for these nuances, data noise injection via character, word, or even phrasal additions could be advantageous. in this sense, the application of synthetic data coupled with noise injection can help address class imbalance, while also training more robust classifiers that are less reliant on explicit instances of derogatory terms, but are more adept at discerning underlying contextual uses of hate speech. there are real-world applications to our homophobia/transphobia detection system as we can refine our model with language-specific and regionspecific information to monitor hate speech on social media directed at lgbtq+ communities. this is particularly useful for languages that are not otherwise as well represented in large language models such as malayalam which saw great improvement in performance with the addition of script-mixed retraining data. ## conclusion we saw an improvement in performance in our retrained homophobia/transphobia classification model when compared with our baseline model. our unique approach to this shared task has shown potential for retraining pretrained language models with spatio-temporal relevant language data to improve the performance of our homophobia/transphobia detection system. counterintuitively, the inclusion of script-mixed language data gave us variable results. we will aim to refine our classification system with other attested methods such as noise injection in order to improve the performance of our system.