bitsa_nlp@ LT - EDI - ACL 2022: Leveraging Pretrained Language Models for Detecting Homophobia and Transphobia in Social Media Comments (2022)

Abstract: Online social networks are ubiquitous and user-friendly. Nevertheless, it is vital to detect and moderate offensive content to maintain decency and empathy. However, mining social media texts is a complex task since users don’t adhere to any fixed patterns. Comments can be written in any combination of languages and many of them may be low-resource. In this paper, we present our system for the LT-EDI shared task on detecting homophobia and transphobia in social media comments. We experiment with a number of monolingual and multilingual transformer based models such as mBERT along with a data augmentation technique for tackling class imbalance. Such pretrained large models have recently shown tremendous success on a variety of benchmark tasks in natural language processing. We observe their performance on a carefully annotated, real life dataset of YouTube comments in English as well as Tamil. Our submission achieved ranks 9, 6 and 3 with a macro-averaged F1-score of 0.42, 0.64 and 0.58 in the English, Tamil and Tamil-English subtasks respectively. The code for the system has been open sourced.

## introduction twenty first century social media has become the epicenter of polarized opinions, arguments, and claims. the ease of information access not only benefits fruitful discussions but also facilitates phenomena such as hate speech and cyber bullying. recently organized workshops and shared tasks have fostered discussions around detection of hate speech, toxicity, misogyny, sexism, racism and abusive content @xcite . while research in processing and classifying offensive language in social media is vast @xcite , there is very little work on detecting sexual orientation discrimination in particular. more so, compared to resource-rich languages such as english and japanese, indic languages such as tamil and malayalam are scarce in well-annotated data. although advancements in large multilingual models have promoted crosslingual transfer learning in indic languages @xcite , there have not been any visible attempts to censor homophobia and transphobia. the perception of the subject matter as being taboo prohibits advancements in data collection, annotation and analysis. curbing sensitive online content is imperative for preventing harm to mental health of the community as well as avoiding divide between minorities. these reasons have contributed towards the need of moderating social media comments spreading any form of hatred towards the lgbtqia+ population. while both -the detection of homophobia/transphobia and the corresponding research in indic languages -is underserved and low-resource, another factor contributing to the difficulty in processing social media texts is code-mixing -a phenomena in which multilingual speakers switch between two or more languages in a conversation with the aim to be more expressive. popular language models tend to perform adversely when applied to code-mixed text and hence newer techniques need to be adopted to handle this situation @xcite . the pretraining and fine-tuning paradigm has taken extensive advantage of transformer based large multilingual models which perform well in cross-lingual scenarios. in this paper we explore the performance of a number of such models when fine-tuned on a dataset for detecting homophobia and transphobia. surprisingly, our experiments also show that these multilingual models exhibit reasonably accurate performance on code-mixing tasks, even without any previous exposure to codemixing during pretraining. the remainder of the paper is organized as follows: section 2 talks about the previous related work in this domain. section 3 gives a detailed explanation of the methods used in the system and section 4 describes the corresponding experimental settings. we mention the detailed results in section 5, conduct an ablation study in section 6 and conclude our discussion with section 7. ## related work to the best of our knowledge no prior work identifying either homophobia or transphobia directly exists in recent literature. however, offensive language detection, in general, in dravidian languages has been the focus of multiple research works in the past @xcite . @xcite at hasoc-dravidian-codemix-fire2020 trained an svm classifier using tf-idf features on code-mixed malayalam text and an xlm-roberta based classifier on codemixed tamil text to detect offensive language in twitter and youtube comments. sai and sharma (2020) fine-tuned multilingual transformer models and used a bagging ensemble strategy to combine predictions on the same task. @xcite developed fusion models by ensembling cnns trained on skip-gram word vectors using fasttext along with fine-tuned bert models. a neural classification head was trained on the concatenated output obtained from the ensemble. a number of approaches have been deployed to tackle code mixing in indic languages as well, since multilingual transformer models lack the complexity to extract linguistic features directly from code switched text. @xcite used a selective translation and transliteration technique to process tamil code-mixed youtube comments for offensive language identification. they converted code-mixed text to native tamil script by translating english words and transliterating romainzed tamil words. similar technique was used by @xcite and @xcite . ## methodology this shared task was formulated as a multiclass classification problem where the model should be able to predict the existence of any form of homophobia or transphobia in a youtube comment. the entire pipeline consists of two main components -a classification head on top of different popular models based on the transformer architecture, and a data augmentation technique for oversampling the english dataset. these components have been explained in further detail ahead. ## experimental setup in this section we review the setup needed to reproduce the experiments. ## conclusion and future work homophobia and transphobia have not been the focus of many umbrella hate speech detection tasks. we examined the ability of pretrained large transformer-based models to detect homophobia and transphobia in a corpus of youtube comments written in english and tamil. experimental results demonstrated that multilingual bert performed the best on both language tasks, and the code-mixed task as well, without being exposed to any codemixing beforehand. this can be attributed to its capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. from section 6 we also observed that the effect of preprocessing was largely dependent on the choice of language setting. this makes sense considering the difference in underlying language constructs. tamil, for instance, does not make use of standard english-based punctuation marks. on the other hand, we conclude that the choice of an effective da techniqe depends on the underlying task and the data source. social media data often lacks linguistic purism and hence, token perturbations such as those introduced by eda did not help. in the future, we would like to adopt a more aggressive da technique such as that involving text generation (text in-filling, generating typos) or an auxilliary dataset (knn, lm decoding). we would also like to evaluate the effect of translation and transliteration on code-mixed text classification.