Sammaan@ LT - EDI - ACL 2022: Ensembled Transformers Against Homophobia and Transphobia (2022)

Abstract: Hateful and offensive content on social media platforms can have negative effects on users and can make online communities more hostile towards certain people and hamper equality, diversity and inclusion. In this paper, we describe our approach to classify homophobia and transphobia in social media comments. We used an ensemble of transformer-based models to build our classifier. Our model ranked 2nd for English, 8th for Tamil and 10th for Tamil-English.

## introduction social media platforms allow people from all walks of life to connect with each other. however, abusive and hateful content on these platforms can take a psychological toll on its users @xcite . lesbian, gay, bisexual and transgender individuals are more vulnerable to mental illness as compared to their heterosexual peers @xcite @xcite . hence, it becomes even more important to be able to detect such hateful content for vulnerable individuals. there has been a lot of work done in the domain of hate speech detection @xcite . there has also been work on hate speech intervention @xcite . shared tasks like semeval 2019 task 6 have focused on identifying and categorizing offensive language on social media @xcite . datasets for this task have been created in multiple languages as well. @xcite created a hindi-english code mixed text dataset for hate speech detection from tweets on twitter. @xcite created a 1000 tweets arabic dataset for offensive language detection with special tags for vulgarity and hate speech. sigurbergsson and derczynski (2020) created a danish hate speech detection dataset containing 3600 user generated comments social media websites. there have been datasets created for greek @xcite and turkish @xcite as well. chakravarthi et al. (2021a) created a code-mixed tamil,malayalam and kannada dataset for offensive language identification. support vector machines, long short-term memory networks, convolutional neural networks and now transformer based architectures have been used to detect hate speech. however, there has not been much work in trying to specifically identify homophobic or transphobic text. in this paper, we will describe our approach for classifying transphobic and homophobic comments in the dataset provided by @xcite as a part of the shared task on homophobia and transphobia detection in social media comments @xcite . ## dataset description the dataset consists of a total of 15,141 comments in 3 languages: english, tamil and tamil-english code-mixed (refer to ## methodology in this section we will describe the models used in our experimentation. ## experiments and results the only pre-procesesing step done on the dataset before training was the change of emojis to text using the demoji library in python foot_0 . our pipeline comprises an ensemble of several transformerbased language models (tlm), namely: bert, roberta, and hatebert for the english track and indicbert for the tamil and tamil-english tracks. three copies of each tlm are used with different parameter initializations in each track. this allows for the copies to capture different features of the data. in addition to this, for each track, a layer of attention is applied to each constituent encoder layer outputs of the tlms. this is necessary since model accuracy macro precision macro recall macro f1 weighted precision weighted recall weighted f1 bert 0.92 0.48 0.42 0.44 0.9 0.92 0.91 roberta 0.93 0.64 0.36 0.36 0.93 0.94 0.9 hatebert 0.94 0.56 0.43 0.47 0.92 0.94 0.92 ensemble 0.94 0.52 0.47 0.49 0.93 0.94 0.94 in the english track, we also use a pretrained hate-speech detection model implemented on hug-gingface @xcite . architecturally, is a byt5-base model @xcite finetuned on huggingface's tweets_hate_speech_detection dataset @xcite . the prediction probabilities are generated by each model of a track are passed as input features to a random forest classifier. this helps further optimize our predictions by weighing the importance of the different architectures for the task. each of the tlm pipelines was finetuned upon cross entropy loss using adamw optimizer @xcite (β 1 = 0.9, β 2 = 0.999, ϵ = 10 -8 ) with an initial learning rate of 2e -5 for 6 epochs each using a linear scheduler. the epoch checkpoint with the highest validation f1 score was selected for further use. the hyperparameters of the random forest classifier were estimated using 10 seeds and 100 iterations of bayesian optimization. the ensemble classifier was trained with a learning rate of 1.0. ## conclusion and future work in this paper, we described our approach for homophobia and transphobia detection in english, tamil and tamil-english. we used an ensemble of three transformed based models along with a pre-trained hate detection model to do the classification for english. our model was ranked 2nd for the english classification task. for the tamil and tamil-english dataset three copies of the in-dicbert model was used to make our ensemble based model. the models placed 8th and 10th for tamil and tamil-english model respectively. in the future, we can use data augmentation methods like paraphrasing and back translation to increase the diversity and quantity of homophobic and transphobic text. we can also incorporate transliteration into the pipeline for tamil-english code mixed text since indicbert is not trained on code mixed text. we could also try to finetune transformers pre-trained on code mixed data.