UMUT eam@ LT - EDI - ACL 2022: Detecting homophobic and transphobic comments in T amil (2022)

Abstract: This working-notes are about the participation of the UMUTeam in a LT-EDI shared task concerning the identification of homophobic and transphobic comments in YouTube. These comments are written in English, which has high availability to machine-learning resources; Tamil, which has fewer resources; and a transliteration from Tamil to Roman script combined with English sentences. To carry out this shared task, we train a neural network that combines several feature sets applying a knowledge integration strategy. These features are linguistic features extracted from a tool developed by our research group and contextual and non-contextual sentence embeddings. We ranked 7th for English subtask (macro f1-score of 45%), 3rd for Tamil subtask (macro f1-score of 82%), and 2nd for Tamil-English subtask (macro f1-score of 58%).

## introduction this document outlines the participation of the umuteam in the workshop on language technology for equality, diversity, inclusion (lt-edi, acl 2022) @xcite . specifically, we describe our participation in a shared task regarding the identification of homophobic and transphobic comments in youtube, written in english, tamil, and a transliteration from tamil to roman script combined with english sentences. homophobic and transphobic messages harm society, and limit individual and collective freedom. therefore, the consequences of this kind of hate-speech is especially dangerous for children @xcite . the details of the provided datasets can be found at @xcite . these datasets are divided into three splits, namely, training, validation, and test. ## related work there are several surveys in the bibliography related with the identification of homophobic and transphobic comments. for instance, the works described at @xcite and @xcite . homophobic and transphobic comments are usually categorised as a form of hate-speech based on sexism or gender discrimination. these surveys indicate that there is a generic pipeline for building hate-speech detectors, that are based on the development of automatic document classification systems. the features for extracting data from textual sources are, usually, statistical methods. to name just a few, we mention the bag of word model, tf-idf weights, topic modelling, and word and sentence embeddings. the models are traditional machine learning's classifiers (support vector machines, logistic regression, or random forest, among others) and different neural network architectures based on convolutional, recurrent neural networks and the usage of stateof-the-art models based on transformers, such as bert. our research group has experience dealing with hate-speech. in @xcite , the spanish misocorpus 2020, concerning misogyny identification, was released and evaluated. this dataset is released into three minor splits, concerning (1) the identification of misogyny towards relevant women, (2) to find the differences between spanish of spain and latin-america, and (3) to identify general traits concerning misogyny, such as stereotypes of derailing. in @xcite , we conduct an in-depth analysis concerning linguistic features and word and sentence embeddings. specifically, we evaluated which are the best strategies to combine these features to build better hate-speech detectors. as part of the doctoral thesis of one of the members of the team, in this shared-task we evaluate a subset of linguistic features that are languageindependent. therefore, a secondary objective of our participation is to observe if the combination of linguistic features and embeddings improves the performance of the automatic document classifiers. ## methodology our methodology can be summarised as follows. first, the pre-process the documents by removing extra spaces, blank lines, certain punctuation symbols, and emojis. just for the english subtask, we also normalised the text by expanding acronyms and transformed the whole text into their lowercase form. second, we extracted four feature sets that include linguistic features (lf), pretrained word embeddings from fasttext (we), sentences embeddings from fasttext (se), and sentence embeddings from bert (bf). third, we conduct an hyperparameter tuning strategy to build a neural network per feature set and one additional neural network that combines all feature sets (knowledge integration). forth, we build two additional systems based on ensemble learning. finally, we evaluate these methods to select the best approach for the final submission. next, we describe the feature sets employed in this work. the first feature set, lf, is a subset of language-independent features computed from the umutextstats tool @xcite . these features are related to stylometry, part-of-speech, emojis, and social media jargon. the second feature set, we, is based on non-contextual embeddings from fasttext. for this we use the pretrained embeddings of english @xcite and the pretrained embeddings of tamil @xcite . fasttext calculates sentences embeddings by averaging word embeddings. the third feature set, se, are sentence embeddings from fasttext. the forth feature set, bf, is based on contextual sentence embeddings. we use bert for english and the distilled version of multilingual bert @xcite . we use the distilled version because our machine could not train large batches with default bert. the sentence embeddings are extracted with the [cls] token @xcite . to obtain the sentence embeddings from bert, we evaluate 10 models with tree of parzen estimators (tpe) @xcite . the evaluated parameters were the weight decay, the batch size, the warm-up speed, the number of epochs, and the learning rate. next, we train a neural network for each feature set and a neural network that combines all the feature sets using a knowledge integration strategy. for each training, we conduct a hyperparameter optimisation stage. the training is performed with raytune @xcite . in this stage, we evaluated shallow neural networks and deep neural networks. the main difference is the number of layers, using only one or two in shallow neural networks whereas deep neural networks use up to 8 hidden layers. another difference is the composition of the neurons in each layer. in shallow neural networks, all the layers have the same number of neurons. in deep neural networks, on the other hand, we arranged the neurons in different shapes (brickshape, triangle-shape, diamond-shape, rhombusshape, short and long funnel-shape). it is worth noting that the knowledge strategy allows to combine the features into the neural network consists in outputting each one into a different layer and then combine all the results into a new hidden layer. this strategy allows us to include two specific architectures with the non-contextual word embeddings from fasttext: convolutional and recurrent neural networks. these networks exploit different characteristics of a text represented as a sequence. convolutional networks exploit the spatial dimension, as it can make up new features from words that are together. recurrent neural networks, on the other hand, exploits the temporal dimension. specifically, we evaluate two bidirectional recurrent layers (bilstm and bigru). besides, we evaluate several activation functions to connect the hidden layers, different learning rates and a dropout for regularisation. in order to observe the performance of the best neural network with the validation split we obtain every confusion matrix (see figure 1 ). ## results the official results in the leaderboard are depicted in tables 3 , 4 , 5 for english, tamil, and tamil-english respectively. we can observe that we achieved good results for tamil and tamil-english, achieving the third and second position in the leaderboard. however, our results were more limited with the english dataset, in which we ranked 7th. we achieved a macro f1-score of 45% in the english dataset (see regarding tamil (see finally, the results from the tamil-english dataset output a macro f1-score of 58% (see ## conclusions in this article, we have summarised the participation of the umuteam in a task concerning the identification of homophobic and transphobic in social media posts. we are very pleased with our participation as we have participated with all the datasets as we have achieved competitive results. as future work, we will continue adapting these techniques to tamil and english, specially those focused on figurative language (del pilar @xcite . one limitation that we found on our approach is that we do not handle code-mixed language properly. as future work, we will explore the reliability of using multilingual resources.