UMUT eam@ T amil NLP - ACL 2022: Abusive Detection in T amil using Linguistic Features and Transformers (2022)

Abstract: Social media has become a dangerous place as bullies take advantage of the anonymity the Internet provides to target and intimidate vulnerable individuals and groups. In the past few years, the research community has focused on developing automatic classification tools for detecting hate-speech, its variants, and other types of abusive behaviour. However, these methods are still at an early stage in low-resource languages. With the aim of reducing this barrier, the TamilNLP shared task has proposed a multi-classification challenge for Tamil written in Tamil script and code-mixed to detect abusive comments and hope-speech. Our participation consists of a knowledge integration strategy that combines sentence embeddings from BERT, RoBERTa, FastText and a subset of language-independent linguistic features. We achieved our best result in code-mixed, reaching 3rd position with a macro-average f1-score of 35%.

## introduction some users make use of social networks to attack others. bullies target vulnerable individuals groups with the goal of putting them down. this harassment is done on basis of traits such as sexual orientation, religious affiliation, gender, or ethnicity. this speech is known as hate-speech and its automatic detection has recently been explored because the number of daily posts on social networks make it impossible to review all of them manually. the biggest challenges of automatic hate classification are the use of figurative language and that it is not enough just to use offensive language to consider a document as hate speech. besides, although the performance of hate-speech detectors is not bad (at least in controlled environments), they are language and cultural dependent. this makes it difficult to automatically detect hope and hate speech in low- * corresponding author resource languages like tamil, where some of the state-of-the-art techniques have yet to be explored. in these working-notes, the participation of the umuteam in the tamilnlp shared task @xcite (acl-2022) is described. in this shared task, the organisers want the participants to detect abusive comments in comments posted in youtube @xcite @xcite . this is a multi-classification task. the labels are misandry, counter-speech, misogyny, xenophobia, hope-speech, homophobia, transphobia, and noneof-the-above. the overall performance of each submission is measured using the macro average precision, recall and f1-score. two datasets are published. one in tamil script and another in tamil using latin characters (codemixed). the comments from youtube are mostly composed by only one sentence. the dataset annotators rate each comment individually (that is, the annotators did not know if the comment is response to another comment or which is the context of the video). the task organisers published the datasets divided into training and development. ## methodology our methodology is depicted in figure 1 . in a nutshell, it can be described as follows. for both datasets, we extract four feature sets: lf, se, bf, and rf. the details of each feature set are described in more detail in these working notes. next, we train a neural network model for each feature set. we use these neural networks to build a new model based on ensemble learning. this new model combines the predictions of each model. besides, we also evaluate a knowledge integration strategy. with the knowledge integration strategy, a new neural network is trained with all the feature sets at once. for this, we connect each feature set to a input layer and combine their weights in a new hidden layer. finally, we select the best strategy and obtain the predictions of the official test split. tamil (native-script) bert embeddings (bf) linguistic features (lf) sentence embeddings (se) results tamil (code-mixed) roberta embeddings (rf) knowledge integration ensemble learning @xcite . it is worth noting that fasttext has a model for tamil @xcite . fasttext provides a tool to extract sentence embeddings. these embeddings are made up of the average of all the words in each document. the embeddings obtained from fasttext are non contextual (they ignore word order). the third and forth feature sets are sentence embeddings from bert (bf) @xcite and roberta (rf) @xcite . in case of tamil, we use multilingual bert @xcite and xlm roberta @xcite . to extract the sentence embeddings from bert and roberta we conduct a hyperparameter se-lection stage that consisted in the evaluation of 10 models with tree of parzen estimators (tpe) @xcite . we evaluate a weight decay between 0 and .3, 2 batch sizes (8 and 16 foot_1 ), four warm-up speeds (between 0 and 1000 with steps of 250), from 1 to 5 epochs, and a learning rate between 1e-5 and 5e-5. once we obtained the best configuration for bert and for roberta, we extract their sentence embeddings extracting the [cls] token @xcite . the next step in our pipeline is the training of the neural network models. for this, we conduct several hyperparameter optimisation stages with tensorflow and raytune @xcite . this stage is used for each feature set (lf, se, bf, rf) and for the knowledge integration strategy (lf + se + bf + rf). each hyperparameter optimisation stage evaluated 20 shallow neural networks and 5 deep neural networks. the shallow neural networks contains one or two hidden layers max with the same number of neurons per layer. for these, we evaluate linear, relu, sigmoid, and tanh as activation functions. the deep-learning networks can be from 3 to 8 layers. besides, each hidden layer can have different number of neurons. these hidden layers and their neurons are arranged in shapes, namely brick, triangle, diamond, rhombus, and funnel. for the deep neural networks we evaluated sigmoid, tanh, selu and elu as activation functions. in these experiments, we test two learning rates: 10e-03 and 10e-04. we also evaluate large batch sizes (128, 256, 512) due to class imbalance. our objective is that every batch has sufficient number of instances of all classes. besides, we also include a regularisation mechanism based on dropout, testing different ratios between .1 and .3. besides, we include the confusion matrix for code-mixed (top) and tamil-script (bottom) in figure 2 . with the confusion matrix, we can observe what are the wrong classifications made by each model. as expected, the none-of-the-above-label (that is, the neutral label) is the label that has the larger number of wrong classifications. in case of tamil-script, we can observe that documents labelled as hope-speech are commonly misclassified. predicted c o u n te r-s p e e c h h o m o p h o b ia h o p e -s p e e c h m is a n d ry m is o g y n y n o n e -o f-th e -a b o v e n o t-ta m il tr a n s p h o b ic x e n o p h o b ia ## results and discussion one of the biggest challenges in this shared task is that the codalab leader board is disabled. therefore, we could not review that the output file is correct. ## conclusions and promising research lines this working notes describe the participation of the umuteam in the tamilnlp-acl2022 shared task, concerning abusive detection in tamil written in tamil-script and code-mixed. in this work, we have combined four feature sets from linguistic features to three types of sentences embeddings. we have combined these features in a knowledge integration strategy. we reached the 3rd position in code-mixed and 9th position in tamil-script. as future work, we will focus on the development of language-independent linguistic features. for example, we have adapted umutextstats to use different pos models from stanza @xcite ), which has allowed to extend the subset of the linguistic features for tamil. besides, we will compile idioms and extending the dictionaries to improve the figurative language identification (del pilar salas-zárate et al., 2020), thus improving the performance of automatic document classification.