B p H igh@ T amil NLP - ACL 2022: Effects of Data Augmentation on Indic-Transformer based classifier for Abusive Comments Detection in T amil (2022)

Abstract: Social Media platforms have grown their reach worldwide. As an effect of this growth, many vernacular social media platforms have also emerged, focusing more on the diverse languages in the specific regions. Tamil has also emerged as a popular language for use on social media platforms due to the increasing penetration of vernacular media like Sharechat and Moj, which focus more on local Indian languages than English and encourage their users to converse in Indic languages. Abusive language remains a significant challenge in the social media framework and more so when we consider languages like Tamil, which are low-resource languages and have poor performance on multilingual models and lack language-specific models. Based on this shared task, “Abusive Comment detection in Tamil@DravidianLangTech-ACL 2022”, we present an exploration of different techniques used to tackle and increase the accuracy of our models using data augmentation in NLP. We also show the results of these techniques.

## introduction the growth of social media platforms has been a significant factor in increasing awareness and connecting the world. social media has changed the conventional way of communication and has introduced certain short forms and slang that are not present in the traditional vocabulary of any language. 1 at the same time, social media platforms have given rise to a new dynamic of cyber harassment utilizing the veil of anonymity that most platforms provide. abusive language is a broad term often used to describe the posts and comments on social media platforms written to cyberbully, spread toxicity, spread hate, hurt others based on sex, caste, or creed @xcite . in the recent past, many social media platforms have updated their guidelines and added moderation policies to curb the spread of abusive language on them. platforms like facebook, youtube, and twitter have added features to report several posts/videos and comments. many social media platforms also employ content moderators to clamp down on abusive language on their platforms. still, this strategy is not sustainable for the long term as social media users continue to grow, and this approach cannot scale @xcite . many content moderators feel the mental and psychological effects of viewing and moderating several extreme contents and are profoundly affected by such content. 2 hence many platforms have started building automated abusive language detection and classification systems to improve their moderation capabilities. in india, vernacular media faces more challenging problems dealing with more diverse languages and code-mixed data. for example, dealing with a vernacular language like tamil is challenging as it is a low-resource language. hence, it has insufficient datasets and code-mixed data, and data belonging to both tamil script and transliterated data. sometimes these challenges can lead to difficulty for the social media platforms to detect and remove the abusive language, leading to skirmishes with the government. 3 sharechat and moj also organized a challenge recently to improve the abusive language detection systems in indic languages and released their proprietary dataset for further research. 4 a shared task on "offensive language detection in dravidian languages" was also introduced in the "first workshop on speech and language technologies for dravidian languages at eacl 2021". this shared task consisted of a large cor-pus of comments/posts in code-mixed languages tamil-english, kannada-english, and malayalam-english @xcite @xcite . further extending this shared task, the organizers of "the second workshop on speech and language technologies for dravidian languages at acl 2022" have released a shared task on "abusive comment detection in tamil" this task focuses specifically on abusive language detection in tamil. it consists of datasets of both tamil as well as code-mixed tamil-english @xcite @xcite . our paper makes a two-fold contribution to the shared task. first, we experiment with the stateof-the-art transformer models pre-trained on indian languages. secondly, we show how data augmentation techniques in nlp perform in this task and how training with word-level augmented sentences affect our model accuracy. we also provide the trained model weights and the implementation code. foot_3 2 related work @xcite investigated multilingual abusive comment detection focusing on english and german languages. they used a publicly available dataset of twitter hate speech made by @xcite for english and for german they used the 2018 germeval shared task data set @xcite . they experimented with different text classification algorithms and found that there was no single algorithm which gave the best results on both the languages. they got best results on the german dataset by using svm , 72.01 f-score and best results on english dataset were obtained using xgboost, 80.49 f-score. pamungkas et al. ( 2021 ) wrote a summary paper on multilingual and multi domain abusive language detection, in the paper the authors highlighted various different techniques and datasets created and used by different researchers to properly define and solve the multilingual challenges related to abusive language detection. in the papers the author mentions several transformer architecture based models like multilingual bert and xlm roberta which are pre trained on corpus of several languages and can be finetuned for various tasks. they also mention various datasets in indic languages like hindi and code mixed hindi-english which are created specifically for the purpose of hate speech/abusive language detection, like hasoc, 2019. 6 khanuja et al. ( 2021 ) published a research paper along with a new transformer language model 'muril' based on the bert architecture which is specifically designed for indian languages. in the paper the authors also compared performance of both multilingual bert and muril on various tasks in indian languages.the model currently has support for 17 indian languages. in the papers the author shows that muril beats multilingual bert across all tasks and benchmarks in indian languages. on the famous xtreme benchmark, multilingual bert gives an average performance of 59.1 whereas muril gives a 68.6 average performance. feng et al. ( 2021 ) published a survey paper on data augmentation approaches utilized in nlp. the paper's authors discussed how data augmentation techniques could help fix the class imbalance. they also discussed various data augmentation methods like backtranslation @xcite and easy data augmentation (eda) @xcite . @xcite presented a novel data augmentation technique called contextual augmentation. in this technique the authors used a bidirectional language model to replace words in the given input with other words according to the context. they further showed how this technique helps improve the accuracy of classifiers based on recurrent neural networks(rnn) and convolutional neural networks(cnn). ## dataset description the shared task on abusive comment detection in tamil-acl 2022 @xcite is a comment classification problem that can be further described as a multi-class text classification problem in tamil native script and tamil-english code-mixed. the main objective is to build two separate systems that can classify comments, one for tamil native script and another for code-mixed tamil-english. the purpose can also be redefined as developing a common system that can classify comments of both tamil and tamil-english languages. this paper treats the objective as building a common system for both languages rather than separate, to have a more standard approach. the dataset was generated by scraping youtube comments belonging to tamil and tamil-english code-mixed languages and annotated on comment level by linguists/annotators based on the platform's set guidelines and code of conduct. the dataset is split into two datasets based on the language. the labels used for annotation of the dataset are misogyny, misandry, homophobia, transphobia, xenophobia, counter speech, hope speech, and none of the above. the dataset is further split into the training and development sets. the dataset consists of rows that contain the comment text and the label assigned to that comment. ## methodology this section discusses the experiments and approaches undertaken to build a system for abusive comment detection. as explained earlier, we create a common system for both tamil and tamil-english languages, and hence for this purpose, we combine the dataset of tamil and tamil-english languages to make a combined dataset. figure 1 shows a flowchart of the different approaches that we explored. ## conclusion we explored the effects of data augmentation techniques on the indic-transformer based classifier created using muril transformer on the task of abusive comment detection in tamil. we observe a negative result in the case of word-level augmentation using contextual models(muril) and an improvement in performance in the case of augmentation using non-contextual word embeddings(tamil fasttext). as we further try to speculate why our augmentation technique based on contextual models failed to yield a better result, we consider the reasons stated in @xcite , which show that data augmentation techniques help improve performance on the task only when the approaches provide a language pattern that is not seen before during pretraining of the transformer model. as both the contextual model for augmentation and the indic-transformer used to create the classifier is muril transformer, we cannot observe new linguistic patterns. also, in kobayashi (2018), the authors observe that augmentation based on contextual models might not be able to remain compatible with the annotated labels of the original input and thus might harm the training process. they suggest using information from both label and context to generate word-level augmentations to control this incompatibility.