Codewithzichao@ D ravidian L ang T ech- EACL 2021: Exploring Multilingual Transformers for Offensive Language Identification on Code Mixing Text (2021)

Abstract: This paper describes our solution submitted to shared task on Offensive Language Identification in Dravidian Languages. We participated in all three of offensive language identification. In order to address the task, we explored multilingual models based on XLM-RoBERTa and multilingual BERT trained on mixed data of three code-mixed languages. Besides, we solved the class-imbalance problem existed in training data by class combination, class weights and focal loss. Our model achieved weighted average F1 scores of 0.75 (ranked 4th), 0.94 (ranked 4th) and 0.72 (ranked 3rd) in Tamil-English task, Malayalam-English task and Kannada-English task, respectively.

## introduction offensive language identification is a research field that has received increasing attention in recent years. especially with the rise of social media platforms, it is essential to identify offensive language on code-mixed social media texts. it is a challenging task to identify offensive language on social media texts. additionally, a lot of work has been done for offensive language identification in languages like english, greek or spanish @xcite @xcite ), but little work has been done for offensive language identification of code-mixed text in dravidian languages. shared task on offensive language identification in dravidian languages (tamil-english, malayalam-english and kannada-english) has changed this situation. the goal of this shared task is to identify offensive language on code-mixed text in dravidian languages. the code-mixed text is collected from social media platforms. it is a comment or post level multilingual classification task that given a comment or post in code-mixed tamil-english language, malayalam-english language and kannada-english language, systems have to classify it into not-offensive, offensive-untargeted, offensive-targeted-individual, offensive-targetedgroup, offensive-targeted-other, or not-in-indentedlanguage. in this paper, we explore multilingual transformers on code-mixed text for offensive language identification in dravidian languages. inspired by the conclusion that lexical overlap among different languages can be helpful to improve the performance of the model in a single language given in @xcite , we combine the training data of three languages and trained our multilingual model on the mixed data. besides, we solve the classimbalance problem existed in training data by class combination, class weights and focal loss. finally, we use adversarial training to further improve performance of our model. with these approaches, we achieved 4th rank in tamil-english, malayalam-english task and 3rd rank in kannada-english task. ## data we used the data provided by the organizers of shared task on offensive language identification in dravidian languages @xcite @xcite , which have been annotated well at comment or post level. the numbers of tamil, malayalam and kannada training data are 35139, 16010 and 6217, respectively. the statistics of data are shown in there are four methods used to preprocess the data as follow: the experimental results show that removing these noise can improve the performance of our model. ## system in this section, we first present our model for offensive language identification task, and then introduce the solution which is used to solve the classimbalance problem and improve the robustness and generalization of our model. ## conclusion this paper describes our solution submitted to shared task on offensive language identification in dravidian languages (tamil-english, malayalam-english, and kannada-english). first of all, we use multilingual models trained on mixed data of three code-mixed languages to obtain better performance in all three of code-mixed languages. secondly, we use class combination, class weights and focal loss to solve the class-imbalance problem existed in training data. finally, we use adversarial training to further improve the performance of our model. we achieved 4th rank in tamil-english, malayalam-english task and 3rd rank in kannada-english task. in future research, we will further consider the differences among different languages