SJ _ AJ @ D ravidian L ang T ech- EACL 2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification (2021)

Abstract: In this paper we present our submission for the EACL 2021-Shared Task on Offensive Language Identification in Dravidian languages. Our final system is an ensemble of mBERT and XLM-RoBERTa models which leverage task-adaptive pre-training of multilingual BERT models with a masked language modeling objective. Our system was ranked 1st for Kannada, 2nd for Malayalam and 3rd for Tamil.

## introduction the task of offensive language identification aims to identify offensive language used in social media texts, which could be in the form of threats, abusive language, insults @xcite . social media platforms like twitter, facebook, youtube are now mediums where a large number of people interact and express their opinions. conflicts and target abuses are common on social media platform. offensive language detection is a form of social media monitoring which is best done by building computational models and automatic methods due to the large stream of data involved. code-switching 2 is a phenomenon common in bilingual and multilingual communities where words from two (or more) languages are used in the same sentence @xcite @xcite and is very commonly used for social media interactions. thus it is vital to build models that are able to process code-switched data for the relevant social media tasks. previous shared tasks in the domain of offensive language identification have focused on identifying offensive language in english @xcite , arabic, 1 code and pretrained models are available at github.com/murali1996/eacl2021-offenseval-dravidian. greek, danish and turkish @xcite . the datasets used for this shared task in dravidian languages are characterized by code-mixing. pretrained bert models @xcite ) and their multilingual versions-mbert and xlm-roberta @xcite have produced state of the art results in multiple tasks. these models have also been used to built best performing sentiment analysis systems in various shared tasks for code-mixed datasets @xcite @xcite . in this paper we present our submission for the shared task on offensive language identification in dravidian languages. we employ task-adaptive pre-training with a masked language modeling objective and ensemble predictions from mbert and xlm-roberta models for our final submission. ## dataset the dataset statistics for the task of offensive language identification in malayalam @xcite and kannada @xcite are shown in ## conclusions in this paper, we present our submission for eacl-2021 shared task on offensive language identification in dravidian languages. our system was an ensemble of 6 different mbert and xlm-roberta models fine-tuned for the task of offensive language identification. the bert models were pre-trained on given datasets in malayalam, tamil and kannada. our system ranked 1st for kannada, 2nd for malayalam and 3rd for kannada. we also proposed a fusion-architecture to leverage character-level, subword-level and word-level embedding to improve performance.