ZYJ 123@ D ravidian L ang T ech- EACL 2021: Offensive Language Identification based on XLM - R o BERT a with DPCNN (2021)

Abstract: The development of online media platforms has given users more opportunities to post and comment freely, but the negative impact of offensive language has become increasingly apparent. It is very necessary for the automatic identification system of offensive language. This paper describes our work on the task of Offensive Language Identification in Dravidian language-EACL 2021. To complete this task, we propose a system based on the multilingual model XLM-Roberta and DPCNN. The test results on the official test data set confirm the effectiveness of our system. The weighted average F1-score of Kannada, Malayalam, and Tami language are 0.69, 0.92, and 0.76 respectively, ranked 6th, 6th, and 3rd

## introduction with the development of the information society, people have become accustomed to uploading content on social media platforms in the form of text, pictures, or videos. at the same time, they also comment on the content uploaded by other users and interact with each other, thus increasing the activity of social media platforms @xcite . inevitably, however, some users will post offensive posts or comments. the use of offensive discourse is a kind of impolite phenomenon which has negative effects on the civilization of the network community @xcite . it usually has the characteristics of causing conflicts and the purpose of publishing intentionally. the publisher of offensive language may use reproach, sarcasm, swear and other language means to achieve intentional offense, and express a variety of intentions, such as disturbing, provoking, and expressing negative emotions @xcite . most people will take measures to respond to offensive words. the way to respond to the direct conflict of offensive words is mainly rhetorical questions, swear, sarcasm and threat, so as to express dissatisfaction, deny and satirize the other party and provoke the other party. this will further cause conflicts and destroy the harmony of the network environment. many social media platforms use a content review process, in which human reviewers check users' comments for offensive language and other infractions, and which comments have been removed from the platform because of the violation @xcite . it is up to the moderator to decide which comments will be removed from the platform due to violations and which ones will be kept. as the number of network users increases and user activity increases, the manual approach is undoubtedly inefficient. therefore, the automatic detection and identification of offensive content are very necessary. however, offensive words often depend on the emotions and psychology of the listener, and some seemingly innocuous words can be potentially offensive, and words that often seem offensive are watered down by the emotions of the listener. this kind of language phenomenon is not uncommon in real life, either unintentionally or deliberately used to achieve the speaker's expected purpose, which is a challenging work for the current detection system. our team takes part in the shared task of offensive language identification in dravidian languages-eacl 2021 @xcite @xcite . this is a classification task at the comment/post level. the goal of this task is to identify offensive language content of the code-mixed dataset of comments/posts in dravidian languages ( (tamil-english, malayalam-english, and kannada-english)) collected from social media. tamil language is the oldest language in indian languages, malayalam and kannada evolved from tamil language. for a comment on youtube, the system must classify it into not-offensive, offensive-untargeted, offensive-targeted-individual, offensive-targeted-group, offensive-targeted-other, or not-in-indented-language. in our approach, the multilingual model xlm-roberta and dpcnn are combined to carry out the classification task. this method can combine the advantages of the two models to achieve a better classification effect. the rest of the paper is divided into the following parts. in the second part, we introduce the relevant work in this field, which involves offensive language detection and text classification methods. in the third part, we introduce the model structure and the composition of our training data. the fourth part introduces our experimental setup and results. the fifth part is the conclusion. ## related work due to the harm of offensive language to the network environment, the identification of offensive language has been carried out for a long time. research so far has focused on automating the decision-making process in the form of supervised machine learning for classification tasks @xcite . as far back as 2012, @xcite proposed a lexical syntactic feature (lsf) framework to detect offensive content in social media, distinguished the roles of derogatory/profane and obscenity in identifying offensive content, and introduced handwritten syntax rules to identify abusive harassment. in contrast to the start-to-end training model, @xcite proposed an effective transfer learning method, universal language model tuning (ulmfit), which can be applied to any task in natural language processing, and has shown significant results on six text classification tasks. subsequently, abdellatif and elgammal (2020) used the ulmfit transfer learning method to train forward and backward models on arabic datasets and ensemble the results to perform an offensive language detection task. although english is currently one of the most commonly spoken languages in the world, work is ongoing to identify the offensive language in other languages that are less widely spoken. @xcite tested the performance of several traditional machine learning models and deep learning models on an offensive language dataset of greek, and the best results were achieved with the attention model of lstm and gru. ozdemir and yeniterzi (2020) ensembled cnn-lstm, bilstm-attention, and bert three models, combined with pre-trained word embedding on twitter to complete the identification task of offensive turkish language, and achieved a good result. a key challenge in automatically detecting hate speech on social media is to separate hate speech from other offensive languages. @xcite used the crowd-sourced hate speech lexicon to collect tweets containing hate speech keywords. they trained a multi-class classifier to reliably distinguish hate speech from other offensive languages, and found that racist and homophobic tweets were more likely to be classified as hate speech, but sexist tweets were generally classified as offensive. @xcite proposed to extract features at different conceptual levels and apply multilevel classification for offensive language detection. the system leverages a variety of statistical models and rule-based patterns, combined with an auxiliary weighted pattern library, to improve accuracy by matching text with its graded entries. @xcite proposed the ensemble of a recursive neural network (rnn) classifier, which combines various characteristics related to user-related information, such as the user's sexist or racist tendencies, and was then fed to the classifier as input along with a word frequency vector derived from the text content. when there is a large amount of labeled data, increasing the size and parameters of the model will definitely improve the performance of the model.however, when the amount of training is relatively small, the large-scale model may not be able to achieve good results, so solving the problem of model training under the condition of a small amount of target data has become a research hotspot. @xcite proposed a hierarchical attention prototype network (hapn) for fewshot text classification, which designed multiple cross-concerns of a feature layer, word layer, and instance layer for the model to enhance the expressive power of semantic space. the model was validated on two standard reference text classification datasets, fewrel and csid. @xcite built on structural correspondence learning, using untagged documents and simple word translation to induce task-specific, cross-language word correspondence. english was used as the source language and german, french, and japanese were used as the target language to conduct the experiment in the field of cross-language sentiment classification. using english data, ranasinghe and zampieri (2020) trained the model by applying cross-language contextual word embedding and transfer learning methods, and then predicted the effect of cross-language contextual embedding and transfer learning on this task in less resourceintensive languages such as bengali, hindi, and spanish. ## conclusion in this paper, we describe our system in the task of offensive language identification for tamil, malayalam, and kannada language. in this model, the xlm-roberta pre-training model is used to extract semantic information features of the text, and dpcnn is used to further process the output features. at the same time, the hierarchical crossvalidation method is used to improve the training effect. the final results show that our model achieves satisfactory performance. in future work, we will try to adjust the structure of the new model, so as to improve its effect more significantly.