Maoqin @ D ravidian L ang T ech- EACL 2021: The Application of Transformer-Based Model (2021)

Abstract: This paper describes the result of team-Maoqin at DravidianLangTech-EACL2021. The provided task consists of three languages(Tamil, Malayalam, and Kannada), I only participate in one of the language task-Malayalam. The goal of this task is to identify offensive language content of the code-mixed dataset of comments/posts in Dravidian Languages (Tamil-English, Malayalam-English, and Kannada-English) collected from social media. This is a classification task at the comment/post level. Given a Youtube comment, systems have to classify it into Not-offensive, Offensive-untargeted, Offensive-targeted-individual, Offensive-targeted-group, Offensive-targeted-other, or Not-in-indented-language. I use the transformer-based language model with BiGRU-Attention to complete this task. To prove the validity of the model, I also use some other neural network models for comparison. And finally, the team ranks 5th in this task with a weighted average F1 score of 0.93 on the private leader board.

## introduction offensive language refers to direct or indirect use of verbal abuse, slander, contempt, ridicule, and other means to infringe or damage the dignity, spiritual world, and mental health of others. it will seriously affect the mental state of others, disrupt work, the life and learning order of others, and seriously pollute the public opinion environment of the entire network @xcite . due to the development of the internet and the popularity of anonymous comments, many offensive languages have spread on the internet and caused trouble to relevant personnel @xcite . relevant organizations should take measures to prevent this from happening. it is unrealistic to judge whether online sentences are completely offended by humans. there-fore, mechanical methods must be used to distinguish whether the language is offensive. the task is to directly test whether the system can distinguish offensive language in dravidian languages. dravidian languages are a group of languages spoken by 220 million people, predominantly in southern india and northern sri lanka, but also in other areas of south asia. the dravidian languages were first recorded in tamili script inscribed on cave walls in tamil nadu's madurai and tirunelveli districts in the 6th century bce. the dravidian languages are closely related languages the are under-resourced (chakravarthi, 2020). existing deep learning and pre-training models have achieved good results on other tasks @xcite , so i use the deep learning method to deal with the related task. according to the latest related research progress, the transformer-based language model has become my preferred model. because the pre-trained and fine-tuned transformersbased models have shown excellent performance in many nlp problems, such as sentiment classification and automatic extraction of text summaries. so i choose albert @xcite as my basic model in this task. to get a more effective and higher accuracy model, bigru combined with attention. to prove the effectiveness of this model, i have also done comparative experiments with other neural networks. in this task, my model is an effective way to perform well. to obtain as much effective information as possible from the limited data, i also use the 5-fold cross-validation method. my model achieves the desired result. the rest of this article is structured as follows. section 2 introduces related work. model and data preparation are described in section 3. experiments and evaluation are described in section 4. section 5 describes the results of my work. the conclusions and future work are drawn in section 6. ## related work there are many competitions about offensive language detection(such as hasoc @xcite and trac (kumar et al., 2018)), and many corresponding methods have been produced. people often tend to abstract this task into a text classification task @xcite ). text classification is called extracting features from original text data and predicting the category of text data based on these features. in the past few decades, many models for text classification have been proposed @xcite . from the 1960s to the 2010s, text classification models based on shallow learning dominated. shallow learning means statistical-based models such as naive bayes (nb), k nearest neighbors (knn) @xcite and support vector machines (svm). compared with earlier rulebased methods, this method has obvious advantages in accuracy and stability. however, these methods still require functional design, which is time-consuming and expensive. in addition, they usually ignore the natural order structure or context information in the text data, which makes learning the semantic information of words difficult. since the 2010s, text classification has gradually changed from a shallow learning model to a deep learning model. compared with methods based on shallow learning, deep learning methods avoid the manual design of rules and functions and automatically provide semantically meaningful representations for text mining. therefore, most of the text classification research work is based on dnn @xcite , which is a data-driven method with high computational complexity. few studies have focused on shallow learning models to solve the limitations of computation and data. the shallow learning model speeds up the text classification speed, improves the accuracy, and expands the application range of shallow learning. the shallow learning method is a type of machine learning. it learns from data, which is a predefined function that is important to the performance of the predicted value. however, element engineering is an arduous and giant job. before training the classifier, we need to collect knowledge or experience to extract features from the original text. the shallow learning method trains the initial classifier based on various text features extracted from the original text. for small data sets, under the limita- ## methodology and data an overall framework and processing pipeline of my solution are shown in figure 1 . in my job, i use the albert model as my base model and take bigru-attention behind it. my model is shown in figure 2 . ## results the output of the classification result is shown in figure 4 . we can see that the label of of f ensive -t argeted -insult -other, of f ensive -t argeted -insult -individual, and of f ensive -t argeted -insult -group