DLRG @ D ravidian L ang T ech- EACL 2021: Transformer based approachfor Offensive Language Identification on Code-Mixed T amil (2021)

Abstract: Internet advancements have made a huge impact on the communication pattern of people and their life style. People express their opinion on products, politics, movies etc. in social media. Even though, English is predominantly used, nowadays many people prefer to tweet in their native language and some- times by combining it with English. Sentiment analysis on such code-mixed tweets is challenging, due to large vocabulary, grammar and colloquial usage of many words. In this paper, the transformer based language model is applied to analyse the sentiment on Tanglish tweets, which is a combination of Tamil and English. This work has been submitted to the the shared task on DravidianLangTech- EACL2021. From the experimental results, it is shown that an F 1 score of 64% was achieved in detecting the hate speech in code-mixed Tamil-English tweets using bidirectional trans- former model.

## introduction recent advancements in the internet technologies and the usage of smart phones changed the behaviour of communication pattern among the people of all walks of life. many people use the social media to post their opinion about various domains, including politics, movies, sports etc. this has led to a sharp increase in offensive posts and hate speech targeted towards an individual or a group or women. recently, the trend of using code-mixed language for tweets has increased, in which the words/phrases from more than one language is used to express their feelings @xcite @xcite . many research works have been reported for identifying the hate speech in english / german tweets using mono-lingual models @xcite , but limited works are performed for under-resourced languages like tamil @xcite . the third tamil sangam period is the period of history from c. 6th century bce 1 to c. 3rd century ce of ancient tamil nadu, kerala and parts of sri lanka (then known as tamilakam) @xcite . the tamilakam (puranuru 168. 18) referred to the whole ancient tamil-speaking territory in the old tamil language, approximately equivalent to the area known today as southern india, consisting of the regions of the present-day indian states of tamil nadu, kerala, parts of andhra pradesh, parts of karnataka, and also known as eelam in northern sri lanka. tamil language is the oldest language in india, all the dravidian languages evolved from tamil language 2 (chakravarthi and muralidaran, 2021; @xcite . the machine learning algorithms with handcrafted features are not sufficient to capture the semantic information from these multi-lingual tweets. this has made the researchers to move towards the sequence models which have proven capability to capture the semantic information from the data using the transformer architecture. among the sequence models, bi-directional encoder representation from transformer(bert) has gained more popularity because of its ability to capture the semantic relationship from both the directions of text and suitable method for understanding contextheavy sentences @xcite @xcite . in this research work, we have studied the effectiveness of bert to identify hate speech in the code-mixed tweets. this work is submitted for the shared task on dravidianlangtech, the first workshop on speech and language technologies for 1 iron age -early historic transition in south indian appraisal 2 tamil-language dravidian languages at eacl 2021. this paper is organized as follows: related works are presented in section 2 followed by the proposed methodology in section 3. experimental details are discussed in section 4 followed by the conclusion in section 5. ## related works sentiment analysis on social media tweets is an important problem being studied for various reasons like identifying the opinion of people about a product, movie or sports etc. extracting the sentiments from the tweets is a challenging task, as the users give their comments explicitly or implicitly. @xcite performed sentiment analysis on movie reviews and proposed a method to determine the explicit opinions by combining the advantages of convolution neural networks with the bidirectional long short memory. the task of identifying the implicit opinions from the tweets is more challenging and many works are reported to address the same. a detailed survey is presented in @xcite . many research works are reported in the literature for hate speech detection in social media tweets. @xcite studied this problem in multi-lingual context and evaluated their deep learning and machine leaning approaches on three different languages viz., english, german and italian. in a study by @xcite on codemixed hindi tweets, the performance of cnn is found to be better than the linear classifiers such as svm. @xcite proposed an ensemble based approach for detecting hate speech in hindi and german languages. to process the multi-lingual queries quickly, differentiating code-mixing and code borrowing is important @xcite @xcite . for this, a relevance based metric is proposed by @xcite to rank the borrowing likeliness of the words in hindi-english tweets. code-mixing is common among many bi-lingual speakers @xcite and to identify hate speech in tamil-english tweets, a corpus is created by @xcite . the overview of the different approaches to address this issue is presented in @xcite . a novel methodology has been proposed by (sainik kumar mahata and bandyopadhyay, 2020) by combining blstm with language tags to identify the hate speech in tanglish tweets. @xcite proposed a new language model bidirectional encoder representations from transformers (bert) that is suitable for many nlp tasks. bert is simple and can be fine-tuned for various downsteam task in nlp. in this work, we have applied bert for the task of identifying the hate speech in code-mixed tamil-english tweets. ## methodology the objective of this shared task is to identify the hate speech in the code-mixed tamil-english tweets that contain any one of the following 6 category labels viz., not-offensive, offensive-targetedgroup, offensive-targeted-individual, offensive-targetedother, offensive-untargeted and not-tamil. the data distribution among the categories is not uniform and 72% of them are not-offensive tweets with the remaining tweets split among the other categories with 7%, 6%, 1%, 8% and 4% respectively. as part of this task, the training and validation set were released with 35,139 and 4388 labelled tweets that followed the same distribution as mentioned above. the code-mixed tanglish tweets contain both tamil and english words and phrases. so, we have converted the tamil words into english terms by using tamil to english mapping corpus. we have utilized nltk(natural language tool kit) package in python to clean the data. for any classification task, the pre-processing steps are important which helps to improve the classifier performance. we have performed some of the cleaning steps like stop word removal,lemmatization and removed the special characters. for example, after pre-processing, the tweet for any classification task, suitable vector representation need to be chosen that can capture the relationship between the terms in multiple aspects. after the pre-processing step, we have used @xcite glove for obtaining vector representation of the words. the reason to choose glove embedding is that, it has the ability to capture the linguistic similarity among the words. in this method, the cosine similarity metric has been employed to find the nearest neighbour, which helps in revealing the rare and relevant terms from the corpus. by this method, we could obtain the interesting patterns among the words and their relationship in a better way. the data set has 6 categories of tweets viz., not offensive (25425), offensive targeted insult group (2557) , offensive targeted insult individual (2343), offensive targeted insult other (454), offensive untargeted (2906) and not-tamil (1454) with imbalanced distribution. as this data set is highly imbalanced, we have applied soup(similarity-based oversampling and undersampling pre-processing) @xcite . in this technique, the number of minority class samples are increased and the number of majority class samples are decreased to make the data set a balanced one. this is performed by removing the most unsafe examples until a desired class cordinality is obtained. the calculation of the safe level is done by using the heterogeneous value difference metric (hvdm). by this method, the multi-class imbalance problem is solved and then we used this balanced data for performing classification task. after applying soup the samples of all the classes are balanced. to determine the sentiment expressed in the multi-lingual tweet, we have also applied the bert model @xcite . bert is the language representation model that extracts the context from the input sentence from both the directions. to capture semantic and linguistic features from the given sentence, a bidirectional encoder representation is applied. in general, tweets may have one or more sentences. bert has the ability to consider these input sentences into a single sequence that can unambiguously result in a better input representation. bert embeddings are effective compared to other language models, as it combines the token embedding, segment embedding and positional embedding. another advantage is that, pre-training of bert combines both masked language model (mlm) and next sequence prediction (nsp). the pre-trained bert can be fine-tuned to suit the downstream tasks. by adding a classification layer, bert can be be used for this task of hate speech identification in code-mixed tamil-english tweets. unlike glove, that deals with word-like tokens, bert considers the word-pieces which are segmented input units. the out-of-vocabulary tokens are also avoided when we use bert language model. compared to other language models, we have chosen bert as it is possible to capture the context in a bidirectional way and efficient than the single directional models. the multi-head attention is applied to identify the key terms that are more important in determining the sentiment of the sentence expressed in tamil. considering the baseline models, such as lstm, blstm, the bert based model performed well on the validation data set. the fine-tuned model was used to predict the sentiment on the released test set. the details of experiments and the results are reported in the next section. ## results and discussion to study the performance of the proposed method, we have conducted the experiments on the released code-mixed tamil-english tweets. all the experiments were carried on a workstation with intel xeon quad core processor, 32 gb ram, nvidia quadro p4000 gpu 8gb. for implementation, we have used python 3, scikit-learn with nltk library. for the base line experiments, we have tried with different vector representatiosn like tf-idf and glove with machine learning techniques. in order to capture the linguistic terms that contribute more to identify the hate speech, we have tried deep learning technique (bert) with attention mechanism. we have applied bert and fine-tuned the parameters. the following parameters were fixed, based on its better performance on the validation set. the experiments were conducted with a learning rate of 3e -5, batch size of 4 with 5 epochs. we have obtained the validation accuracy of 65% for these parameters. we have applied the obtained best model, on the test set and we could achieve the ## conclusion the social media plays an important role in reflecting people's opinion on different issues. this work focused on identifying the hate speech posted in social media that contains code-mixed tamil-english tweets, and is submitted to the dravidianlangtech-eacl2021 shared task. to capture the linguistic terms with higher contextual awareness, we have used bert language model for this task. from the experiments, it is shown that, an f 1 score of 64% has been achieved using this bidirectional transformer based model. the data set is highly skewed, hence suitable techniques for addressing the multiclass imbalance will be explored in future. bharathi raja chakravarthi, vigneshwaran muralidaran, ruba priyadharshini, and john philip mc-crae. 2020a. corpus creation for sentiment analysis in code-mixed tamil-english text. in proceedings of the 1st joint workshop on spoken language technologies for under-resourced languages (sltu) and collaboration and computing for under-resourced languages (ccurl), pages 202-210, marseille, france. european language resources association. bharathi raja chakravarthi, ruba priyadharshini, shubhanker banerjee, richard saldhana, john philip mccrae, anand kumar m, parameswari krishnamurthy, and melvin johnson. 2021a. findings of the shared task on machine translation in dravidian languages. in proceedings of the first workshop on speech and language technologies for dravidian languages. association for computational linguistics. bharathi raja chakravarthi, ruba priyadharshini, navya jose, anand kumar m, thomas mandl, prasanna kumar kumaresan, rahul ponnusamy, hariharan v, elizabeth sherly, and john philip mc-crae. 2021b. findings of the shared task on offensive language identification in tamil, malayalam, and kannada. in proceedings of the first workshop on speech and language technologies for dravidian languages. association for computational linguistics.