Hate and Offensive Keyword Extraction from C ode M ix M alayalam Social Media Text Using Contextual Embedding (2023)

Abstract: This paper focuses on identifying hate and offensive keywords from codemix Malayalam social media text. As part of this work, a dataset for hate and offensive keyword extraction for codemix Malayalam language was created. Two different methods were experimented to extract Hate and Offensive language (HOL) keywords from social media text. In the first method, intrinsic evaluation was performed on the dataset to identify the hate and offensive keywords. Three different approaches namely – unigram approach, bigram approach and trigram approach were performed to extract the HOL keywords, sequence of HOL words and the sequence that contribute HOL meaning even in the absence of a HOL word. Five different transformer models were used in each of the pproaches for extracting the embeddings for the ngrams. Later, HOL keywords were extracted based on the similarity score obtained using the cosine similarity. Out of the five transformer models, the best results were obtained with multilingual BERT. In the second method, multilingual BERT transformer model was fine tuned with the dataset to develop a HOL keyword tagger model. This work is a new beginning for HOL keyword identification in Dravidian language – Malayalam.

## introduction social networking sites are the platforms where users can create their own profiles and communicate with other users regardless of any kind of limitations. the freedom to share any content on social media led to the rise of hate and offensive posts on online social media (osn) @xcite @xcite . hate and offensive posts pose a severe risk to victims' physical and mental health and lead to serious consequences @xcite . this emphasis the importance of automatically detecting hate and offensive content from social media @xcite , @xcite . the identification of words which make the text hate or offensive is even more critical because it helps to restrict users from posting as well as reading comments containing such words. therefore, the automatic extraction of the keywords from a social media post has the equal significance of detecting hate content from a social media post. hol keyword extraction models are available in some languages. however, such models are not yet implemented in dravidian languages like malayalam, tamil, kannada etc. this task is challenging in dravidian languages because dravidian languages are abundant in morphology and can generate numerous word forms by joining a sequence of morphemes to the root word @xcite . besides, the social media posts are codemixed and low-resource for dravidian languages, which poses other challenges in developing an automatic keyword extraction model. despite of the challenges, developing a hol keyword extraction model for dravidian languages is necessary due to its increased use in social media. developing a model on codemix data is really challenging. the unavailability of an annotated dataset for hol keyword extraction on codemix data was the other main challenge. therefore, we developed an annotated dataset where all the hol words are labelled in each social media text. further, we prepared a dictionary of hate and offensive words. thus, through this work, we addressed the main challenge which hindered any research in hol keyword extraction in malayalam by developing the hol keyword extraction dataset. later, we performed an intrinsic evaluation on the dataset using five different multilingual sentence transform-ers. this paper investigates the efficacy of different multilingual transformer-based embedding models for automatically extracting the keywords from malayalam codemixed social media posts. we experimented three approaches namely, unigram approach, bigram approach and trigram approach for this. unigram approach was meant for extracting hol keywords. in addition, we used the transformer-based models to identify the multiword expressions that make a sentence which does not have a hate or offensive word, hate or offensive text. here, we define the multiword expression as a sequence of two words (bi-gram) or three words (tri-gram). we considered the intrinsic evaluation scheme in all these approaches for detecting the keywords and multiword expressions from a social media comment. likewise we developed a transformer based model @xcite and performed various analysis to evaluate the efficacy of our model. the major contributions of this paper are: ## related works hate and offensive content is a pervasive and developing social trend because of the surge of social media technology usage. there are many works related to hate and offensive language identification. in @xcite , the work was on identifying the offensive language in the low-resourced code-mixed dravidian languages -tamil, kannada, and malayalam. they constructed a dataset by transliterating all the code-mixed texts into the respective dravidian language and then pseudo labels were generated for it. they used different pretrained language for extracting the embeddings and then it was given to recurrent neural networks. the best results were obtained when they used the ulm fit model. their model gave an f1 score of 0.79 for tamil-english, 0.96 for malayalam-english and 0.73 for kannada-english. in @xcite , the authors developed three deep neural architectures for offensive language identification in dravidian languages. the first architecture was a hybrid model including a convolutional layer, a bi-lstm layer and a hidden layer. the second architecture contains a bi-lstm and the third architecture contains a bi-rnn. they mainly focused on the code-mixed tamil-english, malayalam-english and kannada-english for their work. on evaluation, the hybrid model gave them the best results with an f1 score of 0.64 for tamil-english, 0.90 for malayalam-english and 0.65 for kannada-english. various approaches were used by different works for identifying hate and offensive language identification. however, we couldn't find any works on hate or offensive keyword extraction from dravidian languages. lack of hol keyword annotated dataset is one of the main reason behind it. on the contrary, works on hate and offensive keyword extraction exists for languages except dravidian. in (sarracén and rosso, 2023), the work was on extracting offensive keyword from english comments. offenseval 2019 and offenseval 2020 were the datasets used for this work. the offensive keyword extraction was done based on the attention mechanism of bert and the eigenvector centrality using a graph representation. on testing, they obtained an f1 score of 0.5687 on off20-off19 and 0.5798 on off19-off20. in @xcite , the authors investigated the role of swear words in detecting the abusive language. they proposed the guidelines for tagging the hol keywords. they developed a swear word abusive dataset for english language using twitter comments. they also performed certain intrinsic evaluations such as sequence labelling on their dataset. they obtained an f1 score of 0.75 for non-abusive swear word, 0.42 for abusive swear word and 0.99 for not a swear word. in @xcite , the authors proposed transformer-based neural tagger for keyword identification (tnt-kid) to extract one or multiword phrase which represents the key aspects of a document. for this task, they collected a dataset of scientific abstracts and extracted keywords. according to their work, keyword tagging task was modeled as a binary classification task and predict if a word in the sequence is a keyword or not. the model was trained and tested accordingly and they obtained an f1-score of 0.63. though the hol keyword extraction was put forward by few works, it is not yet implemented in dravidian languages. this was the major research gap that motivated us for our work. it is necessary to develop a hol keyword extraction model for dravidian languages, because, it is widely spoken in south india and commonly used in social media for posting comments. hence our work focuses on implementing hol keyword extraction in dravidian language. as mentioned earlier, lack of dataset was the main challenge for this. therefore, we created a hol keyword dataset for code-mixed malayalam language. thus we tackled the prime cause for the research gap. our work is a new beginning for hol keyword extracton in dravidian language. ## dataset since dataset for hol keyword extraction were not existing for malayalam language, creation of the dataset was our prime motive. tweets and youtube comments were the sources of data. we extended the existing 'hasco' dataset @xcite to create our new hol keyword dataset. the hasoc dataset comprises of code-mixed malayalam comments labelled as 'hate' or 'not hate'. we focused on the negative comments (labelled as 'hate') for finding the hol keywords. the dataset consisted of 8943 comments. out of that 3092 comments were of hol nature and remaining 5851comments were normal. on analysing the negative comments in perspective of hol keywords, we could notice two types of negative comments. we categorised the comments into two. the first category consisted of negative comments with a hol word(s). the second category contained negative comments which did not have a hol word in it. keyword annotations process then was narrowed down to first category. the dataset creation (annotation) steps are illustrated in fig. ??. the first step was to search for hol keywords from the comments belonging to category one. for the ease of identifying negative words, we created a custom list of hol keywords from swear word website.the hol words in each comment was then identified by referring to this custom list. in order to label the identified hol keywords, we followed the guidelines proposed by @xcite . according to this, each offensive/hate keyword was tagged @xmath0. if any comments contain multiple negative words, that comments can be replicated to mark those words. we tagged the keywords accordingly to create the final annotated dataset. the final list of hol list contained 1082 words in it. the test set used in this work comprised of 756 comments. ## methodology we followed two methodologies for extracting hate and offensive keyword from codemix malayalam social media text. the first method involves an intrinsic evaluation whereas the second method follows a transformer based approach. ## experiments and results in each approach of method-1, the comments were tokenized to form ngrams. five different sentence transformers were used to generate the embedding of the ngram. finally, cosine similarity was employed for finding the hol keyword. the models were trained with 3425 hol comments and 6174 normal comments. later, they were tested with 84 hol comments. as this is a task where hol words are detected, the no.of comments in the test set might not create any bias. the 84 test cases were prepared meticulously. the test cases included only those comments which contained hol words that weren't listed on the custom list. this was done to know how efficiently the model could recognize unseen hol words. the performance of five different sentence transformers in extracting hol words are compared in this section. ## performance of method-2 this section presents the results of method-2. the train set was divided for training and validation in the ratio 8:2. the model was trained and validated for 30 epochs using adam and rmsprop optimizers with learning rate 3e-5. later the model was tested on the test set comprising of 84 comments. the performance of the model on the comments typed in english, comments typed in malayalam and comments including both malayalam and english were evaluated. ## discussion on analysing the dataset and results obtained in method-1, it was seen that in most of the cases the hate words had a similarity @xmath1.7. on analysing the performance of transformer models, we could see that the best results were obtained with multilingual bert. multilingual bert is trained on a large corpus and it has a huge number.of parameters when compared with the other models. this can be one of the reasons behind the excellent performance of the model in all the three categories of comments (typed in english alone, malayalam alone and including both). multilingual bert is followed by multilingual distilbert whose performance is a bit lower in the category of comments including both english and malayalam. multilingual distilbert is a distilled version of multilingual bert with lower number of parameters. this can be the cause for lower performance of multilingual distilbert when compared with the multilingual bert. the indic-abusive-allinone-muril model has a lower performance in the category of comments typed in malayalam alone when compared with the other two categories. but we can see that indicabusive-allinone-muril and multilingual distilbert showed a similar pattern of fair performance and it is better than indic bert. the indic-bert gave a medium performace in all the three categories of comments, but lower than the above 3 models. it can be due to the fewer number of parameters in this model. and finally, the malayalam-codemixed-abusive-muril, gave a lower-than expected result. even though this model is trained on malayalam codemix abusive language, the performance was not as expected. on comparing bigram and trigram approaches, bigram approach yields a better result than the latter. based on the results obtained in method-2, we could see that (the best performance of the model was evident on english based comments. comparatively good performance was seen on malayalam comments and english-malayalam comments.) though we evaluated the performance of the model with various optimizers, adam optimizer was performing best on our dataset. ## conclusion and future work in order to fulfil the gap of lack of annotated data for hol keywords in malayalam, a dataset was created for the same as part of this paper. later, the hate and offensive keywords in the dataset were identified based on cosine similarity using unigram approach. apart from this, hate and offensive sequences were extracted from the sentences even in the absence of a hate word using bigram and trigram approach. on comparing the performance of various sentence transformer models, "bert-base-multilingual-cased" turned out to be the best model for extracting hate keywords from code-mix malayalam social media text. being the best model, the "bert-base-multilingual-cased" was utilized for developing the transformer model in the second method. based on the results obtained in method-2, the best performance of the model was evident on english based comments. comparatively good performance can be seen on malayalam comments and english-malayalam comments. as a future work, the explainability concept @xcite can be employed to improve the performance of the current model. also, the performance of indic-abusive-allinone-muril and malayalam-codemixed-abusive-muril models can be further investigated. likewise, the effect of different dialects can be analysed to know its role in identifying hol keywords.