A Lexicon for Profane and Obscene Text Identification in B engali (2021)

Abstract: Bengali is a low-resource language that lacks tools and resources for profane and obscene textual content detection. Until now, no lexicon exists for detecting obscenity in Bengali social media text. This study introduces a Bengali obscene lexicon consisting of over 200 Bengali terms, which can be considered filthy, slang, profane or obscene. A semi-automatic methodology is presented for developing the profane lexicon that leverages an obscene corpus, word embedding, and part-of-speech (POS) taggers. The developed lexicon achieves coverage of around 0.85 for obscene and profane content detection in an evaluation dataset. The experimental results imply that the developed lexicon is effective at identifying obscenity in Bengali social media content.

## introduction the popularity of e-commerce and social media has surged the availability of user-generated content online. therefore, text analysis tasks such as sentiment classification @xcite @xcite @xcite , hate speech detection @xcite , profane or abusive content identification @xcite have received significant attention in recent years. profanity indicates the usage of taboo or swearing words and is prevalent in social media data across languages @xcite . the presence of swearing, obscene or vulgar words could be linked with hate speech, sexism, and racism. hence, identifying their presence is important to understanding and monitoring online content. although the terms profanity, obscenity, swearing, and vulgarity have subtle differences in their meaning, they are closely connected with some overlapping definitions. thus, in this paper, they have been used interchangeably to refer to filthy content. a lexicon consisting of a list of words with specific annotations can play an important role in various natural language processing tasks, such as sentiment analysis or inappropriate content identification. a profane or obscene lexicon contains words that convey foul, filthy, and profane meanings (e.g., ass, bitch). an obscene lexicon is instrumental for determining profanity, vulgarity, or obscenity in a text. the presence of swearing in english social media has been investigated by various researchers @xcite . @xcite found that the rate of swear word use in english twitter is 1.15%, almost double compared to its use in daily conversation (0.5% -0.7%) as observed in previous work @xcite . @xcite also reported that in a random sampling, they observed around 7.73% tweets containing swear words. furthermore, vulgar word identification can help to improve sentiment classification as shown by various researchers @xcite . in bengali, although few works performed abusive content analysis, none of them focused on determining obscenity or generating resources for identifying obscenity. until now, no lexicon exists in bengali that can help to identify profanity in text data. thus, in this work, the goal is to generate resources for obscenity identification. to construct the bengali obscene lexicon, we propose a corpus-based semi-automatic approach. from an existing bengali obscene corpus, utilizing word embedding and pos tagging, the lexicon is created. to demonstrate the efficacy of this lexicon, we categorize a drama review corpus into profane and non-profane categories based on the presence of swear and obscene terms. we observe that the developed lexicon successfully identifies 85.5% of the obscene or profane reviews in the corpus. ## related work the existence and socio-linguistics characteristics of swearing or cursing in social media have been studied in several studies. @xcite investigated the ubiquity, utility, and contextual dependency of swearing on twitter. @xcite analyzed several sociolinguistic aspects of swearing on twitter text data. several studies investigated the relationship between social factors, such as gender with the profanity, and discovered males employ profanity much more often than females @xcite . other social factors such as age, religiosity, or social status were found to be related to the rate of using vulgar words @xcite . @xcite noticed that the offensiveness of taboo words depends on their context, and found that usages of taboo words in conversational context is less offensive than hostile context. @xcite classified the use of swear words into five categories: dysphemistic; abusive, using taboo words to abuse or insult someone; idiomatic, using taboo words to arouse the interest of listeners without really referring to the matter; emphatic, to emphasize another word; cathartic, the use of swear words as a response to stress or pain. obscenity and profanity filtering has been studied for content filtering, such as parental controls @xcite , cyberbullying detectors @xcite . a more complex application of obscenity filtering is identifying implicitly abusive content, where both the intention of the author and the usage of obscene language need to be considered @xcite . research related to the identification of swearing or offensive words has been conducted mainly in english; therefore, lexicons comprised of offensive words are available in the english language. pamungkas et al. (2020) created swad (swear words abusiveness dataset), a twitter english corpus, where abusive swearing is manually annotated at the word level. their collection consists of 1,511 unique swear words from 1,320 tweets. @xcite manually collected approximately 2,700 dictionary entries, including phrases and multi-word expressions, which is one of the earliest work offensive lexicon creations. the recent work of lexicon creation for hate speech detection was reported in @xcite . another english lexicon of abusive words was provided by @xcite . @xcite explored the vulgar and obscene text in german. they conceived vulgar language is predominantly signaled by an overly lowered language, disgusting and obscene lexicalizations, which is generally banned from any type of civilized discourse. primarily, it refers to sexual organs and activities, as well as body parts and scatologic expressions. in bengali, several works investigated the presence of abusive language in social media data by employing supervised ml classifiers and labeled data @xcite . @xcite utilized linear support vector classifier (linearsvc), logistic regression (lr), multinomial naïve bayes (mnb), random forest (rf), artificial neural network (ann), recurrent neural network (rnn) with long short term memory (lstm) to detect multi-type abusive bengali text. they found rnn outperformed other classifiers by obtaining the highest accuracy of 82.20%. chakraborty and seddiqui (2019) employed ma-chine learning and natural language processing techniques to build an automatic system for detecting abusive comments in bengali. as input, they used unicode emoticons and unicode bengali characters. they applied mnb, svm, convolutional neural network (cnn) with lstm and found svm performed best with 78% accuracy. sazzed (2020b) created a sentiment lexicon that consists of over 500 bengali negative opinion words. however, no annotations regarding obscenity or vulgarity were provided for these negative words. @xcite proposed bengfasttext, a word embedding model for bengali, and incorporated it into a multichannel convolutional-lstm (mconv-lstm) network for predicting different types of hate speech. they compared bengfast-text against the word2vec and glove embedding by integrating them into several ml classifiers and showed the efficacy of bengfasttext for hate speech detection. sazzed (2021a) introduced an annotated bengali corpus of 3000 transliterated bengali comments categorized into two classes, abusive and non-abusive, 1500 comments for each. for the baseline evaluations, the author employed several supervised machine learning (ml) and deep learning-based classifiers. they observed support vector machine (svm) shows the highest efficacy for identifying abusive content. to the best of the author's knowledge, none of these existing works concentrated on recognizing obscene words in bengali social media content. besides, no lexicon exists so that profanity or obscenity can be determined without using any annotated data. this work is the first effort to identify profanity in the context of bengali social media data by introducing a obscene lexicon. ## corpora two bengali obscene datasets are used in this study, one for constructing the lexicon (development corpus), and the other one is for evaluating the performance of the created lexicon (evaluation corpus). ## creation of obscene lexicon part-of-speech (pos) tagger is a text analysis tool that assigns a pos tag (e.g., noun, verb, adjective, etc.) to each word of a piece of text data. as adjectives, nouns, and verbs usually convey opinions, the pos tagger can help to identify words that convey obscenity. some of the popular pos taggers in english are nltk pos tagger @xcite , spacy pos tagger @xcite , etc. we utilize a bengali pos tagger to identify opinion conveying word (i.e., adjective and verb) foot_3 . however, the existing bengali pos taggers are not as accurate as of its english counterpart. hence, manual validation is needed to check the correctness of the pos tags assigned to words. ## obscenity detection in textual content as no lexicon exists in bengali for obscene content detection, we compare the performance of the developed lexicon with several supervised classifiers in the evaluation corpus. ## discussion the results reveal that the developed lexicon is capable of identifying obscene content in bengali social media. it shows higher document-level coverage than in-domain labeled data in original classimbalanced settings (i.e., when the dataset contains mostly non-obscene comments). however, in a class-balanced dataset, we find ml classifiers and the sgd optimizer perform better than the developed lexicon. labeled data are scarcely available in lowresource languages such as bengali; therefore, the developed lexicon can be a practical resource for obscenity identification when labeled data are unavailable. besides, as shown in ## summary and conclusion this study presents a semi-automatic methodology for creating a lexical resource (i.e., an obscene lexicon) to detect obscene content in bengali. an obscene corpus and various text processing tools and resources are leveraged to develop the obscene lexicon. the developed lexicon is made publicly available foot_4 . the proposed methodologies can be adapted to other resource-limited languages to create lexical resources. the efficacy of the obscene lexicon suggests that it can be utilized to distinguish obscenity in bengali social media content when annotated data are unavailable. however, it should be noted that labeling textual content as obscene or vulgar entirely based on obscene or swear words may not be sufficient due to the complexity of the natural languages. still, a well-annotated lexicon of moderate size can assist in identifying profane content; especially, in resource-scarce language. our future work will involve expanding the size of the lexicon by utilizing larger and multi-domain development corpora.