A Dataset for Detecting Humor in T elugu Social Media Text (2022)

Abstract: Increased use of online social media sites has given rise to tremendous amounts of user generated data. Social media sites have become a platform where users express and voice their opinions in a real-time environment. Social media sites such as Twitter limit the number of characters used to express a thought in a tweet, leading to increased use of creative, humorous and confusing language in order to convey the message. Due to this, automatic humor detection has become a difficult task, especially for low-resource languages such as the Dravidian languages. Humor detection has been a well studied area for resource rich languages due to the availability of rich and accurate data. In this paper, we have attempted to solve this issue by working on low-resource languages, such as, Telugu, a Dravidian language, by collecting and annotating Telugu tweets and performing automatic humor detection on the collected data. We experimented on the corpus using various transformer models such as Multilingual BERT, Multilingual DistillBERT and XLM-RoBERTa to establish a baseline classification system. We concluded that XLM-RoBERTa was the best-performing model and it achieved an F1-score of 0.82 with 81.5% accuracy.

## introduction the use of social media sites has increased exponentially over the decade giving rise to vast amount of user generated content. social media sites offer the ability to reach large number of users in real time which enable users to share their experiences easily. the content usually consists of creative and figurative use of languages such as humor, insults, sarcasm and irony. in the past couple of years, research in these linguistic elements has increased tremendously due to requirements in academia as well as in organizations. natural language processing(nlp) has evolved significantly leading to improvements in most of the fundamental tasks like named-entity recognition, sentiment analysis, etc @xcite . while the advancement is not only attributed to improvements in architecture of models but also due to the increased availability of data. plethora of work exists for resource rich languages such as english @xcite @xcite . however, the same cannot be said for low-resource languages originating from the indian subcontinent such as the dravidian languages. telugu is one of the four major dravidian languages that stem from india, it is spoken by more than 75 million people @xcite . hence, it is vital to establish a baseline system for automatic humor detection in telugu language. in this paper, we explore the task of humor detection, one of the critical elements of a natural language @xcite . humor is subtle and yet plays a significant part in our linguistic and social lives @xcite . the primary challenge in working with humor detection is the subjective nature of humor and capturing it in higher order is a challenge in nlp (deoliveira and rodrigo, 2015). yet, a large amount of research work has been carried out on english tweets and has achieved significant results. one of the major challenges in building any social media analysis model in low-resource languages is the unavailability of high quality annotated data for conducting various experiments. in this paper, we address this issue by collecting telugu tweets data by scraping twitter and performing annotations on it. furthermore, the dataset we collected is publicly available 1 . below is an example of humorous tweet in telugu: నవువ్ రాకపోయినా నవేవ్వాలని ఏమంటారో కానీ ? what do you call those people who laugh even when laughter is not induced. context: this tweet intends to mock the judges of a telugu comedy show who laugh a lot. additionally, we trained three multilingual transformer-based models, namely: multilingual bert, multilingual distilbert, and xlm-roberta and compared their performances to establish baseline classification system for humor detection in telugu. the rest of the paper is organized into related works (section 2), detailed description of the dataset (section 3), brief description of the methodology used(section 4), analysis of results (section 5), and finally conclusion (section 6). ## related works plenty of work exists on social media text analysis including humor detection. various works related to humor detection exist in english language, such as statistical and n-gram analysis @xcite , regression trees @xcite , word2vec combined with k-nn human centric features @xcite , convolutional neural networks @xcite and transformer models @xcite . previous work related to humor detection in hindi-english mixed language dataset consists of scraping hindi-english tweets and building n-grams, bag-of-words, lstm, bi-directional lstm, and attention based bi-directional lstm @xcite @xcite . related works for humor detection in telugu language is scarce. vaishnavi et. al @xcite proposed conversational data in telugu language for humor detection and experimented with textgcn, fasttext, multilingual bert, muril, indic-bert, and multilingual dis-tilbert models. automatic humor detection in telugu for twitter data is an under-explored area. according to our knowledge, this paper is the first attempt at building a novel telugu dataset and then proceeding with experimenting on various classifiers for humor detection task. ## modelling we have trained three transformer models on our corpus for humor detection task. transformer models with their improved architecture have achieved state-of-the-art results in numerous benchmark datasets including text classification tasks @xcite . hence, we decided to train these models to establish a suitable baseline for future work. ## results the corpus was split into 80% train and 20% test. we have downsampled the non-humorous tweets to match the number of humorous tweets. at the end, we have 732 training examples and 184 test examples. we have considered macro f1-score as our metric to select the best model out of the three models trained on our corpus. xlm-bert has outperformed other models with a f1-score of 0.82 and an accuracy of 81.5% as shown in model accuracy f1-score multilingual bert 81.5 0.81 multilingual distilbert 73.4 0.73 xlm-roberta 81.5 0.82 table2: results of various models trained and tested on our corpus. ## conclusion and future work in this paper, we introduced the telugu twitter humor dataset and have addressed the need to annotate low-resource languages and create datasets in languages such as telugu. we have also described our data collection and annotation process. additionally, we have trained multiple transformerbased models, namely, multilingual bert, multilingual distillbert, and xml-roberta, to perform automatic humor detection on our collected dataset. the performance of these models is compared and a baseline for classification of humorous and non-humorous telugu tweets is established in this paper. out of the above-mentioned models used for training our data, xlm-roberta performed the best with a f1-score of 0.82 and an accuracy of 81.5%. we would like to expand this work in the future by incorporating the information detected from emojis into the classifiers and by making a multi-modal humor detection classifier as a large number of tweets which were discarded had images present in them too.