Humor@ IITK at S em E val-2021 Task 7: Large Language Models for Quantifying Humor and Offensiveness (2021)

Abstract: Humor and Offense are highly subjective due to multiple word senses, cultural knowledge, and pragmatic competence. Hence, accurately detecting humorous and offensive texts has several compelling use cases in Recommendation Systems and Personalized Content Moderation. However, due to the lack of an extensive labeled dataset, most prior works in this domain haven’t explored large neural models for subjective humor understanding. This paper explores whether large neural models and their ensembles can capture the intricacies associated with humor/offense detection and rating. Our experiments on the SemEval-2021 Task 7: HaHackathon show that we can develop reasonable humor and offense detection systems with such models. Our models are ranked 3rd in subtask 1b and consistently ranked around the top 33% of the leaderboard for the remaining subtasks.

## introduction like most figurative languages, humor/offense pose interesting linguistic challenges to natural language processing due to its emphasis on multiple word senses, cultural knowledge, sarcasm, and pragmatic competence. a joke's perception is highly subjective, and age, gender, and socioeconomic status extensively influence it. prior humor detection/rating challenges treated humor as an objective concept. semeval 2021 task 7 @xcite is the first humor detection challenge that incorporates the subjectivity associated with humor and offense across different demographic groups. users from varied age groups and genders annotated the data with the text's humor and have provided an associated score for the same. it is also quite a generic phenomenon that a text might be humorous to one and normal/offensive to another. rarely has it been noticed that the same content is globally accepted as witty. to the best of our knowledge, @xcite is the first initiative towards annotating the underlying humor as controversial or not. understanding whether a text is humorous and/or offensive will aid downstream tasks, such as personalized content moderation, recommendation systems, and flagging offensive content. large language models (llms) have recently emerged as the sota for various natural language understanding tasks @xcite @xcite . however, typical day-to-day texts, where these models have shown state of the art performance, are less ambiguous than texts having puns/jokes. training and evaluating llms in the context of highly ambiguous/subjective english texts would serve as an excellent benchmark to figure out the current shortcomings of these models. this paper studies various large language models -bert @xcite , roberta @xcite , xlnet @xcite , ernie-2.0 @xcite and deberta @xcite and their ensembles -for humor and offense detection tasks. additionally, we explore a multi-task learning framework to train on all the four sub-tasks jointly and observe that joint training improves the performance in regression tasks. we have achieved significant performance on all the subtasks and have consistently ranked ∼ foot_0 3 rd of the total submissions. we were ranked (1) 21 st with an f-score and accuracy of 94.8% and 95.81% respectively in task 1a, (2) 3 rd with an rmse score of 0.521 in task 1b, (3) 9 th with an f-score and accuracy of 45.2% and 62.09% respectively in task 1c; and (4) 16 th with an rmse score of 0.4607 in task 2. we release the code for models and experiments via github 1 we organize the rest of the paper as: we begin with a description of the challenge tasks followed by a brief literature survey in section 2. we then describe all of our proposed models in section 3 with training details in section 4 and present the experimental results in section 5. finally, we analyze our findings and conclude in section 6, and 7 respectively. ## experimental setup we used pytorch @xcite and hug-gingface @xcite ) library for our models, and google colab gpus for training and inference. we use adamw @xcite and adam (kingma and ba, 2017) optimizer with initial learning rate of 2e -5 for training single task and multi task models respectively. for each of the models we follow a dedicated training pipeline described in subsequent sections. ## results we have trained multiple single task and multi task models using basenet llms like bert, distil-bert, roberta, xlnet, albert @xcite , electra @xcite , deberta, and ernie-2.0. we also learned ensembles of single task models by either training a classification/regression head on concatenated input embeddings or using weighted aggregate of the models' predictions. apart from this, we also explored voting based ensemble of multi-task models. all our models perform comparably on all tasks and the major models are reported in ## analysis 6.1 data augmentation one recurring issue across all our trained models is the high susceptibility to overfitting. data augmentation is a widely accepted solution to reduce overfitting by generating slight variants of the given dataset and is extremely useful for a smaller dataset. one such approach is masked language modelling (mlm), used to perform context-specific data augmentation @xcite and has been used in training llms. however, following this data augmentation during training has consistently degraded the performance of our models. we hypothesize that this is due to the mismatch be-tween the contextual meaning and the associated humor/offense. mlm-based augmentation strategies, with models pre-trained to preserve the sentence's meaning, fail to capture the associated humor/offense. often the selection of words in a sentence is responsible for its humor/offensive rating. replacing such words by their synonyms can change the humor/offense rating substantially. hence, using such a data augmentation approach during training will inject heavy noise in the ground truth resulting in deteriorated performance. ## conclusion we have presented several experiments using large language models like bert, xlnet, etc., and their ensembles for humor and offense detection and rating. we also discuss some of the underlying challenges due to the subjective nature of humor and offense detection task. using these, we explain why standard training practices used to prevent overfitting, like data augmentation, do not work in this context. our experiments suggest that even though these models can reasonably capture humor and offense, they are still far from understanding every intricacy arising out of subjectivity. to tackle some of the problems highlighted in this paper, a compelling direction would be online data augmentation by alternating between training the embeddings and generating new texts to preserve the humor/offensiveness. additionally, pretraining these models on datasets annotated by diverse annotators to capture a more comprehensive world knowledge should further help in generalization.