Amherst685 at S em E val-2021 Task 7: Joint Modeling of Classification and Regression for Humor and Offense (2021)

Abstract: This paper describes our submission to theSemEval’21: Task 7- HaHackathon: Detecting and Rating Humor and Offense. In this challenge, we explore intermediate finetuning, backtranslation augmentation, multitask learning, and ensembling of different language models. Curiously, intermediate finetuning and backtranslation do not improve performance, while multitask learning and ensembling do improve performance. We explore why intermediate finetuning and backtranslation do not provide the same benefit as other natural language processing tasks and offer insight into the errors that our model makes. Our best performing system ranks 7th on Task 1bwith an RMSE of 0.5339

## introduction with the advancement in deep learning methods, nlp tasks like sentiment analysis and opinion mining have achieved high accuracy, however detection of some salient forms of figurative language such as humor remain difficult tasks. being able to infer humor and offense with a high accuracy can help improve and lead to better performance on downstream applications, such as content moderation, sentiment analysis, etc. this would be useful for various downstream applications, such as understanding tweets, reviews and feedback. however, humor detection is not trivial. what makes identifying humor hard? humor can consist of styles ranging from sarcastic to slapstick comedy, and it factors in both individual preferences and underlying cultures. context, sounds, and vision or any combination of these can be key in building to a punchline @xcite . humor appreciation is also highly subjective, as age, gender, and socio-economic status often impact the perception of a joke. @xcite identify 3 ways that classification of humor is difficult: our contributions are as follows: (1) we explore whether intermediate finetuning on other humor and offense datasets is helpful for this task, (2) we seek to identify if backtranslation augmentation is useful for humor detection, (3) we show that multitask learning is helpful when classifying and scoring both humor and offense, and (4) we find that ensembling different language models leads to improved results on some tasks. the code for our experiments is available at our github repository foot_0 . ## related work the challenge of humor detection has gained traction since 2017. @xcite , explains in their proposal that prior work has explored humor detection as an objective task, averaging all annotations for a joke, to produce a single classification or rating. this treats humor as an objective concept, which is not the case. this motivated their challenge for semeval'21 @xcite to explore these dimensions of humor and offense. we discuss some of the previous efforts to explore humor and offensiveness in the following paragraphs. @xcite explains how text in reviews is quite complex as they can be sarcastic, humorous, or hateful. an ordinary sentiment analysis would fail to perform well in such cases. they first extract features pertaining to sarcasm, humor, hate speech, and sentiment, and then use these features to inform sentiment classification. their work is quite sensitive to catching negative sentiment, however, it does not do as well when sentiment changes halfway through the text. it also does not address the subjectivity of humor. colbert (annamoradnejad, 2020) is among the first to use bert @xcite for humor detection, reaching 98% classification accuracy and outperforming variants using recurrent neural networks and convolutional neural networks. @xcite is another work that uses bert to classify if a tweet is a joke or not and predict how humorous the tweet is. the work of @xcite explores extending humor detection capability by trying to assess whether or not a joke is humorous. they use transformers to identify humorous jokes based on ratings from reddit pages, reaching human-level performance. earlier work, like @xcite , use recurrent deep learning methods with dense embeddings to predict humorous tweets. in order to factor both meaning and sound in their analysis, they use glove embeddings combined with a novel phonetic representation as input to an lstm. hossain et al. (2020) hosted the semeval'20 event for humor detection in news headlines. the event challenged participants to classify whether an original headline or an altered headline is funnier and rate the funniness of the edited headline on a 0-3 humor scale. the winning teams @xcite combined the predictions of several models using sentence pair regression and ensembled the pre-trained language models bert , gpt-2, (radford et al., 2019) roberta, @xcite , xlnet @xcite , transformer-xl, and xlm @xcite to form the final prediction. similar to humor detection, there has been some work to explore offense in text. semeval '19 had a task @xcite , aimed at identifying and categorizing offensive language in social media. the top performing teams used ensembles of random forest, linear models, recurrent networks, and pretrained transformer language models. our work is motivated by the semeval challenges which encourage interesting techniques to handle multiple word senses, cultural knowledge, and pragmatic competence. through this challenge, we try to detect humor and explore the subjectivity of humor appreciation with a controversy score to examine the variance in humor ratings for each different text. ## system overview as a starting point, we use the huggingface transformers library 2 , along with their large collection of pretrained language models. from there, we finetune these transformers on the competition training dataset during development and the combined training and development datasets for the final evaluation phase. building on this basic paradigm, we experiment in four different ways with the goal of improving model performance: (1) using various datasets for intermediate finetuning, (2) using backtranslation to expand the competition datasets, (3) training multitask models to predict all labels simultaneously, and (4) ensembling predictions using different pretrained language models as starting points. ## error analysis one of the possible sources of confusion and bias in our model seemed to be centered around atypical punctuation such as question marks and exclamation marks. for example, when a question mark was placed in the middle of a sentence, the model often erroneously labels it humorous regardless of the actual content. when manually reviewing the data, we found that the vast majority of texts that contain a mid-text question mark are humorous due to their setup and punchline structure. without balancing with negative examples with similar structures, the model can become reliant on punctuation structure rather than the actual relationship between the words. another driver of error seems to be the actual source of the competition dataset. through further analysis, we found that the vast majority of the negative examples seemed to be sourced from tweets. this can be seen in the length distribution of the dataset; there is a sharp cutoff around 140 characters, which used to be the maximum length for a tweet. however none of the other datasets we found or compiled ourselves (for humor tasks) when pulling individual examples, we found that tweets tended to use more colloquial language, with a greater variety of punctuation, vocabulary, and capitalization when compared to news headlines. one final potential driver of error were song lyrics and quotes. there were a proportionally large number of movie, song, and tv show quotes used in the dataset, by a rough estimate based on sampling, approximately 5% of the example fell in one of those categories. our model was often able to differentiate between these quotes, though it was not something that was found in our own custom datasets. after performing this deep dive analysis on our results, and seeing the various areas of where our model got confused, we believe that the primary reason our models did worse with the inclusion of extra datasets was due to the source of our data. the wider range of punctuation, capitalization, and vocabulary expressed in twitter posts was not well captured by utilizing news headlines as a negative source, and thus likely allow our model to use syntax and punctuation structure as a substitute for the actual substance of the text. ## conclusion in this competition, we explored the use of intermediate finetuning, backtranslation augmentation, multitask learning, and ensembling of different pretrained models. unlike in many natural language processing tasks, intermediate finetuning on other related datasets provided no benefit in this task, perhaps because prior datasets used non-humorous texts that were much easier to identify. next, although backtranslation augmentation did not improve performance, this is still a noteworthy result because it indicates that humor is likely not preserved through paraphrasing. finally, multitask learning across humor and offense detection, as well as ensembling of different pretrained language models improved overall performance.