D uluth at S em E val-2020 Task 7: Using Surprise as a Key to Unlock Humorous Headlines (2020)

Abstract: We use pretrained transformer-based language models in SemEval-2020 Task 7: Assessing the Funniness of Edited News Headlines. Inspired by the incongruity theory of humor, we use a contrastive approach to capture the surprise in the edited headlines. In the official evaluation, our system gets 0.531 RMSE in Subtask 1, 11th among 49 submissions. In Subtask 2, our system gets 0.632 accuracy, 9th among 32 submissions.

## introduction humor detection is a challenging problem in natural language processing. semeval-2020 task 7 @xcite 1 focuses on detecting humor in english news headlines with micro-edits. specifically, the edited headlines have one selected word or entity that is replaced by editors, which are then graded by the degree of funniness. accurate scoring of the funniness from micro-edits can serve as a footstone of humorous text generation @xcite . inspired by the incongruity theory @xcite , we believe that contrast and surprise is a key ingredient of humor. we instantiate this intuition with a contrastive framework. we then systematically compare three widely used models: cbow, bert @xcite , and roberta @xcite , providing a benchmark for this task. our best system, based on roberta, achieves compelling performance for both subtasks. our code is available on github. 2 ## related work early humor recognition systems are mostly based on traditional machine learning methods, such as support vector machine, decision tree, naive bayes, and k-nearest neighbors @xcite . besides, an n-gram language model shows good performance @xcite in learning a sense of humor from tweets. yet n-gram models are limited to a small number of context words. pretrained language models based on transformer @xcite can obtain contextual information of a whole sentence. among this family, bert has been used to assess the humor in tweets and jokes @xcite . enlightened by these recent advances, we use bert to judge the funniness of edited news headlines. we additionally experiment with roberta, a robustly optimized variant of bert. lastly, several works also attempt to explicitly model incongruity and surprise of humourous text, focusing on homophonic puns. @xcite formalizes incongruity as a mixed effect of ambiguity and distinctiveness, quantified by entropy and kullback-leibler divergence. @xcite proposes a local-global surprisal measure based on the log-likelihood ratio, to assess whether a sentence is a pun. however, we focus on a broader definition of humor, and formulate incongruity as an input pair to a dual encoder framework. task id original headline edit score label subtask 1 33210 california and president trump are going to war with each other monkeys 1.8 / 1664 what if sociologist had as much influence as economists? donkeys 2.8 / subtask 2 9934 chibok girls reunited with families smartphones 1.8 1 14279 chibok girls reunited with families cats 0.0 10920 gene cernan, last astronaut on the moon, dies at 82 dancer 1.2 1 9866 gene cernan, last astronaut on the moon, dies at 82 impregnated 0.8 the humicroedit dataset @xcite provides the training, development, and test data for this task. we also use additional training data from the funlines dataset @xcite . the dataset statistics are summarized in appendix a. in subtask 1, the goal is predicting the funniness score of an edited headline. the score z ranges from 0 to 3, where 0 means not funny and 3 means very funny. in subtask 2, the goal is to predict the funnier between an edited sentence pair. for labels @xmath0, 1, 2}, 0 implies two headlines are equally funny, 1 implies the first is the funnier, and 2 implies the second is the funnier. examples of the two subtasks are in ## methods what are the important characteristics of humor? the incongruity theory, a dominant theory of humor, states that "it is the perception of something incongruous-something that violates our mental patterns and expectations" @xcite . therefore, we hypothesize an edited headline is funny if the edit words are semantically distant from the context words or the original words. this can be exemplified by the first two examples in similarly for headline 1664, given the context what if [mask] had as much influence as economists, humans might fill in occupation-related words like "scientist" or "sociologist" (as in the original headline). however, the edit word is "donkeys", which is a surprising prediction and is considered very funny (scored 2.8 out of 3). in this section, we describe a concrete architecture that models the strength of contrast and surprise, which translates into the funniness score.