S em E val-2020 Task 7: Assessing Humor in Edited News Headlines (2020)

Abstract: This paper describes the SemEval-2020 shared task “Assessing Humor in Edited News Headlines.” The task’s dataset contains news headlines in which short edits were applied to make them funny, and the funniness of these edited headlines was rated using crowdsourcing. This task includes two subtasks, the first of which is to estimate the funniness of headlines on a humor scale in the interval 0-3. The second subtask is to predict, for a pair of edited versions of the same original headline, which is the funnier version. To date, this task is the most popular shared computational humor task, attracting 48 teams for the first subtask and 31 teams for the second.

## datasets the data foot_0 for this task foot_1 is the humicroedit dataset described in our previous work @xcite . this dataset contains about 5,000 original headlines, each having three modified, potentially funny versions for a total of 15,095 edited headlines. the original headlines were collected from reddit (reddit.com) via the popular subreddits r/worldnews and r/politics, where headlines from professional news sources are posted everyday. these headlines were published between 01/2017 and 05/2018, they are between 4-20 words long, and they are sampled from headlines written by 25 major english news sources. the data was annotated using workers from amazon mechanical turk, who were screened using a qualification phase to find expert headline editors and judges of humor. the editors were instructed to make a headline as funny as possible to a generic wide audience by applying a micro-edit, which is a replacement of a verb/noun/entity in the headline with a single word. examples are shown in five judges were asked to rate the funniness of each edited headline using the following humor scale: 0 -not funny 1 -slightly funny 2 -moderately funny 3 -funny the funniness of an edited headline is the mean of the ratings from its five judges. for further details and analysis of the dataset, we refer the reader to @xcite . for our task, we randomly sampled the humicroedit dataset into train (64%), dev (16%) and test (20%) sets such that all edited versions of an original headline reside in exactly one of these sets, as opposed to the sampling in @xcite which allowed overlap of original versions of headlines among its dataset partitions for a slightly different humorous headline classification task. we also provided additional training data 3 from funlines 4 @xcite , a competition that we hosted to collect humorous headlines at a very low cost. the data collection approach for humicroedit and funlines are mostly similar, but funlines additionally includes headlines from the news categories sports, entertainment and technology, and its headlines were published between 05/2019 and 01/2020, for a total of 8,248 annotated headlines. more than 40% of the participating teams, including the winning team, made use of the funlines data. ## task description the objective of this shared task is to build systems for rating a humorous effect that is caused by small changes in text. to this end, we focus on humor obtained by applying micro-edits to news headlines. editing headlines presents a unique opportunity for humor research since headlines convey substantial information using only a few words. this creates a rich background against which a micro-edit can lead to a humorous effect. with that data, a computational humor model can focus on the exact localized cause of the humorous effect in a short textual context. we split our task into two subtasks. the dataset statistics for these subtasks are shown in ## overview of participating systems the dominant teams made use of pre-trained language models (plm), namely bert, roberta, elmo (peters et al., 2018), gpt-2 @xcite and xlnet @xcite . context-independent word embeddings, such as word2vec @xcite , fasttext @xcite and glove word vectors @xcite , were also useful. the winning teams combined the predictions of several hyperparameter-tuned versions of these models using regression in an ensemble learner to arrive at the final prediction. next, we summarize the top systems and other notable approaches. first, we note that for subtask 2, most systems relied on the model they developed for subtask 1. this involved using the model to estimate a real number funniness rating for each of the two edited headlines, and selecting the one which achieved the higher estimated rating. as a result, there was a strong correlation between teams' placements in subtask 1 and subtask 2, with the top 3 teams in both tasks being the same. ## analysis and discussion the outputs of 48 participating systems for subtask 1 and 31 for subtask 2 present an opportunity to not only study individual solutions and numeric results, but to also take a deeper qualitative look at the output of these systems. here, we collectively analyze the performance of the top 20 systems per subtask to find aggregate trends that characterize the general approaches and the challenges of assessing humor itself. to better understand which funniness ranges are particularly hard for systems to assess, we study the performance of the systems as a function of ground truth funniness. as shown in figure 2 , we grouped the edited headlines into funniness bins of width 0.2. for each bin, we plotted the mean absolute regression errors for the top 20 systems aggregated (max @xmath0.547), the winning hitachi system (@xmath1.497), the 19 other systems and baseline (@xmath2.575). in general, all these systems have their minimum error at a funniness score of about 1.0. while the hitachi system stands out somewhat in its superior performance at the two extremes of the funniness scale, the other systems follow generally the same pattern, and none appear to be outliers. assessing more extreme humor (or lack thereof) appears to be harder since all the systems have larger errors toward the extremes of the funniness scale. this may also be due to the non-uniform distribution of ground truth funniness scores in the dataset (shown as the blue curve), with the extreme values being less frequent. figure 3 shows the systems' antipodal rmse, an auxiliary metric for subtask 1, which we calculated by considering only the x% most funny headlines and x% least funny headlines, for @xmath3, 20, 30, 40} in the rmse metric. the systems are ranked by their overall rmse for subtask 1. it appears that some of the systems further down the ranking are doing much better at estimating the funniness of the extremes in the dataset than their superiors. for example, the large dip shows the system ranked 41 (hahackathon) is performing better at estimating the funniness of the top 10-40% most/least funny headlines than several systems ranked before it. this sug-gests that combining these approaches can yield better results, for example, using some selected systems to rank certain subsets of headlines. ## conclusion and future perspectives we provided 15,095 edited and humor-rated, potentially funny headlines and defined subtasks for (1) rating the funniness of each one and (2) determining the funnier headline from a pair that came from editing the same original headline. both humor subtasks were popular, attracting 48 and 31 teams respectively, showing that shared tasks can unify the relatively smaller humor research community. for both subtasks, the highly rated solutions show that pre-trained language models work well for rating humor. for subtask 2, nearly all the participating teams used their solution from subtask 1 for ranking the two headlines. for subtask 2, we found that larger disparities in ground truth funniness made ranking easier and that incongruity in a headline was positively correlated with more accurate ranking of humor. for subtask 1, we discovered that, over the range of funniness scores, the top systems were most accurate at rating humor near the middle of the funniness range where we had the most training data. for future contests like this, we advocate for more uniformly labeled humor data, though that can be hard and expensive to collect. another direction worth pursuing is humor recognition in a closed setting such as reading comprehension, where both annotators and systems make judgments based only on a limited amount of provided contextual information. this would constrain the problem, setting a well-defined scope, and potentially lead to stronger annotator agreements. we also believe that focusing on specific labeled forms of humor, such as incongruity, sarcasm, irony, puns, and superiority would be advantageous. this could help to better understand how different modeling strategies can identify different root causes of humor. we would also want to design subtask 2 to be more independent of subtask 1 to encourage fresh approaches for subtask 2. finally, improving the common sense and world knowledge understanding capabilities of ai systems will be crucial for substantially improving the performance of computational humor systems. we hope that both the current results and the dataset in this task provide a stepping stone towards this goal.