Can Humor Prediction Datasets be used for Humor Generation? Humorous Headline Generation via Style Transfer (2020)

Abstract: Understanding and identifying humor has been increasingly popular, as seen by the number of datasets created to study humor. However, one area of humor research, humor generation, has remained a difficult task, with machine generated jokes failing to match human-created humor. As many humor prediction datasets claim to aid in generative tasks, we examine whether these claims are true. We focus our experiments on the most popular dataset, included in the 2020 SemEval’s Task 7, and teach our model to take normal text and “translate” it into humorous text. We evaluate our model compared to humorous human generated headlines, finding that our model is preferred equally in A/B testing with the human edited versions, a strong success for humor generation, and is preferred over an intelligent random baseline 72% of the time. We also show that our model is assumed to be human written comparable with that of the human edited headlines and is significantly better than random, indicating that this dataset does indeed provide potential for future humor generation systems.

## introduction understanding and identifying humor has long been a goal for natural language understanding systems @xcite @xcite , with many attempts seeking to identify whether a sentence is a joke. these systems have seen impressive gains in recent years @xcite , with systems achieving scores in the mid 90's. as such, other areas of humor research have grown in popularity, including distinguishing between jokes @xcite and generating humorous text @xcite . this rise in popularity has even translated to a semeval task of predicting the level of humor in text @xcite . in their work, as well as others @xcite ) that seek to understand various aspects of humor, the authors note that their work may be influential in helping create systems that can automatically generate humor. to the best of our knowledge, however, no work has attempted to explore whether these humor prediction datasets encode information that can be used by a generative system. instead current systems rely on retrieveand-edit models @xcite or models based on word senses @xcite . the recent works of @xcite ) have created pairs of minimal changes that turn a regular news sentence into a humorous news sentence, by only changing one phrase. because of its popularity and impact, as well as the clear insight that can be gained from minimal pair datasets @xcite , we choose to examine the former as an initial exploration of what can be done. our contributions include: in the humor generation area, previous approaches have relied strongly on templated approaches for specific types of puns or jokes @xcite @xcite , such as "i like my coffee like i like my [insert phrase here]." others have used templates based on word similarity or uncommonness to generate humorous samples @xcite . recent work has started to break off from the template trend, creating abstractive models such as an rnn that creates puns @xcite , a retrieve-and-edit model that adds surprise to create jokes @xcite , and a pun gan with a word-sense disambiguater as a discriminator @xcite . however, none of these models employ a humorous corpus to generate their jokes, leading to the question: are they useful for models attempting to generate humor? our work also utilizes methods from stylized text generation @xcite , where work has shown success with parallel data @xcite as well as dealing with the lack of such data @xcite . these methods, recently employing transformer models proposed by @xcite , have also been applied to formality of language (etinger and black, 2019) and sarcasm @xcite . however, to the best of our knowledge we are the first to use style transfer for humor generation. this dataset further assumes that the reader is aware of the original headline (from already popular news, for example), so that the additional word play in the edits will be humorous in light of the original headlines and topic (example instances are shown in original president trumps golden age of trolling edited president trumps infinite of trolling random president big spenders accentuation age of trolling translated president trumps golden age of sassy trolling original how cbs news reported the last national military parade in 1991 edited how cbs news reported the last national military buffet in 1991 random how ides hemophiliac reported the last national military parade in 1991 translated how cbs news choreographed the last national military parade in 1991 original trump lawyers scramble to prepare for new stage of russia probe edited trump lawyers scramble to prepare for new stage of dog probe random trump lawyers scramble to prepare for new bailey of russia probe translated trump lawyers scramble to prepare for new eggs of russia probe baseline. to show the effectiveness of our model on the data, we use a baseline that would generate similar surprisal effects in headline edits. we recognize the capacity of the human mind to make connections when there are none, thus, we want a baseline that randomly replaces words in a sentence and relies on that connective ability. however, a purely random model would be too naive, creating headlines that are ungrammatical and unintelligible. thus, we create an intelligent random model that probabilistically replaces specific parts of speech with other words in that same part of speech, capitalizing the replacement phrase if the original was capitalized. we randomly replace nouns, nounphrases, verbs, and adjectives, in order to replicate human edits in the original dataset. ## experiments we use the test set described in section 3, consisting of over 2k instances. in order to show a comprehensive view of our model, we compare the human and random edits with our translated samples. for convenience in writing, we term these models edited, random, and translated. we label the original non-humorous headlines, original. example instances of each humor system are displayed in ## results we see the results of the experiments in example 1: repeats original: couple who rented condo to pruitt pays fine to d.c. edit: couple who wore wizard to pruitt pays to d.c. d.c. d.c. example 2: not humorous original: china says to ban some petroleum exports to north korea edit: china says to ban some 2005 to north korea example 3: no change original: california to sue trump administration for repeal of fracking rules edit: california to sue trump administration for repeal of fracking rules we see instances in we see that despite the above mentioned limitations, our humor translation model matches the performance of human generated edits. as humor is a a linguistic phenomenon that depends upon the human receiving it to appreciate the humor, it is difficult to generate humor that is better than human generated content. however, our results indicate that the humicroedit dataset of pairs combined with our translation model is able to provide creative ways of reformulating headlines. as this work is exploratory and non-exhaustive, this gives a positive signal that the communities' efforts in humor collection have strong potential for further advances in humor generation. ## conclusion in this work, we explored whether humor prediction data, such as the humicroedit dataset from semeval task 7, could be used to generate humor, examining whether this humor provides more than random surprisal effects. we use these human edited headlines as training for a machine translation system that automatically "translates" normal headlines into humor. we then build a intelligent random system as a baseline, showing that our generative headlines are significantly better than random effects, illustrating that our results are due to more than spurious correlations. we further find that our system's humorous headlines are preferred equally with those of the human generated edits, with equal proportions of crowdsourcers thinking these headlines are human generated and humorous. as this initial positive result shows that a humor prediction dataset can be used successfully for generating humor, we hope that future generative systems for humor will consider utilizing and improving from such resources.