“Judge me by my size (noun), do you?” Y oda L ib: A Demographic-Aware Humor Generation Framework (2020)

Abstract: The subjective nature of humor makes computerized humor generation a challenging task. We propose an automatic humor generation framework for filling the blanks in Mad Libs® stories, while accounting for the demographic backgrounds of the desired audience. We collect a dataset consisting of such stories, which are filled in and judged by carefully selected workers on Amazon Mechanical Turk. We build upon the BERT platform to predict location-biased word fillings in incomplete sentences, and we fine-tune BERT to classify location-specific humor in a sentence. We leverage these components to produce YodaLib, a fully-automated Mad Libs style humor generation framework, which selects and ranks appropriate candidate words and sentences in order to generate a coherent and funny story tailored to certain demographics. Our experimental results indicate that YodaLib outperforms a previous semi-automated approach proposed for this task, while also surpassing human annotators in both qualitative and quantitative analyses.

## introduction computer-generated humor is an essential aspect in developing personable human-computer interactions. however, humor is subjective and can be interpreted in different ways by different people. humor requires creativity, world knowledge, and cognitive mechanisms, which are extremely difficult to model theoretically. generating humor is considered by some researchers as an ai-complete problem @xcite , hence humor generation was largely studied in specific settings. language preferences vary with user demographics @xcite @xcite @xcite , and this has led to approaches leveraging the demographic information of users to obtain better language representations and classification performances for various nlp tasks @xcite @xcite . humor is a universal phenomenon that is used across all countries, genders, and age groups @xcite . likewise, there are variations in how humor is enacted and understood due to demographic differences @xcite @xcite @xcite . however, to the best of our knowledge, the effect of user demographics in computational humor generation has not been studied. in this paper, we introduce yodalib, an automatic humor generation framework for mad libs r -a story-based fill-in the blank gamewhich also accounts for the demographic information of the audience and story coherence. we use location as the demographic dimension. yo-dalib has three stages: (1) a candidate selection stage in which candidate words are selected to fill the sentences in each story, (2) a candidate ranking stage to assess if a filled-in (transformed) sentence is a funny version (transformation) of the original sentence, and (3) a story completion stage to join individually funny sentences to form complete mad lib stories that are humorous. fig. 1 shows an example filled-in mad lib. this paper makes four main contributions: (1) we collect a novel dataset for location-specific humor generation in mad lib stories, which we carefully annotate using amazon mechanical turk (amt). 1 (2) we propose yodalib, a location-specific humor generation framework that builds on top of bert-based @xcite components while also accounting for location and story coherence. yodalib typically generates funnier mad lib stories than those created by humans and a previously published semi-automatic framework. (3) we present qualitative and quantitative analyses to explain what makes the generated stories humorous, and how they differ from the other completions. (4) finally, we outline the similarities and differences in humor preferences between two countries: india (in) and united states (us), in terms of certain linguistic attributes. to the best of our knowledge, ours is the first computational study to automatically generate humor in a mad lib setting while also incorporating the demographic information of the audience, and analyzing its effect in terms of various linguistic dimensions. ## related work there is a long history of research in general theories of humor @xcite @xcite @xcite . in computational linguistics, a large body of humor research involves humor recognition, and it typically focuses on specific types of jokes @xcite @xcite @xcite @xcite . research work on humor generation has been largely limited to specific joke types and short texts, such as riddles @xcite , acronyms @xcite , or one-liners @xcite . in general, it is very difficult to apply humor theories directly to generate humor, as they require a high degree of commonsense understanding of the world. owing to the subjective nature of humor, there have been recent efforts in collecting datasets for humor; @xcite collected a dataset of jokes and funny dialogues in russian from various online resources, and complemented them carefully with unfunny texts with similar lexical properties. they developed a fine-tuned language model for text classification with a significant gain over baseline methods. @xcite introduced the first multimodal language (including text, visual and acoustic modalities) dataset of humor detection, and proposed a framework for understanding and modeling humor in a multimodal setting. in socio-linguistics, the relationship between humor and gender is widely studied. @xcite found that new zealand women more often shared funny personal stories to create solidarity, while men used other strategies to achieve the same goal. more recently, location has become central in sociolinguistics @xcite . @xcite indicated that humor styles vary with countries, and humorous communications from korea, germany, thailand and us, had variable content for funny advertising, while sharing certain universal cognitive structures. the effect of demographic background on language use has gained significant attention in computational linguistics, with several efforts focused on understanding the similarities and differences in the language preferences, opinions and behaviors of people @xcite @xcite @xcite . conversely, there has been work to leverage these demographic differences in language preferences between various groups, to develop better models for nlp tasks, such as sentiment analysis @xcite , word representations @xcite , sentiment, topic and author attribute classification @xcite , and word associations @xcite . however, to our knowledge, none of this recent work accounts for demographic information in humor recognition or generation tasks. we find inspiration in recent work by @xcite , who collected a humor generation dataset with mad lib stories, and proposed a semi-automated approach to aid humans in writing funny stories. we go one step further and propose a fully-automated bert-based demographic-aware humor generation framework. we further study the influence of location on humor preferences via amt and seek to emulate such preferences in our automatically generated stories. our work is similar to @xcite , as our goal is to engage readers from different demographic groups by generating funnier versions of stories to read. ## data collection mad libs r is a fill-in-the-blank game @xcite to create funny stories. a mad lib is a textual story template consisting of a title and a short story, with some of the words masked. players are prompted to provide replacement words for the masked entries based on the provided hints (e.g., part-of-speech (pos) tags, bodypart, food) without having read the story. the replacement words are then filled in the story; the resulting mad lib is usually funny, with the humor aspect coming from the nonsensical filled-in words in an otherwise coherent and sensible story. we use stories curated by @xcite , namely fun libs, as (1) mad libs are copyrighted and hence it is difficult to release datasets, and (2) experimentation with fun libs allows comparison of our approach with that proposed by @xcite . 2 we discard 4 of the 50 fun libs, as their themes cater to a us audience, foot_2 and replace them with 4 new stories we created following the heuristics devised by @xcite . the data annotation is undertaken by two parties: the players who fill-in the blanks to create funny stories, and the judges who assess the filled-in stories in terms of their funniness. we assume the threestage annotation framework devised by @xcite : judge selection, player selection and story annotation, with a few revisions to account for location-specific annotations. judge selection. this is done via a linguistic and a demographic survey. turker judges from each country are given seven pre-filled stories, three of which are taken directly from wikipedia, with some words underlined as if they were filled-in words, while the remaining four stories were filled-in by english speakers from the corresponding country, who were instructed to create funny stories. we instruct the judges to select for each story a grade from {0: not funny, 1: slightly funny, 2: moderately funny, 3: funny}. to filter spam responses, verification questions are presented that can be answered only after reading the stories. the task ends with a demographic survey prompting the judges to provide their age group, nationality, gender, education, occupation, and income level. selected turkers are those who (i) assign 0 to the wikipedia stories, a grade from {1, 2, 3} to at least three of the remaining four stories, (ii) answer the story-based questions correctly, and (iii) spend at least 4 minutes to complete the task. 50 us and 43 in judges are selected from 60 and 100 candidates respectively, ensuring that they are unbiased in judging the funniness of a filled-in story. player selection. players are expected to be good at writing funny stories. for this, we obtain four stories from wikipedia, mask some words, and provide hint types next to them. to avoid excess workload to the turkers (leading to possible filling bias), two tasks are created for each country on amt, each with two stories and a demographic survey, with instructions to fill the stories to make them funny, and answer the demographic survey. further instructions are provided: the filled-in words must (i) occur in english dictionary, (ii) have exactly one word written in latin alphabet, (iii) agree with the provided hint types, and (iv) not be slang words or sexual references, as these lead to shallow humor. in each country, the filled-in stories are graded on the 0-3 humor scale by 5 qualified judges from that country, to reduce the effect of variations in humor preferences, and be representative of an audience rather than an individual on amt. players are selected if their mean funniness grade (mfg), mean of the 5 judgements, ≥ 1 for at least one story. 30 in and 26 us players are selected from 80 and 60 turkers respectively. annotating fun libs. for each story, we ensure that a qualified turker can participate as either a player or a judge but not both. the stories are filled by 3 players from each country following the player selection instructions, and are judged by 5 judges from the same country. the judges rate the overall funniness, coherence, deviation from the story title (on the 0-3 scale), whether incongruity 4 was applied by the player ({yes, no}), the humor contribution of each filled-in word ({'funny', 'not funny'}), and a verification question. in addition to same country judgements, we also obtain judgements from the opposite country for the filled-in test stories, to allow for cross-country analyses. we compare coherence to incongruity and deviation, to understand their effect on humor. the krippendorf's alpha @xcite values for in and us are 0.214 and 0.173 respectively, which indicate positive agreements among amt judges, and are comparable to those obtained by @xcite , who crowd-sourced a dataset of humorous edited news headlines on the same funniness scale. the total amt cost is about $1,200. ## automatic humor generation framework we first describe the location-specific training components, namely the location-biased language model and the location-specific humor classifier. then, we focus on the story generation pipeline that follows three stages: (1) a candidate selection stage where possible word replacements for the blanks are generated using a location-biased language model; (2) a candidate ranking stage where the selected candidates are ranked by their humor contribution to the sentences they occur in, using a location-specific humor classifier; (3) a story completion stage where funny stories are created by selecting the top ranked funny transformations for each sentence, and concatenating them to obtain complete stories. ## evaluation we evaluate three methods to write funny stories: for each of these, we obtain 5 judgements on amt. for each test fun lib, grading is done by judges from both locations on the 3 filled-in stories from ft, and the top 10 stories generated from mlm and yodalib. we grade more stories generated by yodalib as: (1) unlike the manually crafted ft stories , these are generated automatically, and hence require minimal effort (once the yodalib framework is in place), and (2) both story funniness and average word coherence scores for the top 10 stories differ only in the fourth decimal place. hence we treat them as the best diverse humorous variations generated for the given fun libs (best10). the stories from ft and yodalib approaches are created in location-specific settings, while mlm does not account for the desired location slant. stories from mlm are ranked based on their filled-in word probabilities of fitting the corresponding contexts. hence, we expect the top-ranked mlm stories to be more congruous and less humorous than the low-ranked stories. 7 the quality of stories is determined by human subjects, as this task is subjective. humor generated by both ft and yodalib is preferred to that by mlm, by both in and us judges. this is expected, as mlm fills the stories with more plausible words, and hence does not introduce any surprise aspect that is essential for humor. the average mfg for mlm increases in the top10 setting, confirming that stories become more humorous as more incongruous words are used to fill them. in the ft stories, both in and us judges prefer us-written humor (1.57, 1.41) to in-written humor (1.17, 1.39). on the other hand, in-written humor is liked more by us judges (1.39), and us-written humor by in judges (1.57). hence, an average us turker writes better stories than an average in turker for mad libs, and judges in general enjoy humor written by turkers from a different country more than of their countrymen. this is in contrast to the general expectation that people prefer humor originated from their own group due to a better understanding of the various location-specific subtleties, suggesting that seeing things in a new light possibly contributes to the surprise and creativity factors essential for humor generation. ## conclusion in this paper, we studied location-specific humor generation in mad libs stories. we first collected a novel location-specific humor dataset on amt, by selecting players and judges to obtain ground truth data. next, we proposed an automated location-specific humor generation framework to generate possible candidates to fill-in the blanks, to rank these candidates based on their humor contributions to form funny sentences, and to complete the stories by selecting the best transformations for the constituting sentences. our approach outperformed a simple language model and human players (in most cases) in generating funny stories. we also performed a detailed demographic-based analysis of our dataset. we found that humor created with us slant is in general preferred to in slant humor by both in and us judges. in judges seemed to have a more lenient outlook towards humor, while us judges have higher expectations possibly in terms of coherence, which is also reflected in the better quality humor generated in the us-specific setting. when turkers wrote funnier stories, they did so in a coherent manner, indicating the vast potential of coherence in generating humor, contrary to the general incongruent take of mad libs. humor from our approach is generated primarily via incongruity and deviation, despite our preliminary measures to incorporate coherence. we believe that our proposed bert-based approach is general, and can be used for other affect-based nlp tasks with minor changes. in the future, we aim to extend our research in several directions, using (1) a larger number of and a wider variety of mad lib-like stories; and (2) other demographic dimensions (e.g., gender, age group, occupation), and more groups within each dimension (e.g., singapore, canada, england for location; arts, engineering, fashion, publishing for occupation).