Stimulating Creativity with F un L ines: A Case Study of Humor Generation in Headlines (2020)

Abstract: Building datasets of creative text, such as humor, is quite challenging. We introduce FunLines, a competitive game where players edit news headlines to make them funny, and where they rate the funniness of headlines edited by others. FunLines makes the humor generation process fun, interactive, collaborative, rewarding and educational, keeping players engaged and providing humor data at a very low cost compared to traditional crowdsourcing approaches. FunLines offers useful performance feedback, assisting players in getting better over time at generating and assessing humor, as our analysis shows. This helps to further increase the quality of the generated dataset. We show the effectiveness of this data by training humor classification models that outperform a previous benchmark, and we release this dataset to the public.

## introduction while some data for machine learning tasks, like image object detection, is relatively easy to annotate, generating data that depends on human creativity is quite difficult. unlike many objective tasks, creativity is much less constrained, people do not always agree on the quality of creative output, and measuring creativity often requires more effort. one such creative endeavor is humor, which is subjective and hard to define, making consensus difficult. factors such as human preferences, mood, world knowledge, and complex interrelationships play important roles in determining what is funny, making it challenging to generate humor datasets. one approach to crowdsourcing creative data is to hire workers, as we did for humorous headlines in @xcite . an alternative is to make a game for gathering the data. games are especially suitable for gathering creative output because it is enjoyable to both generate and rate creative artifacts, particularly humor. the effectiveness of games for labeling data has been shown in previous work @xcite . in this paper, we introduce funlines foot_0 , an online game for generating funny news headlines for humor research. we explore and evaluate this fun, competitive way of motivating people to contribute creative text, addressing some of the special challenges of generating humor data mentioned above. generating a dataset of creative artifacts for machine learning requires ratings of the artifacts. this has been done for sites like reddit (reddit.com) for jokes and photofeeler (photofeeler.com) for photos, both of which allow casual users to upvote/downvote creative content from others. the hafez research project lets users rate machine generated poetry @xcite . in fun-lines, one of the two main tasks for players is to rate the funniness of headlines that have been edited by other players. humor is inherently social, and the headlines' collective exposure to, and rating from, humans preserves this important feature. the other main task in funlines is to edit regular headlines to make them funny. to simplify machine learning and humor analysis, we want funny headlines that are highly constrained. in funlines, players start with a regular headline from a news source and they change a single word or entity to make it funny, giving data that is particularly suitable for understanding the tipping point between serious and humorous. this contrasts with unfun.me, which is also a game for generating pairs of funny and unfunny headlines @xcite . with unfun.me, players start with a funny headline and make it serious, where their edits are unconstrained but encouraged to be minimal. because it starts with regular headlines, funlines has an enormous amount of raw source material. also related to funlines' editing tasks are projects that help users create humor, like humortools @xcite and libitum @xcite . by deploying funlines, we learned how players behaved, and we responded with modifications to encourage players to provide creative, high quality data. we show that over time, players made significant improvements in both the quality of their humor and consistency of their ratings. we also show that the resulting data is effective for training humor detection systems. in fact, funlines uses one of our humor detectors to give instant feedback when players edit a headline. we compare the funlines data to humicroedit, which is a humorous headlines dataset we collected previously from crowd workers @xcite . funlines data is less expensive, higher quality, and leads to improved automated humor detection. overall, we show that a competitive game is effective for gathering data for a human creativity task based on an experiment with a large number of users. ## the funlines humor game funlines is designed to collect a large volume of rated, humorous headlines. it makes the humor generation process fun, interactive, competitive, rewarding and educational, keeping players engaged. in funlines, players can attempt two tasks: (i) edit regular news headlines to make them funny, and (ii) rate the funniness of headlines edited by other players. they receive feedback for these actions, which helps them get better at the game. players are ranked in our performance-based leader-boards, which offer prize money. we now describe the various aspects of the game that make it work. ## gathering data in this section, we describe our player recruiting strategies, our gathered dataset, and its comparison against humicroedit @xcite . ## analysis of player behavior we examine how players adapted to the funlines competition and the headlines they attempted. ## detecting humorous headlines we investigate if the funlines dataset of 8,248 fully rated headlines is suitable for humor detection. ## conclusion and future work funlines is an online game for generating funny headlines. while creative data can be difficult to obtain, funlines makes it easier by taking advantage of the inherent fun of creativity and competition. we described the game, including a rich set of feedback for players to assess their own performance along with controls and incentives for them to create funny headlines. our deployment attracted 290 players for a total cost of us$ 1,000. compared to our earlier work for gathering the same type of data from only turkers, funlines produced funnier headlines with better rating agreement and at nearly 60% lower cost per headline. we showed how players' performance improves over time, both in terms of their headline quality and rating consistency. we showed how the funlines data is effective for training machine learning models to detect and to rate humorous headlines. furthermore, this data has already found use in a recent semeval shared task on humor recognition @xcite foot_2 . having built models for detecting humor, we see this data as the foundation of the automatic creation of humorous headlines in a generate-and-test approach. more generally, funlines is a prototype for gathering datasets of creative artifacts from people that is engaging, interactive, competitive, rewarding, educational and inexpensive. in future, we would like to extend the funlines data collection setup to a more general crowdsourcing framework, for example, to collect style transfer data.