Funny3 at S em E val-2020 Task 7: Humor Detection of Edited Headlines with LSTM and TFIDF Neural Network System (2020)

Abstract: This paper presents a neural network system where we participate in the first task of SemEval-2020 shared task 7 “Assessing the Funniness of Edited News Headlines”. Our target is to create to neural network model that can predict the funniness of edited headlines. We build our model using a combination of LSTM and TF-IDF, then a feed-forward neural network. The system manages to slightly improve RSME scores regarding our mean score baseline.

## introduction how the computer recognizes, generates, and analyzes humor is an interesting but meaningful challenge for artificial intelligence (ai). not only creativity and sophistication of language are needed for the sense of humor, but also world knowledge, empathy, and cognitive mechanisms which are not so easy for technologists to model theoretically. several stable and capacity large enough datasets have been built for training models in computational humor studies. these datasets may contain different contexts @xcite set up funny one-liners which contain more than 24,000 onliner jokes. @xcite collects regular and sarcastic amazon product reviews to identify sarcasm on a sentence level or for a specific phrase. @xcite introduce the self-annotated reddit corpus (sarc) which is a corpus that has 1.3 million sarcastic statements for training and evaluating systems for sarcasm detection. @xcite set up mad libs, which is a popular fill-in-the-blank game judged by the human. and then, @xcite introduce a novel dataset humicroedit that contains original media news headlines from an online posted website reddit and enables several humor tasks for humor understanding, predicting, generating, recommending and ranking. the latest one is funlines, which is a competitive game that not only allows players to edit news headlines but also rate the funniness of headlines edited by others. each dataset has its unique purpose, but how to understand headlines and what makes a few words in one headline so funny are becoming new problems for ai. in this paper, we want to test how machines understand humor generated by such short edits rely on humicroedit to predict the mean funniness of edited headlines. ## related work how to generate humor is not as easy as we think, especially for computers. @xcite developed a formal model producing riddles and implemented it in jape, a computer program which generates simple punning riddles and confirmed that jape's output texts are indeed jokes after evaluating the program by 120 children. @xcite established hahacronym to realize the irony reanalysis and generation of acronyms in a concentrated but unrestricted context. petrovic and matthews (2013) rely on large quantities of unlabeled data to deal with generating a fixed syntactic structure -[i like my x] like [i like my y], z -where x and y are nouns and z describes x and y, and develop an algorithm called libitum that helps humans generate humor in a mad lib, which is a popular fill-in-the-blank game. the algorithm is based on a machine-learned classifier that determines whether a potential fill-in word is funny in the context of the mad lib story. @xcite use an online game for players to change a single word or entity in a regular headline from a news source and edit regular news headlines and that makes the humor generation process highly constrained. as for humor detection, @xcite utilize the semi-supervised sarcasm identification algorithm (sasi), which is the first robust algorithm for recognition of sarcasm, to test two different database: 5.9 million tweets from twitter and 66,000 product reviews from amazon. @xcite also use 50,000 texts retrieved from twitter to evaluate the representativeness and relevance of the humor and irony. @xcite approach the detection of irony and humor as a classification problem applying supervised machine learning methods to the twitter corpus. besides twitter, @xcite apply methods from metaphor identification to "that's what she said" (twss) recognition. @xcite use the self-annotated reddit corpus (sarc), a 1.3 million sarcastic statement corpus, to detect sarcasm. the analysis of humor can be divided roughly into two different levels: word-level and text-level (or more specifically, sentence-level). in word-level, engelthaler and hills (2018) collect 4,997 words to rate humor norms by analyzing several factors that influence the judgments. based on that, @xcite analyzing the semantic, phonological, orthographic, and frequency of words to predict the original humor rating standard and the score of previously unsettled words. their reliability is higher than the half reliability in the original standard, which is estimated by segmentation based on gender or age line. while in sentence-level, @xcite set up a dataset that contains crowd-sourced captions that are judged by people from the new yorker. then they formulate a pairwise comparison task by giving a cartoon and two captions based on the data to let the computer determine which one is funnier. ## task descriptions this task is based on the publication of @xcite . it studies the humorous of edited headlines and releases its dataset called humicroedit @xcite . the task @xcite asks us to predict the funniness of an edited headline, given its original headline and edited headline. each headline is allowed to only change one word or one named entity. ## results we achieve a score of 0.57237 rmse in the competition and rank 30th place. then we later modify our system and further improve the result to 0.56786 rmse score. even though we do not reach a higher position, but we conduct several ablation studies to understand which subsystem contributes most. ## conclusion in this shared task, we present a neural network system combined with lstm and tf-idf where it slightly improves rmse compared to our baselines. though our system only ranks 30th place of the competition, we still construct a neural network system that predicts funniness score roughly and we study the contributions of our subsystems where we find that the combination of lstm, tf-idf and subtraction slightly improved the predictions.