Buhscitu at S em E val-2020 Task 7: Assessing Humour in Edited News Headlines Using Hand-Crafted Features and Online Knowledge Bases (2020)

Abstract: This paper describes a system that aims at assessing humour intensity in edited news headlines as part of the 7th task of SemEval-2020 on “Humor, Emphasis and Sentiment”. Various factors need to be accounted for in order to assess the funniness of an edited headline. We propose an architecture that uses hand-crafted features, knowledge bases and a language model to understand humour, and combines them in a regression model. Our system outperforms two baselines. In general, automatic humour assessment remains a difficult task.

## introduction humour aims at generating amusement and laughter and can for this reason be considered one of the features enabling the creation of relationships in the interactions between humans. understanding humour requires factual knowledge, context comprehension and-arguably-intelligence. multiple factors play a role in the definition of humour, such as geographical location, culture, level of education and many others. this obviously makes the task of humour detection very hard for machines and artificially intelligent systems. in recent years, researchers operating in the field of computational linguistics have started to look into the topic, and a lot of progress has been made since the seminal paper by @xcite . however, the quality of data sets leaves many questions unanswered, mainly because they are made of single punchlines or because sentences are divided into binary categories. @xcite makes a remarkable effort on creating a data set of edited headlines where each headline is assigned a score representing the intensity of humour of that headline. this innovative data set enables researchers to conduct studies on a more granular level and may unlock novel techniques to get closer to a more efficient and successful computational model of humour. in this paper we propose an architecture that accounts for multiple factors that we believe play an important role in detecting the intensity of humour in a headline. in order to analyse the sentences, we include hand-crafted features extracted from the sentence itself and enable the system to look for the meaning of unknown objects using nell (never-ending language learning) @xcite . this paper is a description of a system providing a solution to the semeval2020 task 7 @xcite , which ranked 22 nd out of 49 teams. the task is comprised of two sub-tasks. the first task is a regression task aimed at predicting the humour intensity of an edited headline. the second one is a classification task in which it is required to select the funnier headline out of the two provided. our main focus was on the first sub-task, as predicting the humour intensity of the two headlines would imply establishing which of the two has the higher score. another related work dedicated to the effort to assimilate external knowledge is the study reported in @xcite . the work introduced an approach to leverage external knowledge bases, such as nell @xcite and wordnet @xcite (a lexical database), in order to integrate the background knowledge and enhance the learning on lstm. finally, with the extensive work done in @xcite on humour generation, the goal of the study was to generate a carefully curated dataset of news headlines with simple edits, based on robust generation strategies that emphasise free form over traditional jokes with a strong template. this facilitates further research into the shared tasks described and performed in this report. ## data analysis the data set consists of micro-edits on headlines: one word has been replaced by another word, e.g. "how trump just made america (pilates) less safe". five mechanical turks are asked to assign a score between 0 -3 to each headline (0: not funny, 1: slightly funny, 2: moderately funny, to 3: funny) @xcite . the overall score of the headline is then the average of those five scores. similarly to @xcite , we find the scores to have a correlation with the headline length -measured as number of tokens present in it -and the relative position of the replaced word within the headline. the humour increases if the edit happens toward the end of the headline, as can be seen in figure 1 . ## system description in this section we outline the structure of our system and go into details on the different components. the proposed system consists of three encoders which handle three different types of inputs. sections 4.1 to 4.3 explain the inputs and how they are handled in each of the three encoders. section 4.4 outlines how the results from each of the three encoders are combined and processed. section 4.5 goes through the training parameters and extra tricks we use to get more performance from our model. the overall structure of the model is shown in figure 2 . section 4.6 explores further improvements we have developed after the official submission deadline. ## results in this section we outline the results gathered from experiments with both the official model and the alternative model created after the official submission deadline. as main evaluation metric we use root mean square error (rmse), which is defined as improvements we present two additional runs as introduced in section 4.6. results on the test set are shown in the two last rows in ## discussion in this section we show results of an ablation study of our official model and discuss limitations of it. ## conclusions we proposed a simple neural model which uses three decoders to model humour intensity of edited headlines. our official submission obtained an rsme of 0.55115 (top scoring team: 0.49725). our ablation study shows that the most important information is word identity of the micro-edits, followed by knowledge base representations. however, we note that the way that we implement the knowledge encoder and phonemic information is somewhat ineffective in capturing the information we hoped it would, which leaves room for future work on this challenging task.