LT 3 at S em E val-2020 Task 7: Comparing Feature-Based and Transformer-Based Approaches to Detect Funny Headlines (2020)

Abstract: This paper presents two different systems for the SemEval shared task 7 on Assessing Humor in Edited News Headlines, sub-task 1, where the aim was to estimate the intensity of humor generated in edited headlines. Our first system is a feature-based machine learning system that combines different types of information (e.g. word embeddings, string similarity, part-of-speech tags, perplexity scores, named entity recognition) in a Nu Support Vector Regressor (NuSVR). The second system is a deep learning-based approach that uses the pre-trained language model RoBERTa to learn latent features in the news headlines that are useful to predict the funniness of each headline. The latter system was also our final submission to the competition and is ranked seventh among the 49 participating teams, with a root-mean-square error (RMSE) of 0.5253.

## introduction research in the field of computational humor can be divided into three distinct tasks: detecting, assessing and generating humor. humor detection is performed by either distinguishing humorous items such as one-liners, jokes or texts from their non-humorous counterparts @xcite , or by ranking these items according to their respective funniness @xcite . moreover, researchers have tried to gain insights in the techniques and mechanisms that underlie humor production @xcite @xcite . such insights can be used for automatic humor generation @xcite @xcite . due to the subjective nature of humor and the deep conceptual common-sense knowledge that it requires, tasks involving humor remain challenging ai problems. part of the challenge also lies in the scarcity of publicly available datasets. for the semeval shared task on assessing the funninness of edited news headlines @xcite , @xcite introduce a novel dataset, named "humicroedit", that can be used for all three tasks described above (see section 2). the shared task is split into two sub-tasks, namely (i) a regression task to predict the mean grade of all annotators for a given edited sentence, and (ii) a binary classification task to select the funnier of two edited headlines. we only participated in the regression sub-task. the remainder of this paper is structured as follows. in section 2, we provide an overview of the dataset that was used for the competition. section 3 introduces our two machine learning systems, namely (i) a feature-based regression model (nusvr), and (ii) a state-of-the-art deep learning system. next, in section 4, our results for the two systems are compared and discussed. finally, section 5 concludes this paper. ## data analysis while most existing datasets in the field of computational humor are constructed for binary classification purposes @xcite (funny or not), the organizers of the competition provided a novel resource ("humicroedit") that contains humor intensity labels for edited news headlines. the dataset contains around 15,000 headlines @xcite , divided into train (9,653 headlines; 64%), development (2,420; 16%) and test (3,025; 20%) data. the original headlines were collected from reddit, and experts from amazon mechanical turk were asked to create and to estimate the funniness of minor edits to the headlines. the suggested changes were applied to one word or collocation while respecting the following rule: entities or nouns are replaced by nouns only, verbs are replaced with verbs. the scores from annotators ranged from 0 (not funny at all) to 3 (very funny). all instances were rated by 5, 10, or 15 annotators. for each instance, all provided grades and their average were given, the average being the value that models must learn to predict correctly. the distribution of the average grades is visualised in figure 1 , which shows that the dataset contains less highly ranked "funny" samples than lowly ranked "not funny" instances. we examined the headlines and their grades to find any patterns that could prove useful. the original tokens before replacement are mainly single words (94.9%), rarely two (4.9%) and sporadically three words (0.2%). the replacement tokens, on the other hand, are always a single word. we attempted to identify common topics in the original headlines by using latent dirichlet allocation (lda) and keyword extraction based on term frequencyinverse document frequency (tf-idf) scores. the first approach did not provide reasonable results as it was hard to separate the particular independent topics because most of them contain a lot of common terms. with the second approach, we found that the keywords in the headlines are mainly named entities (tags 'person', 'location' and 'organization'). for this reason, we extracted only nouns with those ner tags for all the funniest (with a mean grade higher or equal to 2.0) and the least funny headlines (with a mean grade less or equal to 1.0) and used them to calculate tf-idf scores. these scores were in turn used to select keywords. surprisingly, we did not discover any particular patterns or specific topics for those groups because they mostly had common keywords. we also took into account topics proposed by @xcite for edited words, which the authors ranked according to how they influence the funniness of a headline. they also proposed a set of keywords for each headline. we compared the proposed keywords of the funniest topics with those obtained from the original headlines, but did not find any useful overlap. ## system descriptions in this section we will start by introducing the features that we engineered for a nusvr system (section 3.1), followed by a description of our best performing system, which is a fine-tuned transformer-based language model (section 3.2). ## experimental results we hypothesised that our systems would perform better on instances where annotators agreed more on the funniness score than on the instances where they disagreed. to verify this, we calculated the correlations between the squared error and the annotation entropy and between the squared error and the standard deviation of the ratings. entropy, in this context, can be seen as the amount of uncertainty among or agreement between the annotators to give the same score to a specific instance. 1 the correlations given in ## conclusion in this research, we propose two machine learning systems to tackle the regression task of predicting the funniness of edited news headlines. while our feature-based machine learning system obtains a rmse of 0.5720, it is outperformed by our deep learning system which achieves a rmse of 0.5253. in addition, the pearson correlation between the predicted scores and the given mean scores is better for the latter system. for future work, it would be interesting to test the performance of our best-performing pipeline on other humor detection datasets in order to validate its portability and robustness.