Ferryman at S em E val-2020 Task 7: Ensemble Model for Assessing Humor in Edited News Headlines (2020)

Abstract: Natural language processing (NLP) has been applied to various fields including text classification and sentiment analysis. In the shared task of assessing the funniness of edited news headlines, which is a part of the SemEval 2020 competition, we preprocess datasets by replacing abbreviation, stemming words, then merge three models including Light Gradient Boosting Machine (LightGBM), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representation from Transformer (BERT) by taking the average to perform the best. Our team Ferryman wins the 9th place in Sub-task 1 of Task 7 - Regression.

## introduction with the increased influence of social media on modern society, massive news emerges on the internet every single day. to attract people's attention, editing news headlines to make them more humorous is now an effective approach. however, considering the amount of headlines, it is impractical to manually modify each headline one by one. to solve this problem, some automatic methods to add humor in news headlines are needed. assessing the funniness of edited news headlines plays a significant role there. here we present the method to automatically assessing the funniness of edited news headlines using machine learning algorithms. in semeval-2020 task 7: assessing the funniness of edited news headlines, nearly all existing humor datasets are annotated to study whether a short text is funny @xcite . the goal of the task is to determine how machines can understand humor generated by short edits. the task has been divided into two sub-tasks: 1) given the original and edited headline, predicting the mean funniness of the edited headline; 2) given the original headline and two edited versions, predicting which edited version is the funnier of the two. the two sub-tasks are independently evaluated by root mean square error (rmse) on the overall test set. since our team has participated in multiple tasks in semeval-2020 simultaneously, we only selected the sub-task 1 in task 7. in this task, we encountered two main problems: (i) training single model has no significant effect; (ii) the raw dataset fits bidirectional encoder representation from transformer (bert) badly. to address problem (i), we adapt and fine-tune the merged model, such as increasing weights of edited words in light gradient boosting machine (lightgbm), and input the sentences before and after modification as features in long short-term memory (lstm). then we merge three models, which are lightgbm, lstm and bert respectively. to accelerate model training, we improve lightgbm with word2vec, fasttext, and global vectors for word representation (glove). and to reduce , we use lstm with max pooling and min pooling, supplementing with enhanced sequential inference model (esim). we also use multi-sample dropout, frequent pattern growth (fpg), 5-fold cross validation, and last-5-[cls] @xcite to enhance the effect of bert. besides, we adopt some preprocessing methods such as abbreviation replacement and word stemming to achieve a better result on bert as mentioned in problem (ii). in the rest of this paper, we organize the content as follows. related work of humor detective and the merged models will be presented in section 2. section 3 introduces data description, details of preprocessing, and the methodology of our models. experimental results are discussed in section 4. we also present the conclusion of our work at the end of paper. ## related work humor is a complex emotion, which needs the ability to perceive fully understanding of the connection or relationship between objects in various occasions. in recent years, automatic humor recognition technology has developed rapidly. nevertheless, to the best of our knowledge, there are not much work in this domain, because detecting humor requires a lot of cross-domain and cross-cultural knowledge, which makes it complex. for instance, a joke that is generally interesting to british may not be equally funny to chinese because of different culture background. on a binary humor identification task, cattle and ma have explored the use of semantic relatedness features based on word associations, in the meanwhile, they identify several factors that make word associations a better fit for humor @xcite . a rich set of features for text interpretation and representation to train classification procedures in detection of irony and humor in tweet has been proposed in 2014, which achieves state-of-the-art performance in cross-domain classification experiments @xcite . yang et al. have developed a simple and effective method to extract anchors that enable humor in a sentence @xcite . this methodology is effective in automatically distinguishing between humorous and non-humorous texts. there is a technique that can be implemented in the task to improve the performance. lstm, was firstly proposed in 1991 @xcite , is an extension of recurrent neural network, which have been implemented in several natural language processing (nlp) tasks such as sentiment classification, neural translation, and language generation etc @xcite . some practical attempts have been carried out in this field. nabil hossain et al. have used lstm with 16 hidden unites to detect humorous headlines @xcite . bertero and fung show how the lstm effectively models the setup-punchline relation reducing the number of false positives and increasing the recall @xcite . ## conclusion in this work, we have presented our system for semeval sub-task 1. in sub-task 1, we have trained three models by fine-tuning the parameters and preprocessing the dataset to detect humor in news headlines. after comparing three methods, lightgbm performs best on the rmse while bert does the opposite, and lstm is somewhere between the two. considering a compromise, we merge them by taking the average of their predicted values. to conclude, the evaluation results indicate that our system is capable of detecting humor in news headlines robustly. however, how to tune the parameters is nontrivial, and there are more efficient ways to be explored, which could yield better performance, especially on bert. in the future, in order to increase the accuracy of the system, we plan to improve the performance of bert by taking more preprocessing mechanisms and deeper feature extraction on edited words, and fine-tuning parameters further.