WUY at S em E val-2020 Task 7: Combining BERT and Naive B ayes- SVM for Humor Assessment in Edited News Headlines (2020)

Abstract: This paper describes our participation in SemEval 2020 Task 7 on assessment of humor in edited news headlines, which includes two subtasks, estimating the humor of micro-editd news headlines (subtask A) and predicting the more humorous of the two edited headlines (subtask B). To address these tasks, we propose two systems. The first system adopts a regression-based fine-tuned single-sequence bidirectional encoder representations from transformers (BERT) model with easy data augmentation (EDA), called “BERT+EDA”. The second system adopts a hybrid of a regression-based fine-tuned sequence-pair BERT model and a combined Naive Bayes and support vector machine (SVM) model estimated on term frequency–inverse document frequency (TFIDF) features, called “BERT+NB-SVM”. In this case, no additional training datasets were used, and the BERT+NB-SVM model outperformed BERT+EDA. The official root-mean-square deviation (RMSE) score for subtask A is 0.57369 and ranks 31st out of 48, whereas the best RMSE of BERT+NB-SVM is 0.52429, ranking 7th. For subtask B, we simply use a sequence-pair BERT model, the official accuracy of which is 0.53196 and ranks 25th out of 32.

## introduction humor, a high-level form of human communication, is omnipresent, including in the social media as well as real-life situations. the proper use of humor can have a positive impact on our lives. an automatic identification of humor can help in better understanding the structure and theory of humor. moreover, correctly understanding humor is important for improving the performance of many natural language processing applications such as a sentiment analysis and intention mining. to evaluate the progress of automatic humor assessment, semeval-2020 task 7 1 aims to study how machines can understand humor generated by applying short edits to news headlines @xcite . a dataset, called humicroedit @xcite , with a total of 15,095 english edited news headlines collected from reddit (reddit.com) along with mean humor scores, was developed. with this dataset, humor is generated after a short editing, e.g., "president vows to cut taxes hair". subtask a focuses on the prediction of the mean humor of an edited headline, and subtask b involves predicting the funnier of two edited headlines. to address this problem, we propose two systems foot_0 . the first (bert+eda) is a regression fine-tuned single sequence bidirectional encoder representations from transformers (bert) model @xcite with easy data augmentation (eda) tool foot_1 , which is an implementation of the data augmentation theory of @xcite . the second (bert+nb-svm) is a hybrid of the regression fine-tuned sequence-pair bert and a naive bayes-support vector machine (nb-svm) model @xcite estimated on tfidf features. here we combine the strength of deep learning and classical machine learning. bert+nb-svm outperforms bert+eda. both systems are described in the later sections, although the official result for subtask a is only from bert+eda. for subtask b, we simply use a sequence-pair bert model. the official accuracy is 0.53196, which rankes 25th out of 32. in the rest of this paper, studies related to automatic humor recognition are introduced in section 2. an overview of the system is presented in section 3. we then describe the experiments and results in section 4. finally, we provide some concluding remarks in section 5. ## related work the task of automatic humor recognition refers to deciding whether a given sentence expresses a certain degree of humor. however, this remains a challenge @xcite because there is no universal definition of humor, and an understanding of a same sentence depends on personal background of the readers. previous studies on humor recognition have focused on the binary classification problem (humorous or not). in @xcite , a joke recognizer was developed to determine whether a discovered wordplay makes the text funny by learning the statistical patterns of text in n-grams. in @xcite , semantic features designed to recognize humor based on four structures are described: incongruity, ambiguity, interpersonal effect and phonetic style. moreover, the authors proposed an effective maximal decrement method to extract humor anchors in the text. in addition to linguistic features, some other studies have tried to make use of spoken or multimodal signals for improvement. for instance, in @xcite , a classical supervised decision tree classifier with a set of acoustic-prosodic features and linguistic features is used to recognize humor in conversations from a comedy television show. many recent studies have attempted to utilize a neural network for humor detection. in de oliveira and rodrigo (2015), recurrent neural network (rnns) and convolutional neural networks (cnns) are applied to humor detection in a dataset on yelp reviews. they found that the cnn model outperformed rnn with two points. in @xcite , a cnn architecture is constructed using highway networks and applied to a balanced large-scale dataset called "pun of the day", of both english and chinese texts. in this paper, we constructed systems based mainly on bert and nb-svm models. bert is pretrained on unlabeled text (wikipedia) through joint conditioning on both the left and right contexts in all layers. we chose bert becacuse it has obtained state-of-the-art results on 11 natural language processing tasks, including text classification and regression datasets like glue @xcite . in addition we chose nb-svm model based on the study in @xcite , which demonstrated that nb achieves better results for short snippets of sentiment tasks, whereas svm is better for longer documents. an svm model using nb log-count ratios as features performs well across different tasks. bert focuses on the semantics, and nb-svm focuses on linguistics, and this combination performed well at human annotation challenge at iberlef 2019 @xcite . ## conclusion in this study, we developed and compared systems constructed using bert and nb-svm models to deal with the humor assessment in newly edited headlines at semeval 2020 task 7. the combination of the sequence-pair regression bert model and the nb-svm model performed well for subtask a. however, the eda tool was not completely successful in improving the outcome. moreover, our solutions struggle with the comparison of two sentences with same humor score. the official rmse for subtask a is 0.57369 and ranked 31st out of 48 whereas the best rmse of our submission is 0.52429, which ranks 7th. the official accuracy for subtask b is 0.53196, which ranks 25t out of 32. in future studies, we aim to focus on the using of a large bert model and the development of other useful data augmentation tools.