XSYSIGMA at S em E val-2020 Task 7: Method for Predicting Headlines’ Humor Based on Auxiliary Sentences with EI - BERT (2020)

Abstract: This paper describes xsysigma team’s system for SemEval 2020 Task 7: Assessing the Funniness of Edited News Headlines. The target of this task is to assess the funniness changes of news headlines after minor editing and is divided into two subtasks: Subtask 1 is a regression task to detect the humor intensity of the sentence after editing; and Subtask 2 is a classification task to predict funnier of the two edited versions of an original headline. In this paper, we only report our implement of Subtask 2. We first construct sentence pairs with different features for Enhancement Inference BERT(EI-BERT)’s input. We then conduct data augmentation strategy and Pseudo-Label method. After that, we apply feature enhancement interaction on the encoding of each sentence for classification with EI-BERT. Finally, we apply weighted fusion algorithm to the logits results which obtained by different pre-trained models. We achieve 64.5% accuracy in subtask2 and rank the first and the fifth in dev and test dataset 1 , respectively.

## introduction humor detection is a significant task in natural language processing. most of the available humor datasets treat it as a binary classification task @xcite @xcite , i.e., examining whether a particular text is funny. it is interesting to study whether a short editing of a text can turn it from non-funny to funny. in this paper, we examine whether the headline is funniness or which word substitution is more funniness after a short editing. such research helps us to focus on the humorous effect of word changes. our goal is to determine how the machine understands the humor generated by such brief editing and to improve the accuracy of model prediction by different strategies, including rewriting the input sentences of model, enhancing local inference of features and applying the strategies of data augmentation and model ensemble. the competition dataset comes from humicroedit @xcite , a novel dataset for research in computational humor. each edited text headline is labeled into a score of 0-3 by 5 judges with an average final score. the competition consists of two subtasks: subtask 1 is to predict the average funniness of the edited headlines given the original headlines and the edited headlines and subtask 2 is to predict which edited text is more funniness in comparison given the original headlines and two edited ones. this competition is significant because it is helpful for the task of generating humorous texts @xcite , which can be applied to chatbots and news headline generation. in this paper, we only report our implementation on subtask 2. we conduct several key data preprocessing, including data cleaning, constructing different sentence pair inputs, delivering semi-supervised clustering of the replacement words, and adding clustering labels as the features of the model inputs. after that, we apply bert to obtain the word embeddings of the two input sentences and calculate the difference of the feature information with soft alignment. next, we enhance the local inference information and concatenate them into a sequence to the softmax layer for classification. finally, we apply data augmentation and the pseudo-label iterative learning strategy to enrich the data information from the embeddings of three pre-trained models, bert, xlnet, and roberta. cross validation and weighted voting ensemble are conducted to improve the model performance. the contributions of this paper can be summarized as follows: ## related work recently, humor detection has become a hot research topic. khodak et al. introduces the self-annotated reddit corpus (sarc), a large corpus for training and evaluating the task of sarcasm detection. @xcite apply semi-supervised learning technique to identify sarcasms on two very different datasets and discuss the differences between the datasets and the algorithms. @xcite investigate the automatic method of detecting irony and humor in social networks. they cast the problem into a classification problem and propose a rich set of features and text representation to train the classifier. @xcite detect double-entendres, address this problem in a classification approach that includes features that model those two characteristics. as a powerful semantic information feature extractor proposed by @xcite , bert has two significant points that we can utilize: (1) feature extraction with transformer encoder, and training with mlm and nsp strategy pre-training; (2) the two-stage model of pre-training for large data scale pre-training and fine-tuning training for specific tasks. it demonstrates the effectiveness of bidirectional pretraining for language representation. unlike the previously used unidirectional language models for pretraining, bert applies a masked language model to achieve pre-trained deep bidirectional representations, which is the first representation model based on fine-tuning and achieves the most advanced performance on a large number of sentence-level and token-level tasks, also stronger than many task-oriented architectures. nowadays, researchers have proposed to construct auxiliary sentences in nlp tasks to enrich the input of bert. for example, @xcite apply bert to optimize fine-grained emotion classification and compare the experimental results of single sentence classification and sentence pair classification based on bert fine-tuning, analyze the advantages of sentence pair classification, and verify the validity of conversion method. @xcite propose to unify the qa problem and the text classification problem into a reading comprehension problem with the help of auxiliary data to enhance the model performance, and demonstrate that bert is more suitable to handle natural language inference (nli) dataset. clark et al.(2019) also show that the bert model can learn more semantic information from inferential data. @xcite concludes that if bert is only adopted as a feature expression tool, the input side of bert is just to enter question or passage separately, and take out the [cls] mark of bert high-level as the semantic representation of question or passage, this method is far less effective than entering question and passage on the bert side at the same time which let transformer do the matching process of question and passage. ## experimental setup the official dataset consists of 13,694 labeled data in training, development, and test. the additional training dataset and the development dataset are released afterwards while, the test dataset consisting of 2,960 sentences. in order to make the development set with nearly the same size of the test set, we conduct 5-fold cross-validation. for each fold, we perform data augmentation on the training dataset, the development dataset remaining unchanged. similarly, we conduct data augmentation with 5-fold crossvalidation again after adding pseudo-label, plus three full-volume data (without division development dataset) as the training dataset, we get 13 datasets in the end. the data pre-processing stage includes abbreviation reduction, spelling correction, word stem extraction, upper and lower case letter conversion, special symbol processing and other operations, at the same time, the data with the same training sample but inconsistent labels after the conversion of upper and lower case letters are screened out. ## results in subtask 2, we achieve the final results of 0.64460 accuracy and 0.25411 reward score, which was ranked the fifth in the test phrases. in ## conclusion and future work this paper presents a method to detect humor in edited news headlines of the enhancement inference bert model based on auxiliary sentences. we first creatively make a variety of differently auxiliary sentence pair inputs, and then use the bert pre-trained model to encode the sentence pair. secondly, we perform the enhancement of local inference information on the sentence pair features. while utilizing we make data augmentation and pseudo-label. finally, we carry out a multi-model weighted ensemble strategy to enhance the model performance. in the future, we will exploit multi-task learning to simultaneously learn subtask 1, a regression task, and subtask 2, a classification task. by sharing the information among the tasks, we can gain more knowledge and apply a more precise data representation.