YNU - HPCC at S em E val-2020 Task 7: Using an Ensemble B i GRU Model to Evaluate the Humor of Edited News Titles (2020)

Abstract: This paper describes an ensemble model designed for Semeval-2020 Task 7. The task is based on the Humicroedit dataset that is comprised of news titles and one-word substitutions designed to make them humorous. We use BERT, FastText, Elmo, and Word2Vec to encode these titles then pass them to a bidirectional gated recurrent unit (BiGRU) with attention. Finally, we used XGBoost on the concatenation of the results of the different models to make predictions.

## introduction the humicroedit dataset was created for research in computational humor @xcite . the authors employed amazon mechanical turk annotators to edit a single word of some fifteen thousand news headlines in order to make them funny. five other annotators then graded each edited title on a scale of 0-3, with 3 being the funniest. titles, despite being very short (the longest were around twenty words), convey a lot of information. using one word edits is a way to make minimal changes for an expressed purpose. these two factors make the humicroedit dataset a useful tool for studying computational humor. during the competition, the organizers published the funlines dataset, which improved on humicroedit in terms of cost per title by gamifying the process: they provided cash rewards for the best annotators @xcite . the performance of annotators was measured on a mix of how funny their edits were and how well their grades tracked the overall average. players were able to improve in both categories and the authors posit that this was due to the availability of live feedback. on average, the funlines dataset is funnier than humicroedit and there was better agreement on the grades. the funlines dataset was made available to participants of the competition in january. there were two sub-tasks for semeval task 7: 1) for each headline and its edit, predict the mean of the grades assigned by the five annotators; 2) decide which of two title-edit pairs is funnier. we use a few popular pretrained language models in a bigru ensemble for the first task, and we use those results for the second task. the rest of the paper is organized as follows. section 2 presents a brief review on humor assessment and some nlp tools. section 3 presents the bigru ensemble model in detail. section 4 contains some discussion of the results and section 5 is a short conlcusion. ## background humor presents some challenges that other nlp tasks like entailment and analogy, for example, do not: it is highly contextual and subjective. because of the latter, we may prefer to ask how funny something is or to whom is it funny, and when, over a question like "is this funny? yes or no". so, attempts at computational humor lend naturally to regression tasks. continuous representations have proven to be a good basis from which to transform a natural language task into a computational one. in order to produce continuous representations, many researchers have relied on the distributional hypothesis, which states that the meaning of a word is the assortment of humicroedit light heavy fasttext wiki news fasttext com. crawl word2wec google news bert base uncased bert large uncased bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh fasttext wiki news fasttext com. crawl word2wec google news bert base uncased bert large cased elmo lstm1 bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh bigru attention dense(50) relu dense(1) tanh xgboost prediction preprocessing embedding 15-fold cross-validation concatenate individual model predictions contexts in which it appears. here we briefly mention the four language models for producing continuous representations that we use for task 7. word2vec trains a shallow neural network to predict a word from its context (skip-gram) model or the context from a word (continuous bag-of-words) @xcite . fasttext does something similar but trains on characters, whose n-grams are averaged to get word vectors @xcite . this allows fasttext to perform better than word2vec on out-of-vocabulary (oov) words. elmo trains continuous representations by using an lstm to predict the next token in a sequence of tokens, and it does this in reverse too, with a bidirectional lstm @xcite . the representations are then combined linearly, creating context dependent word representations. bert uses an attention style, multi-level encoder-decoder structure called a transformer, which can model relationships between tokens that are farther apart than can lstms @xcite . the bert transformer uses masking on about 15% of tokens-sub-word entities of a non-fixed length. in pretraining, bert also asks the model to predict whether two sentences are sequential. bert allows users to select vectors from specific levels of the transformers and combine them in various ways. another feature, and one that is relevant here, is that the user can choose to not pool the vectors at all, and this results in word vectors. ## system overview as shown in figure 1 , this system is a stacked bidirectional gru ensemble. the first level uses cross validation on 11 different embeddings. predictions of the first level are passed to an xgboost regressor to get the final output. the results for sub-task 1 were directly used to decide which of the two headlines was funnier for sub-task 2, so the following system is designed simply to predict how funny the annotators thought the edited headlines were. ## results and discussion our model placed third in both sub-tasks with an rmse of 0.51737 for the first and an accuracy of 0.65906 for the second. over the course of this competition, much of our improvement was due to time consuming trial-and-error in composing the ensemble. it is surprising to us that fasttext and word2vec have individual model scores on par with the model using bert encodings. in fact, as the simple average of the results produced by each model was much better than any individual model, but not as good as xgboost. the baseline of predicting the mean is actually in the ballpark of some individual models we tried. to evaluate the utility of this project, it might help to compare any model with human performance, perhaps using the funlines data, but we reserve this idea for later. another idea to pursue is to make this into a classification task: humans, especially those who read news, would presumably be pretty good at deciding whether a title has been edited for the purpose of making it funny. ## conclusion this project used a bigru ensemble with xgboost to predict how funny amazon mechanical turk annotators found edited news headlines. a third place finish on both subtasks of the related semeval task 7 was achieved largely by using a large number of models for encoding the titles, and by tuning an ensemble for this task.