Smash at S em E val-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating (2020)

Abstract: The use of pre-trained language models such as BERT and ULMFiT has become increasingly popular in shared tasks, due to their powerful language modelling capabilities. Our entry to SemEval uses ERNIE 2.0, a language model which is pre-trained on a large number of tasks to enrich the semantic and syntactic information learned. ERNIE’s knowledge masking pre-training task is a unique method for learning about named entities, and we hypothesise that it may be of use in a dataset which is built on news headlines and which contains many named entities. We optimize the hyperparameters in a regression and classification model and find that the hyperparameters we selected helped to make bigger gains in the classification model than the regression model.

## introduction verbal humor uses a variety of linguistic features, such as synonymy, wordplay, and phonological similiarities, as well non-linguistic features like world knowledge, to produce a comic effect. that such a broad set of skills are required to understand humor, has led several researchers to deem that computational humor is an ai-complete problem @xcite . there is a relatively longstanding body of research into humor detection in a limited domain, such as knock-knock jokes @xcite , one-liners @xcite and humorous news articles from the satirical news publication the onion @xcite . however, the use of shared tasks has attracted more attention and interest in the field since 2017. while previous challenges have focused on collecting twitter data @xcite , semeval 2020 @xcite took an original approach and generated the data by collecting news headlines and then asking annotators to edit one word in the headline to make it humorous @xcite . these headlines emulate those of the onion. the edits shown below indicate the location of the substitution and the word to be inserted. the edited headlines were then rated for humor by subsequent annotators. sub-task a was to predict the mean funniness score of the edited headline. in sub-task b, the systems saw two edits of the same headline, and predicted which one had achieved the higher mean funniness score. ## previous work excluding work on puns, there have been three humor detection shared tasks in recent years: semeval 2017 @xcite , haha 2018 @xcite and haha 2019 @xcite . as the tasks and data have varied between them, direct comparison is not possible. however, a comparison of approaches to the tasks shows some interesting trends. semeval 2017's entries were evenly divided between feature engineering approaches and deep learning systems, with both achieving competitive results. the highest ranking team in the official results for task a, svnit @xcite , used an svm with incongruity, ambiguity and stylistic features. the second highest-ranking team, datastories @xcite opted for a siamese bi-lstm with attention. interestingly, a remarkably simple system prevailed in task b: duluth @xcite used the probability assigned to the text by a bigram language model instead of the output of a classifier to make predictions. entries to haha 2018 were divided along similar lines. the winning system used naive bayes and ridge regression models optimized with an evolutionary algorithm @xcite with the runner up using a bi-lstm with attention @xcite . haha 2019 saw a sea change towards the use of transfer learning models, such as bert @xcite and ulmfit @xcite . these models leverage large amounts of data and transformer attention models to learn contextual relations between words. adilism @xcite used multilingual bert base uncased and extended the language model training without labels, before finetuning their system with the dataset labels. the second place system used an ensemble of a bert model and ulmfit, with naive bayes and svm classifiers. the majority of the top entries to this task used bert in some way, although one noted that it did not improve performance as expected @xcite . as bert models are trained on a masked-language model and sentence prediction task, they capture mainly word-level and sentence-level information. by comparison, ernie 2.0 @xcite ) -henceforth ernie -aims to capture more lexical, syntactic and semantic information in corpora, by training on eight different tasks in a continual pre-training framework. knowledge masking features among these eight tasks, and is implemented by treating a phrase or entity as an entire unit, instead of masking the constituent words. the distinction in how bert and ernie learn is illustrated in how they learn the following sentence: harry potter is a series of fantasy novels written by j. k. rowling ## experiments we experimented with optimizing three hyperparameters: learning rate (1e-06, 0.0001 or 0.001), batch size (16, 32 or 64) and number of epochs (3, 4 or 5). for the sake of brevity, we report only the three