A mobee at S em E val-2020 Task 7: Regularization of Language Model Based Classifiers (2020)

Abstract: This paper describes Amobee’s participation in SemEval-2020 task 7: “Assessing Humor in Edited News Headlines”, sub-tasks 1 and 2. The goal of this task was to estimate the funniness of human modified news headlines. in this paper we present methods to fine-tune and ensemble various language models (LM) based classifiers to for this task. This technique used for both sub-tasks and reached the second place (out of 49) in sub-tasks 1 with RMSE score of 0.5, and the second (out of 32) place in sub-task 2 with accuracy of 66% without using any additional data except the official training set.

## introduction while most of the studies are interested in whether a chunk of text is funny, this study examined whether an atomic change can transform a non-funny headline into a funny one. the study focused on two sub-tasks. the first task is to the determine how funny the micro change in the headline is. the second task is to choose between two micro changes of the original headline, which one is funnier. this task is part of semeval 2020 workshop @xcite . there were 49 groups in the first sub-task and 32 groups in the second. contextual word embeddings is currently one of the most popular ways to create a numeric representation of a document. it is capable of capturing the context of a word in a document, semantic and syntactic similarity, relation with other words, etc. pre-trained models of this kind, such as @xcite @xcite , can be fine-tuned to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. however, due to the fact that these models usually have a large number of parameters, over 300 million for the models mentioned above, they tend to overfit very quickly when used without any modification on small datasets. for example, bert-large @xcite has over 30,000 times more parameters than the semeval-2020-task-7-dataset has training examples. in this paper, we present a way to improve the performance of systems that are based upon language models (lm) or contextual token representation models by adding an l2-regularization term on the difference between the pre-trained lm and the fine-tuned model. for convenience, we'll use the term lm in this paper despite the fact some of the models we refer to are word representation models and not proper language models. the paper is organized as follows: section 2 describes the background for this task and an explanation of the dataset. section 3 describes our system architecture and presents the l2-regularization loss for training the language model and our ensemble method. section 4 describes our experimental setup, and section 5 summarizes our results. ## background the "assessing humor in edited news headlines" task, shared by semeval 2020, consisted of two sub-tasks. the first sub-task was a regression problem in which we needed to predict how funny a micro change in news headline is. micro change was defined as one of the following replacements: @xmath0, @xmath1. each edited headline was scored by five judges, each of whom assigned a grade from 0 (not funny) to 3 (funny). the funniness level of each headline is the mean of its five funniness grades. example of sample from the data is presented in ## experimental setup the competition dataset is described at @xcite . for the first sub-task the dataset consists of 12,071 training samples and 3,024 test samples. before the training phase we randomly set aside 1% of the training data for validation. the model is then trained for 15 epochs, and a version of the model