SO at S em E val-2020 Task 7: D eep P avlov Logistic Regression with BERT Embeddings vs SVR at Funniness Evaluation (2020)

Abstract: This paper describes my efforts in evaluating how editing news headlines can make them funnier within the frames of SemEval 2020 Task 7. I participated in both of the sub-tasks: Sub-Task 1 “Regression” and Sub-task 2 “Predict the funnier of the two edited versions of an original headline”. I experimented with a number of different models, but ended up using DeepPavlov logistic regression (LR) with BERT English cased embeddings for the first sub-task and support vector regression model (SVR) for the second. RMSE score obtained for the first task was 0.65099 and accuracy for the second – 0.32915.

## introduction humor is inherent in human beings, but difficult to be detected by the machines. therefore, a challenge of automatic humor recognition and analysis, based on data of different genre and language, has lately received a great amount of attention. specifically, several shared tasks were organized within the frames of evaluation workshops, i.e. semeval-2017 task 6 "learning a sense of humor", aimed to analyze humorous responses submitted to a comedy central tv show @midnight in english @xcite , haha task at ibereval 2018 with the sub-tasks of automatic detection and rating of humor in spanish tweets @xcite and etc. semeval-2020 task 7, however, presented a slightly different type of challenge, namely, an attempt to investigate how small edits can turn a text from non-funny to funny. sub-task 1 was to predict the mean funniness of the edited headline and sub-task 2 was intended to predict which of the two given edited versions of the original headline was the funnier or were they equal. all the data were in english, for more details about the dataset, see @xcite , about the task itself, see the task paper @xcite . the aim of this paper is two-fold: describing my approaches for both sub-tasks and analyzing the obtained results. in this study, i experimented with two models: for the first sub-task i used a relatively new one -deeppavlov logistic regression with bert english cased embeddings @xcite and for the second task i chose a well-known one -svm version for regression, namely svr @xcite , with word n-gram features. in sub-task 1 i obtained rmse equal to 0.65099 and in sub-task 2 accuracy was 0.32915. i also tried to use deeppavlov bert-based model during the post-evaluation period, which perfomed better than two previously-mentioned ones. my repository can be found on github https://github.com/aniton/so semeval-2020 news headlines . ## background in this section, i would briefly introduce the input and the output sets in order to form a better idea of the sub-tasks. ## system description deeppavlov @xcite as an open source framework based on tensorflow @xcite and keras @xcite has lately gained much attention among the developers. it contains three available models, namely, bert, keras and sklearn. for sub-task 1 i used sklearn logistic regression. the parameters for this system were chosen as following: @xmath0, @xmath1. deeppavlov also trained various word and sentence multilingual bert-based embeddings. as the embeddings features have a great impact on system performance, for the first sub-task i decided to employ tf-idf weighted 100-dimensional bert english cased embeddings. for sub-task 2 i chose rbf kernel support vector regression (svr). i used word n-grams as features, maximum word n-gram size was set to 5 and the parameter c to .1. this time i did not use any pre-trained word embeddings. as data preprocessing has a great impact on system performance, i applied it to the news headlines in both sub-tasks (see section 3.1). i also made use of 14 billion word iweb corpus (in particular, i used a list of the most frequent english bigrams) (see section 3.2). i did not use any external sources except for the mentioned one. ## results and analysis in both sub-tasks i tried to experience with traditional machine learning approaches, namely lr and svr. below, i present and discuss the results for each sub-task. ## conclusion and future directions in this paper i presented the contribution of so team to semeval-2020 task 7. during the evaluation period i experienced with traditional machine learning algorithms, namely lr and svr, in order to evaluate the funniness of the edited news headlines. despite the fact that i used pre-trained bert embeddings as input features to lr model, during the post-evaluation period i discovered that bert-based model achieves much better results in the given regression task. while analyzing the test dataset, i also noticed that edited news headlines had top grades if they contained names of some famous people/events regardless of the substitute. for instance, there were a great amount of headlines about trump, which were highly evaluated. thus, in the future i could try to make use of the different lists of popular political and media figures, since they are a target of admiration, jokes and hate. a list of representatives of top twitter profiles from different countries ( https://www.socialbakers.com/statistics/twitter/profiles ) could serve as an example of such lists. in @xcite this source was also used within the frames of semeval-2019 task 6 (offenseval), see @xcite , in order to predict whether the tweet was aggressive or not and if yes, who or what was the target of aggression.