J oke M eter at S em E val-2020 Task 7: Convolutional Humor (2020)

Abstract: This paper describes our system that was designed for Humor evaluation within the SemEval-2020 Task 7. The system is based on convolutional neural network architecture. We investigate the system on the official dataset, and we provide more insight to model itself to see how the learned inner features look.

## introduction this paper deals with estimating the humor of edited english news headlines @xcite . the illustration of tasks is in figure 1 . the original text sequence is given, which represents a title, with the annotated part that is edited along with the edit itself. our responsibility is to determine how funny this change is in the range from 0 to 3 (inclusive). this is called sub-task 1. we also participate in the sub-task 2, in that we should decide which from both given edits is the funnier one. for the second task, we used the approach of reusing the model from the first task as it is described in section 4. so we are focusing the description on sub-task 1. kushner to visit mexico following latest trump tirades therapist 2.8 sub-task 1 sub-task 2 kushner to visit mexico following latest trump tirades therapist oklahoma isn't working. can anyone ¡x this failing american state? ok which is funnier? mean grade edit official results were achieved with a convolutional neural networks (cnns) @xcite ), but we also tested numerous other approaches such as svm @xcite and pre-trained transformer model @xcite . the humor is a very subjective phenomenon, as can be seen from the inter-agreement on label annotation in sub-task 1 dataset 1 . the given data labels do not allow us to learn a sense of humor of a human annotator because the dataset does not specify from whom the grade comes. so, for example, if we have one annotator that likes dark humor and all the others not, we will be considering such a replacement as not humorous no meter if it is excellent dark humor or not. in other words, we may say that we are searching for some most common kind of humor. the dominant theory of humor is the incongruity theory @xcite . it says that we are finding humor in perceiving something unexpected (incongruous) that violates expectations that were set up by the joke. there are samples, in the provided dataset, that uses the incongruity to create humor. moreover, according to @xcite , we can see a positive influence of incongruity on systems results for the dataset. ## related work computational humor is usually divided into two groups: recognition and generation. humor generation focuses on creating the humor itself, so it can be a system that is able to tell a joke. recognition can be a binary (funny or not) classification task, but in our work, we also want to know how funny a sequence is by rating it with a grade in a given interval. @xcite , which introduced the dataset used in this work, tried to create models that can classify if a given edited title is funny or not, so they were training just binary classifiers, in contrast with our regression approach. many other works use this binary approach @xcite @xcite @xcite . they use various models, such as lstm @xcite , cnn, and bert @xcite , to deal with this task. our official results were achieved with a cnn model that is inspired by architecture presented in @xcite . their architecture is compact to its size (one layer cnn), so it provides the advantage of using less computational resources, in contrast with big models like bert. even with the small size, they were able to achieve promising results for the sentence classification task. also, such a small model allows us to gain better insight into what is going on underneath. ## data in this section, we would like to point out some interesting facts about the used data. we are focusing on the data for sub-task 1 because we used the model trained on the sub-task 1 for sub-task 2 (more in section 4). for each example, the dataset @xcite provides annotation in the form of grades (0,1,2 and 3) of humor assessment in sorted descending order and the mean of these grades. in most cases, there are five grades per dataset sample (sometimes more). in our work, we always use the first five grades. as can be seen from graphs in figure 2 , the dataset is imbalanced, and the high grades are rare. also, we investigated how the dataset is imbalanced when considering just single n-th grade and omit the others. though we can say, from the graph on the right side of the very same figure, that there is still imbalance, we can see that mainly the 2. and 3. positions seem to have smaller fluctuance among the number of samples per grade than the other positions. 0.0-0.3 0.3-0.6 0.6-0.9 0.9-1.2 1.2-1.5 1.5-1.8 1.8-2.1 2.1-2.4 2.4-2. we also did further analysis to determine prediction quality for the case when we would have an oracle classifier always predicting the grade on the n-th position. the results are in another thing that we decided to investigate is how the rmse score would look like if we always predict the same grade. the results are in ## system description the main inspiration for our model architecture (jokemeter) comes from zhang and wallace's (2015) work. the used model has cnn architecture illustrated in figure 3 . firstly the input sequence is assembled. the edit is inserted into the original title after the part that is being edited. additionally, we add a slash separator, and the whole original/edit location is delimited with two hashtags. in this way, we were able to add input for the model that has complete information about the original and the edited title. we also include tokens to mark the start and the end of a title. the reason behind these tokens is that we want to encode information into an n-gram, whether it is from the beginning/end of a title or not, and possibly make it easier for the model to learn the setup and punchline humor. kushner to @xmath0, 3, 4 and 8. each with two ¢lters. convolution 128 number of tokens in sequence zero padding leakyrelu (0.01) conv. features s l -3 s l -7 s l -2 s l -1 max pooling dropout (0.5) pooled features 3 2 1 0 (1 per ¢lter) concatenation select embeddins of the edit avg edit original tittle embedding of the edit 128 concatenation linear transformation softmax grades prob. we tokenized the input by albert @xcite pre-trained sentencepiece (kudo and richardson, 2018) tokenizer. each token is assigned a 128-dimensional embedding from 30 000 size vocabulary. right before the convolution, is added zero-padding of size one on both sides of the sequence. each sequence is 512 tokens long; shorter sequences are padded with unique padding tokens. we used four different convolution filter region sizes 2, 3, 4, and 8. for each size, we had two filters. these filters are followed with @xcite activation (negative slope is 0.01). we also experimented with a model (jokemeterboosted) that uses 2048 filters for each size, 2048-dimensional embeddings, and does not use the embedding of the edit. in the final part of our model, we do max pooling in order to get one feature per filter (8 in total). we concatenate the features to compose a vector, and then that vector is again concatenated with the edit embedding. the edit embedding is an average of embeddings of all tokens the edit is composed of. on the very end, we perform linear transformation followed by the softmax to get probabilities for each grade. with that configuration, we would not be doing a regression task, so at the test time we must do one final calculation that will transform these probabilities into a grade from the continuous interval [0,3]: feature 0 1 region size 2 3 4 8 2 3 4 8 σ 0 0.22 0 0.21 0 0.13 0.23 0 r s --0.43 --0.52 --0.6 -0.44 - where the g is a grade, s is a input sequence, and p i is the estimated probability for grade i. in the case of sub-task 2, we run the model for both titles separately, and in the end, we made the decision by comparing their estimated grades. ## experiments in this section, we describe the experiments we performed, not just with the model that is described in section 4. apart from models based on neural networks, we compiled several baselines: decision tree classifier (dtc) @xcite , svm, k-nn, and naive bayes classifier (nbc). also, we experimented with a model that uses transformer architecture (albert-base-v2). the neural models were implemented with pytorch @xcite . for the albert and the tokenizer, we used hugging face @xcite implementations, and for non-neural models, the implementations from the scikit-learn @xcite were used. we did two kinds of experiments for all models. the first kind uses all five grades for each sample (all-grades training) during the training. every sample is copied five times, and a grade is assigned to each of them, so we have five samples that have the same content, but each can have a different grade. on the other hand, the second kind of experiment uses only the 3rd grade (3-grade training), which has for oracle classifier the best score (see ## conclusion the system description was provided, and we compare the achieved results of the official model with several other models, including the baseline and the best team in the competition.