LMML at S em E val-2020 Task 7: S iamese Transformers for Rating Humor in Edited News Headlines (2020)

Abstract: This paper contains a description of my solution to the problem statement of SemEval 2020: Assessing the Funniness of Edited News Headlines. I propose a Siamese Transformer based approach, coupled with a Global Attention mechanism that makes use of contextual embeddings and focus words, to generate important features that are fed to a 2 layer perceptron to rate the funniness of the edited headline. I detail various experiments to show the performance of the system. The proposed approach outperforms a baseline Bi-LSTM architecture and finished 5th (out of 49 teams) in sub-task 1 and 4th (out of 32 teams) in sub-task 2 of the competition and was the best non-ensemble model in both tasks.

## introduction machines that can recognize and understand humor can prove to be invaluable in applications like chat bots, personal digital assistants in order to make communication more fun and humane, story and script generation to provide comical relief or even in recommendation engines that can provide better recommendations to people on what netflix stand-up show they can watch next. at the end of the day we all live to laugh don't we? surprisingly enough, there hasn't been much work in the field of ai along these lines. the organizers of semeval-2020 task 7 @xcite released a new dataset in the english language and created a couple of sub tasks that can hopefully take us a step forward in creating machines that better understand humor. haha -humor analysis based on human annotation @xcite , @xcite started in 2018 was a similar task, where the dataset comprised of spanish tweets and participants were asked to classify the tweets as either a joke or not and also rate the jokes on a scale of 0-5. in their overview paper, the authors describe that teams that used transformer based models such as bert @xcite and ulmfit @xcite , along with techniques like slanted learning rates, domain specific language modeling etc. proved valuable. my approach takes inspiration from this and focuses on transformer models. semeval 2017 task-6 @xcite consisted of sub-tasks asking participants to rank the funniness of tweets that had a specific hashtag. they mention that some of the top teams used siamese networks @xcite , @xcite based approaches. @xcite introduce the concepts of local surprisal and global surprisal their work suggests that for a pun to be considered good, the pun-word must have high agreeableness in the local context of where it occurs in the sentence but a lower level of agreeableness in the global context of the entire sentence. i take inspiration from this idea as well. my hypothesis is that an edit in a sentence can be funny if the edited sentence continues to make sense and also have a bit of a twist. my model attempts to model the global agreeableness of the replaced and original word in the news headline. the remainder of the paper will be as follows. in section 2, i will briefly describe the task. section 3 will detail the system design and architecture. section 4 will state the implementation details. section 5 will cover details of all experiments, results and interesting findings. it will be followed by a small section containing my final thoughts. ## implementation details all experiments mentioned below were conducted making use of pytorch foot_1 , the huggingface foot_2 @xcite library was used for transformer architectures. spacy foot_3 for glove @xcite . batch size is fixed at 64, i use adam (kingma and ba, 2014) with a learning rate of 2e-5 as my optimizer. for all the experiments the best validation score and the test score of the model corresponding to the best validation score are reported. i use a linear warm-up scheduler @xcite with the warm-up period equal to 10% of the total number of steps. the model is trained for 5 epochs and the model with the best validation score is used at test time. a dropout of 0.1 is applied to all the transformer architectures and 0.3 to both the linear layers. i clip the norm of the gradients to 1.0. unless stated otherwise all of the transformer models were uncased apart from roberta, for roberta i use the roberta-base model. all the transformer models are pre-trained models and are not trained from scratch. for all experiments reported in this the random seed is fixed to be 12. the max sequence length is fixed to 50. i make my code publicly available foot_4 in the form of a jupyter notebook. the results are shown in the ## conclusion my final submissions to the competition for sub-task 1 and sub-task 2 resulted in a 5th and 4th place finish in the competition and was the best non-ensemble model in both tasks amongst the final submissions. the final submissions actually corresponded to a siamese bert based architecture, where @xmath0,c org ,s edit ,e edit ] which i obtained by performing a search on random seeds. i didn't experiment with the roberta model at that point of time after the release of the extra dataset. the submitted model achieved a validation score of 0.5186. and a test score of 0.5202. for sub-task 2 i observed that the model with the best accuracy did not necessarily need to be the same as the one that had the lowest rmse for sub-task 1, which is quite surprising. in sub-task 2 my official submission obtained an accuracy of 0.6465 on the validation set and 0.6468 on the test set. in conclusion, this paper presents a siamese transformer based approach, that makes use of features that center around the focus words and their impact against other tokens. from the experiments shown above its tough to conclude if the siamese architecture or if any of the features in particular are responsible for an improved performance of the model, it looks like just following the best practices of training transformer networks can yield very good results. in the future i would like to probe the model to better understand why it deems one sentence to be funnier than another, it would also be interesting to study if a model that can generate jokes can also grade how funny a joke is and vice versa.