LRG at S em E val-2020 Task 7: Assessing the Ability of BERT and Derivative Models to Perform Short-Edits Based Humor Grading (2020)

Abstract: In this paper, we assess the ability of BERT and its derivative models (RoBERTa, DistilBERT, and ALBERT) for short-edits based humor grading. We test these models for humor grading and classification tasks on the Humicroedit and the FunLines dataset. We perform extensive experiments with these models to test their language modeling and generalization abilities via zero-shot inference and cross-dataset inference based approaches. Further, we also inspect the role of self-attention layers in humor-grading by performing a qualitative analysis over the self-attention weights from the final layer of the trained BERT model. Our experiments show that all the pre-trained BERT derivative models show significant generalization capabilities for humor-grading related tasks.

## introduction humor is a communicative ability that produces a sense of laughter by imparting an amusing tone in conversations. if we can teach our models how to generate, detect, and grade humor in language, it would be a big step ahead in natural language processing. nearly all the existing datasets for humor-related tasks such as "pun of the day" @xcite and "16000 one liners" @xcite focus on humor-detection in a single piece of text as a binary classification task. previous tasks on humor-grading consist of datasets comprising of originally humorous texts @xcite . semeval-2020 task-7 @xcite focuses on short-edits based humor-grading. the dataset used for this shared-task was introduced by @xcite . it consists of short-edits applied to a piece of text (news-headlines), which changes it from non-funny to potentially funny. this shared-task consists of two independent sub-tasks based on this dataset. the first sub-task comprises of a regression task, where given the original and the edited news-headline, one must predict the funniness of the edited headline (humor-grading) on a scale of [0,3], where '0' conveys 'least funny' and '3' conveys 'most funny'. the second sub-task comprises of a classification task, where given two edited versions of the same news-headline, the model must predict the funnier one. in this work, we propose using bert @xcite and its derivative models (collectively referred to as "bert models" further throughout the paper) for short-edits based humor-grading. we mainly follow two approaches: in the first one we use pre-trained bert models with a final regression layer to get a humor-grade for each edited news-headline, and for the second one we use masked-word language-modeling to fine-tune the pre-trained bert models on the task dataset, which are further trained for the humor-grade regression task. we compare the results of these two approaches for each sub-task. we also test the generalization capabilities of the bert models for this task by using the funlines dataset @xcite , which is provided as an additional dataset for this shared-task. the code for this work is made publicly available as a github repository. in this work, we test the ability of the bert models to perform humor-grading tasks on the humicroedit dataset (official task dataset) and funlines dataset (additional dataset). both of these are english shortedits based humor-grading datasets. the humicroedit dataset consists of 15,095 edited headlines, and the funlines dataset consists of 8,248 edited headlines. both the datasets share the same format. ## methodology since this is a short-edits based humor-grading dataset, the original text context is also crucial while grading the humor. to incorporate the original text context, we use a two-sentence input based approach with the bert models, as shown in figure 1 . we concatenate the original and the edited news-headlines (separated and padded with special tokens of the respective bert model) and feed it to the models in a tokenized format. further, we inspect the effectiveness of masked language model fine-tuning based pre-training of the bert models. inspired from @xcite , we initialize the bert models with their respective fine-tuned language models weights. the implementation details of the models are described in section 4. we follow a masked word prediction based language modeling approach to fine-tune the bert language models on the entire dataset. since the datasets do not come from an open-domain source and are limited to news headlines, all the words in the text are masked for prediction. we use only the edited humorous texts for language modeling pre-training. all the layers of the language models are trained with a maximum sequence length of 256 tokens for masked word prediction. the trained weights from these fine-tuned language models are used to initialize the model weights for sub-task 1. for sub-task 1, humor grading is considered a regression task over the humor grades of the edited news-headlines. we experiment with two types of weight-initializations for this task: weights from the language model fine-tuned on the dataset and the original pre-trained bert model weights. we fine-tune the bert models for the regression task by appending a fully-connected layer. this layer takes in the pooled output embedding from the bert models and returns a humor grade value. the models are trained with the mean-squared error between the predicted and ground-truth humor grades as an objective function. the model weights are optimized for this objective function by adam optimizer. for sub-task 2, we directly use the sub-task 1 models for a zero-shot inference over the pairs of edited news-headlines, as shown in figure 1 . after getting the humor grades for both the edits of a news-headline, we label the one with a higher grade value as the more humorous one. ## experimental setup for all our experiments, we use the pytorch foot_1 implementations of the smallest base variants of the pre-trained bert models by hugging face foot_2 @xcite . the data distribution across the training-development-test splits used for our experiments is shown in ## discussion for the final submission, we use bert with masked word language model fine-tuning for both the sub-tasks. on the final test-set leaderboard, our system ranks 14 th for sub-task 1, with a rmse loss of 0.53318 and 13 th for sub-task 2, with an accuracy of 0.62177. further, in the post-evaluation experiments, we analyze the performance of all the bert models on both the humicroedit and the funlines dataset. overall, all of the bert models perform better on the humicroedit dataset, which might be attributed to its relatively larger size as compared to the funlines dataset. bert models use the [cls] token as the pooled output representation for the entire input sequence. the [cls] token is ultimately used for the humor-grading tasks in our experiments. in order to interpret the bert models' decisions, we analyze the multi-head self-attention weights of the [cls] token from the final layer of the trained bert model, as shown in figure 2 . taking a closer look at the attentionvisualization over these attention weights reveals that the model learns to assign more attention to certain parts of the text, which play a role in imparting funniness to the text as a whole. in general, some attention heads (2 nd and 12 th head) learn to assign more attention to the edited words as compared to other words in the text. the attention from the 2 nd and 12 th head is represented by orange-colored connections in figure 2 . figure 2 -1 and figure 2 -3 represent the attention visualization of some of the best predictions from the humicroedit and funlines dataset, respectively. in these cases, we observe that the [cls] token firmly attends the edited words by the 2 nd and 12 th attention head. on the other hand, figure 2 -2 and figure 2 -4 represent some of the worst predictions from the humicroedit and funlines dataset, respectively. here, the [cls] token divides its attention relatively more uniformly across the entire input sequence, with relatively lesser attention from the 2 nd and 12 th attention head. the same trend is observed across most of the samples from both of the datasets for all the bert models. this reveals the importance of self-attention in the decision making of the bert models for short-edits based humor grading. ## conclusion in this work, we tested the ability of bert and its derivative models for short-edits based humorgrading tasks. our experiments showed that the bert models perform better with their original pretrained weights, and there is no need for language model fine-tuning on the humorous texts. the tests also revealed that the bert models show good generalization capabilities for humor-grading tasks by performing significantly well with zero-shot inference across the sub-tasks and cross-dataset inferences. the qualitative analysis over the self-attention weights of the bert model revealed the attentive bias shown by some of the attention heads towards the edited parts of the text. it also showed the importance of self-attention in the decision making ability of the bert models. we plan to extend our work by testing autoregressive and generative transformer models for humorgrading related tasks. one can also experiment with different language modeling techniques and different input formats for the short-edits based humor-grading tasks. to further test the abilities of the bert models for humor-grading, we plan to test our approach on other humor-related datasets such as the "pun of the day" and the" 16000 one liners" datasets.