D uluth NLP at S em E val-2021 Task 7: Fine-Tuning R o BERT a Model for Humor Detection and Offense Rating (2021)

Abstract: This paper presents the DuluthNLP submission to Task 7 of the SemEval 2021 competition on Detecting and Rating Humor and Offense. In it, we explain the approach used to train the model together with the process of fine-tuning our model in getting the results. We focus on humor detection, rating, and of-fense rating, representing three out of the four subtasks that were provided. We show that optimizing hyper-parameters for learning rate, batch size and number of epochs can increase the accuracy and F1 score for humor detection

## introduction humor detection poses a challenge to humans, not least because of the mix of irony, sarcasm, and puns which underlie humor. to understand the funniness of humor requires a certain grasp of context, culture and, for some, even country. if rating the funniness of humor is any challenge, ranking its offensiveness is even more so, especially when doing so requires an appreciation of the sensibilities of the humor target -whether race, religion, and/or gender -and, in most cases, context. it is little wonder humor detection has been central to nlp tasks in the past few years. the last few semevals have featured tasks focused exclusively on either detecting the funniness of humor @xcite @xcite or detecting offense @xcite . what semeval-2021 task 7 seeks to do is combine the detection of both humor and offense for a given corpus. our approach uses pretrained roberta model @xcite trained on a roberta classifier implemented by huggingface @xcite . the intuition here is that roberta model achieves state of the art performance for tasks requiring contextual information. our goal is to measure the effect of varying three hyperparameters -batch size, learning rate, epoch size -whilst maintaining default values for others. our results show that varying the three hyperparameters can increase performance for humor detection, humor ranking, and offense rating. the codebase for our participation of this semeval task is available on github foot_0 2 related work earlier works on humor detection and rating showed modest gains. with the advent of attention mechanism @xcite , though, and the transformer model a few years later @xcite , not only has interest in the nlp community on humor detection has soared, but performance on humor and offense detection has increased @xcite . this has been particularly so in the last few years, where humor detection and offense rating have been featured in some of semeval tasks. semeval-2019 task 6 on offense rating @xcite attracted 800 participants and 115 submissions, the interest prompting a second semeval-2020 task 12 the following year @xcite . around the same period, humor rating have attracted similar interest @xcite in semeval tasks. to the best of our knowledge, however, semeval-2021 task 7 @xcite is the first to measure humor and offense for a given task. most of the winning teams @xcite @xcite , for both humor and offense rating alike, implemented bert and its variants, including albert and roberta, in their model, an approach that tended to yield the best results. and more often than not, the teams exploit ensembles of bert, gpt-2, roberta and their variants @xcite , whilst others stick to a single pretrained model. our approach in this task is to use roberta model, an approach that will fine-tune a select number of hyperparameters and to measure the model performance for every change of hyperparameter set. ## system overview in this section, we review our system's adoption of the pretrained roberta model @xcite for semeval tasks. we also describe the bayesian hyperparameter optimization technique, which we used in our hyperparameter sweeps for selecting optimal values for learning rate, batch size, and epoch cycles. ## results in this section we present the results of our se-meval tasks and our analysis for each step.we present, in the first step, our baseline method and results. we then follow up with the results obtained evaluation phase; here, we analyse the impact of the manual sweep and finetuning on the pretrained roberta model on the subtasks. in the last step, and using the results of the codalab scores, we analyse the impact of hyperparameter sweeps on the scores during the post-evaluation phase. ## error analysis in an attempt to measure our model's predictions against the annotations by humans, we calculate the confusion matrix, comparing the predicted results with the values in the gold test for subtask 1a. figure 1, the confusion matrix during the evaluation phase, shows comparable results for false positives and false negatives. in figure 2, however, the number of false positives (34) are almost twice the number of false negatives, which is so because the train set has more label 1 data (4932) than label 0 data (3068), a slight difference that can lead to the false positives. again, the total number of true positives (596), almost twice the number of true negatives ( 351 ), shows the model is biazed towards positive labels, which will make it difficult to generalize. moreover, as shown in the higher number of accurate predictions for both the evaluation phase(926) and post-evaluation phase(947) shows our model is efficient in detecting humor. however, the rmse scores for humor ratingthat is task 1b, including even the improved rmse score of 0.58 during the post evaluation phasestill lags behind the rmse results for task 2a , the offense-rating subtask. and what this suggest is that our model performs better with offense rating than with humor rating. and that might suggest that the higher scores associated with task 1a are based on the model's ability to detect offense, which explains why most of the false positive texts also contain offensive content. 7 discussions and future works one major limitation of our approach was that the hyperparameter runs, during the evaluation phase, were experimented only on task 1a, a binary classification task, and the results applied on the two regression tasks to train our model, which may explain the subpar results for the task 1b and task 2a. however, the steps taken during the post-evaluation phase, by independently running the sweeps on each subtask, showed substantial increase in performance. in addition, the roberta model, as implemented by huggingface, is used as is, without any modification to either the classification layer on top of the roberta model or any of the pretrained layers. in the future, it will be worth pursuing how modifying either of the layers will impact on humor and offense detection. overall, however, our system shows that getting optimal values for learning rate, batch and epoch size can yield higher performance for humor detection. ## ethical considerations the training of roberta, along with other language models such as bert and its variants, has been shown to be costly, both for its effect on the environment and finance @xcite . again, the embeddings used for these language models tend to amplify racial, sexist, and homophobic biases. mindful of these tendencies, our model experiments included steps to minimize bias and reduce energy cost. what semeval-2021 task 7 @xcite intends to achieve is not only to rank humor but also to rate humor offensiveness, the first of any semeval task. to achieve this, the dataset contains as much as humor as hate, which covers racial slurs, gender bias, trans/homophobic comments, etc. knowledge of what ranks as offensive in humorous text can help our system moderate humorous content. to ensure that dataset used for training and development do not over-represent hegemonic viewpoints, @xcite , organizers for the semeval-2021 task 7, employed annotators from disparate backgrounds, in age, gender, political views, to ensure that humor ratings and rankings, a subjective process, reflected the varied viewpoints. annotators were limited to english speakers, however, which implies that the system's ability to detect and identify humor is largely reflect views inherent in the english language. training and testing were carried out on 1 v100 gpus with less than 16gb of memory, a step taken to ensure minimal, if any, impact on the environment. the model, however, is prone to classifying offensive content as humorous, which may suggest that applications based on our model will be more likely to rate as humorous any content that might be deemed offensive.