S em E val 2021 Task 7: H a H ackathon, Detecting and Rating Humor and Offense (2021)

Abstract: SemEval 2021 Task 7, HaHackathon, was the first shared task to combine the previously separate domains of humor detection and offense detection. We collected 10,000 texts from Twitter and the Kaggle Short Jokes dataset, and had each annotated for humor and offense by 20 annotators aged 18-70. Our subtasks were binary humor detection, prediction of humor and offense ratings, and a novel controversy task: to predict if the variance in the humor ratings was higher than a specific threshold. The subtasks attracted 36-58 submissions, with most of the participants choosing to use pre-trained language models. Many of the highest performing teams also implemented additional optimization techniques, including task-adaptive training and adversarial training. The results suggest that the participating systems are well suited to humor detection, but that humor controversy is a more challenging task. We discuss which models excel in this task, which auxiliary techniques boost their performance, and analyze the errors which were not captured by the best systems.

## introduction humor is a key component of many forms of communication, and so it is commanding an increasing amount of attention in the natural language processing (nlp) community @xcite @xcite . however, like much of figurative language processing, humor detection requires a different perspective on several traditional nlp tasks. for example, the problem of reducing lexical or syntactic ambiguity differs when ambiguity is key to some humor mechanisms. tackling these challenges has the potential to improve many downstream applications, such as content moderation and human-computer interaction @xcite . however, humor is a subjective phenomenon, which evokes varying degrees of funniness in its audience, while also provoking other reactions such as offense, in certain listeners. the perception of humor is known to vary along the lines of age, gender, personality and other factors @xcite @xcite . that humor can also evoke offense may be partly due to differences in acceptability judgements across demographic groups, and may also be in part due the use of humor to mask hateful or offensive content @xcite . @xcite expand on this by highlighting that it is common for societies to explore the link between humor and offense, free speech and respect. hahackathon is the first shared task to combine humor and offense detection, based on ratings from a wide variety of demographic groups. task participants were asked to detect if a text was humorous and to predict its average ratings for both humor and offense. we also introduce a novel humor controversy detection task, which represents the extent to which annotators agreed/disagreed with each other over the humor rating of a joke. a humorous text was labelled as controversial if the variance in the humor ratings was higher than the median humor rating variance in the training set. ## related work computational humor detection is a relatively established area of research. @xcite were one of the first to explore recognising wordplay with ngrams. @xcite experimented with 16,000 one-liners and 16,000 non-humorous texts, using a featuredriven approach. more recently, @xcite turned to online domains, by detecting humor on twitter with a view to improving downstream tasks such as sentiment analysis and opinion mining. workshops on humor detection have become more prominent with each shared task, and have attracted many new researchers to the field. se-meval 2017 @xcite featured hashtag wars, a humor task with a unique data annotation procedure. this task featured tweets that had been submitted in response to a number of comedic hashtags released by a comedy central program. the top-10 response tweets were selected by the show's producers and the winning tweet was selected by the show's audience. based on these labels, (top-10, winning tweet, and other) the sub-tasks required competitors to predict the labels, and to predict which text was funnier, given a pair tweets. the winning systems were split between feature-driven support vector machines (svms) and recurrent neural networks (rnns). the first spanish-language humor detection challenges were the haha tasks in 2018 @xcite and 2019 @xcite . these collected data from more than fifty different humorous twitter accounts, representing a wide variety of humor genres. the sub-tasks asked competitors to predict if a text was humorous, and to predict the average funniness score given to the humorous texts. in the first year, the top teams used evolutionary algorithms to optimize linear models like naive bayes, as well as bi-directional rnns. in the second year, the top teams started to use pre-trained language models (plms) like bert @xcite and ulmfit @xcite . most recently, @xcite generated data for their task by collecting news headlines, and asking annotators to make a micro-edit to the headline to render it funny. these edited headlines were rated for funniness by other annotators. the sub-tasks were to rank the funnier of two edits, and to predict the average funniness score given by the annotators. the winning teams used ensembles of various plms, and rnns. ## task description and evaluation we divided our tasks into four subtasks. task 1a: humor detection this was a binary classification task to detect, given a text, if the majority label assigned to it was humorous or not. this was evaluated using f-score for the humorous class and overall accuracy @xmath0: humor rating prediction this was a humor rating regression task. participants predicted the average rating given to texts from 0-5. texts which had not been labelled as humorous by our annotators did not have a humor rating, and predictions for these texts were not counted towards the final score by our scoring system. the metric for this task was root mean squared error (rmse). task 1c: humor controversy detection this task was also a binary classification task to predict whether the humor ratings given to the text showed it to be controversial or not. this was based on the variance in the ratings being higher or lower than the median variance in the training set humor ratings. this was also evaluated using f-score and accuracy. ## benchmark systems we created simple, linear benchmarks using sklearn @xcite for the classification tasks which consists of a naive bayes classifier with bag of words features. for the regression tasks, we used a support vector regressor with term-frequency inverse document frequency features. we also built a bert-base classification/regression model which was run for one epoch, with a batch size of 16 and a learning rate of 5e-5, for all sub-tasks. as this system out-performed the linear benchmarks on all sub-tasks, we refer to this as the baseline in the rest of the paper. ## conclusion we provided 10,000 texts annotated for humor and offense by a broad range of annotators. transformer models were a dominant approach to this task, with the exception of the humor controversy task, which proved to be difficult for most teams, and in which a simple, rule-based system achieved one of the top-3 scores. as multi-task learning setups proved more effective than single-task learning demonstrates, this that there is some correlation between humor and offense detection. it was also interesting to note which model adaptations were useful and which were not. finally, an analysis of the errors in humor analysis reveals some types of humor which may be captured inaccurately, even by the most powerful models.