Crossing the Line: Where do Demographic Variables Fit into Humor Detection? (2020)

Abstract: Recent humor classification shared tasks have struggled with two issues: either the data comprises a highly constrained genre of humor which does not broadly represent humor, or the data is so indiscriminate that the inter-annotator agreement on its humor content is drastically low. These tasks typically average over all annotators’ judgments, in spite of the fact that humor is a highly subjective phenomenon. We argue that demographic factors influence whether a text is perceived as humorous or not. We propose the addition of demographic information about the humor annotators in order to bin ratings more sensibly. We also suggest the addition of an ‘offensive’ label to distinguish between different generations, in terms of humor. This would allow for more nuanced shared tasks and could lead to better performance on downstream tasks, such as content moderation.

## introduction interest in computational humor (ch) is flourishing, and since 2017, the proliferation of shared humor detection tasks in nlp has attracted new researchers to the field. however, leading researchers in ch have bemoaned the fact that nlp's contribution is not always informed by the long and interdisciplinary history of humor research @xcite . this may result in the creation of humor detection systems which produce excellent evaluation results, but which may not scale to other humor datasets, improve downstream tasks like content moderation, or contribute to our understanding of humor. a central issue is the conception of humor classification tasks as humor-or-not, similar to image classification's view of an image as dog-or-not. however, while one can be an expert in whether or not an image depicts a dog, and this is stable within and between cultures, humor is more nuanced than that. unlike image classification: ## previous work semeval 2017 posed two humor detection tasks. task 7 @xcite covered puns, which we do not include here as the identification/interpretation of puns is less ambiguous than other forms of humor, except in the case that the audience does not possess the tacit linguistic knowledge required to understand them @xcite . ## methodology the datasets featured in both semeval tasks had tight constraints on the genre of humor involved. this led to high inter-annotator reliability, but may not generalize well to other forms of humor. the spanish tasks featured no such constraints, however, there was extremely low inter-annotator agreement, suggesting that the dataset is noisy, and that a system which is built on this may also fail to generalize. this proposal aims to include a wide range of genres, and to increase the reliability of the annotations by collecting information on wellknown latent variables in humor appreciationthe demographic characteristics of the humor audience/annotators. this will allow for more nuanced tasks, as an alternative to simple humor-or-not definitions. ## contribution to computational humor in line with ch research, we affirm that humor is a moving target in terms of differing interpretations between demographic groups and across the lifetime. our dataset will be the first to model the reception of a wide variety of humor genres from twitter, presented to users of different demographics. it will also be, to the best of our knowledge, the first ch dataset to take into account the ratings of non-binary annotators. in line with @xcite , we aim to use clustering methods on the humor and/or offensive texts to determine themes that evoke these classes for different groups. we also aim to explore whether theories of humor, such as surprisal, superiority and incongruity are equally appreciated among different groups. ## conclusion humor detection and rating is a multi-faceted problem. we hope that the inclusion of demographic information will shift the state of the art away from objective classification, towards a more subjective approach. future qualitative work could also suggest further variables whose inclusion would enhance our knowledge of humor perception. this could set a new standard for shared tasks which aim to model humor in future, and could outline a methodology that can be replicated with other cultures and languages.