“So You Think You’re Funny?”: Rating the Humour Quotient in Standup Comedy (2021)

Abstract: Computational Humour (CH) has attracted the interest of Natural Language Processing and Computational Linguistics communities. Creating datasets for automatic measurement of humour quotient is difficult due to multiple possible interpretations of the content. In this work, we create a multi-modal humour-annotated dataset (~40 hours) using stand-up comedy clips. We devise a novel scoring mechanism to annotate the training data with a humour quotient score using the audience’s laughter. The normalized duration (laughter duration divided by the clip duration) of laughter in each clip is used to compute this humour coefficient score on a five-point scale (0-4). This method of scoring is validated by comparing with manually annotated scores, wherein a quadratic weighted kappa of 0.6 is obtained. We use this dataset to train a model that provides a ‘funniness’ score, on a five-point scale, given the audio and its corresponding text. We compare various neural language models for the task of humour-rating and achieve an accuracy of 0.813 in terms of Quadratic Weighted Kappa (QWK). Our ‘Open Mic’ dataset is released for further research along with the code.

## introduction humour is one of the most important lubricants of communication between people. humour is subjective and, at times, also requires cultural knowledge as humour is often dependent on stereotypes in a culture or a country. at times, even cultural appropriation is used to convey humour, which can be offensive to minority cultures 1 @xcite . the factors listed above, along with the underlying subjectivity in humour render the task of rating humour, difficult for machines @xcite . the task of humour classification suffers due to this subjectivity and the lack of datasets that rate the "funniness" of content. in this paper, we propose rating humour on a scale of zero to four. we create the first multimodal dataset foot_0 using standup comedy clips and compute the humour quotient of each clip using the audience laughter. the validity of our scoring criteria is verified by finding the overall agreement between human annotation and automated scores. we use the audio and text-based signals to process this multi-modal data to generate 'humour ratings'. since humour annotation is subjective, even the data annotated by humans might not provide an objective measure. we reduce this subjectivity by taking laughter feedback from a larger audience. to the best of our knowledge, no previous literature has proposed an automatically humour-rated multimodal dataset and used it in ml model-building to automatically obtain the humor score. standup comedy is an art form where the delivery of humour has a much larger context, and there are multiple jokes and multiple related punchlines in the same story. the resulting laughter from the audience depends on various factors, including the understanding of the context, delivery, and tonality of the comic. standup comedy seems to be an ideal choice for a humour rating dataset as it inherently contains some feedback in terms of the audience laughter. we believe a smaller context window restricts computational models, but we know this is not the case for the human audience. hence, our approach utilises live audience laughter as a measure to rate the humour quotient in the data created. we also believe that such an approach can generate insights into what aspects of stories and their delivery make them funny. our humour rating model is partly inspired by the character "tars" from the movie "interstellar", which generates funny responses based on adjustable humour setting @xcite ). an essential step in developing such a machine that can adjust its "funniness" is to create a model that can recognize and rate the "funniness" of a joke. with this work, we aim to release a dataset that can help researchers shed light on the humour quotient of a particular text. the key contributions of this paper are: (a) creation and public release of an automatically rated multi-modal dataset based on english standup comedy clips and (b) manual evaluation of this dataset along with humour-rating quotient defined on a likert-scale @xcite . ## related work most of the previous work on computational humour has been towards the detection of humour. smaller joke formats like one-liners which have just a single line of context, have been used @xcite . language models like bert are used for generating sentence embeddings, which have been shown to outperform other architectures in humour detection on short texts (annamoradnejad, 2020). since humour depends on how the speaker's voice changes, the audio features, and language features have been used as inputs for machine learning models for humour detection. @xcite use audio and language features to detect humour in the big bang theory sitcom dialogues. @xcite passed audio and language features from a conversation dataset into an rnn to create a chatbot that can detect and respond to humour. @xcite built a multi-modal dataset that uses text, audio, and video inputs for humour detection. there are existing datasets that rate the humour in tweets and reddit posts, with the help of human annotators @xcite @xcite . creating human-annotated datasets is costly in terms of both time and money and has been one of the noted issues for creating humour datasets. @xcite used time-aligned user comments for generating automated humour labels for multimodal humour identification tasks and found good agreement with manually annotated data. however, none of the previously existing datasets are created with standup comedy clips. we present the first multi-modal dataset that uses a non-binary rating system. we use standup comedy clips which makes our dataset scalable and diverse. the dataset is novel in terms of the use of long contextual jokes (∼ 2 mins) and audience laughter which helps annotate the funniness in each clip in an automated manner. ## dataset acquisition and pre-processing in this section, we describe the creation of our multi-modal dataset and the manual evaluation performed with the help of human annotators. we gather 36 english language standup comedy shows from 32 comedians available on the web, where the length of each original clip is ∼ 1 hour. we further segment them manually into 927 ∼ 2 minute long clips. the standups are chosen based on the clarity of the audience feedback laughter. we choose comics from diverse categories of gender, nationality, and culture to ensure representation and reduce bias. while segmenting them, we ignore the clips, which results in laughter on interaction with the audience/personal jokes. we also create text files with the transcript for each audio clip from multiple online sources @xcite . we collect data for "unfunny" samples by gathering ted talk audio clips with similar speech delivery modes like standup comedy. we also segment them into 128 ∼ 2 minute audio clips and create text files of their transcript foot_1 . clips were manually trimmed from the complete audio such that the entire context for the joke is available within the clip. this results in the overall set of ∼ 2 minute clips described above. finally, we acquire 519 ∼ 2 minute audio clips and corresponding transcript text files in our dataset. the train-test split is chosen to be 70-30. ## experiment setup and methodology in this section, we describe the features used for the humour rating prediction task along with the additional pre-processing in detail. ## results in bert-derived language models showed considerable improvement from baseline performance. roberta large outperforms all the other language models and shows an improvement of 12% points over the baseline glove score. since roberta is pre-trained on datasets that contain text in a storylike format similar to standup comedy text @xcite , roberta large can be seen performing better than all the other textual features. analysing the confusion matrix of these models shows that roberta large and bert large can distinguish different levels of humourousness quite well. they show the highest accuracy in identifying the nonfunny clips. distilbert could not perform as well as bert large because humour needs better quality text embeddings to understand the full context, which distilbert cannot provide due to the lower number of parameters in the model. larger models with embedding dimensions of 1024 (bert large , roberta large ) and 2048 (xlm) performed better than smaller models. a larger neural network would need a dataset of significant size to train, which also shows that our dataset is reasonably sized. for bert base , when we increased the bi-lstms in the initial layer from 256 to 512, we see a slight improvement in the quadratic weighted kappa value which shows that larger embeddings need a bigger neural network to classify accurately. we further probe our best-performing model with an ablation test and observe that audio-based features (0.66 qwk) outperform text-based features (0.48 qwk). this contradicts what was observed by @xcite as humour in standup often depends on the tonality and the wellenunciated punchlines. ## discussion analysis of the predicted ratings show that our model can identify non-funny clips and most funny clips with very high accuracy. in cases of error in assigning ratings to the intermediate funny clips, the assigned ratings are not off by more than one rating point, for e.g., a clip rated 3 is assigned a rating of 2 or 4. this error should not be considered as a failure of our model since assigning a precise funniness rating in a definite way to intermediate funny clips is hard even for humans. so it is reasonable to expect our model to commit similar errors in assigning the ratings as a human would. in the individual confusion matrices obtained for both feature sets, we observe the maximum incorrect predictions among the classes 2/3 and 3/4. we correlate these results with the human annotation and observe that even human annotators differ mostly in these two classes. all our annotators observed that clips with a moderate amount of laughter could be rated either as 2 or as 3, since such annotations are difficult to be discerned to a particular class. additionally, we observe that there are only 16/1055 cases where none of the ratings of the three human annotators match with each other. out of which, only one rating differs (4, 1, 2) with a difference @xmath0. in the other 15 ratings, the difference between the highest and lowest human ratings @xmath1.g., 4-2-3). there are around ∼400 cases where 2 annotators fully agree. the rest of the ∼600 ratings are where all three annotators fully agree in their ratings for each clip shown to them. as we evaluate the clips misclassified by our model, we observe that 1) sarcastic and ironic statements generate human laughter, but our model does not detect it, 2) a certain kind of jokes which are morbid also categorized as "dark humour" is consistently classified with lower scores, whereas there is a lot of human laughter generated during such jokes, 3) subtle comparisons, for example, the usage of internet to smoking where the comedian tries to imply that both are harmful to health; are classified as "mildly funny (1 or 2)" by our model. we further evaluate clips with human annotation score @xmath2. despite providing detailed guidelines which required our annotators to focus only on audience laughter, they could have possibly focused on the content. due to this subjectivity, we believe that our annotators may have misclassified a few clips. we trace the reasons to 1) countryspecificity, thereby leading to less comprehension by the annotator, or 2) insensitivity towards the feelings of females, or 3) bias against a country/race which again leads to the diminished absorption of the joke. this observation validates our initial discussion on the subjectivity of humorous content, along with the observation that annotator a (female) has consistently scored such clips lower than annotator b and c (males). however, we also observed that the audience laughter in such clips is more consistent with the scores provided by annotators b and c. ## conclusion and future work we propose a novel scoring mechanism to show that humour rating can be automated using audience laughter, which concurs well with the humour perception of humans. we create a multi-model (audio & text) dataset for the task of humour rating. with the help of three human annotators, we manually evaluate our scoring mechanism and show a substantial agreement in terms of qwk. our evaluation shows that our scoring mechanism can be emulated with the help of pre-existing language models and traditional audio features. our neural network-based experiments show that the output obtained using various language models like roberta show an agreement with our scoring mechanism. despite the inherent subjectivity in humour and its different perceptions among humans, we propose a method to rate humour and release this dataset under the cc-by-sa-nc 4.0 license for further research. in the future, we would like to evaluate this scoring mechanism with the help of more human annotators. we aim to extend the dataset with the help of more standup comedy clips. further experiments can be conducted to compare the contribution of audio, video and text features with a more detailed analysis. we would also like to perform experiments by including more audio features like line spectral frequencies, zero-crossing rate, and delta coefficients. with the release of this dataset, we hope that research in computational humour can be taken further.