When to Laugh and How Hard? A Multimodal Approach to Detecting Humor and Its Intensity (2022)

Abstract: Prerecorded laughter accompanying dialog in comedy TV shows encourages the audience to laugh by clearly marking humorous moments in the show. We present an approach for automatically detecting humor in the Friends TV show using multimodal data. Our model is capable of recognizing whether an utterance is humorous or not and assess the intensity of it. We use the prerecorded laughter in the show as annotation as it marks humor and the length of the audience’s laughter tells us how funny a given joke is. We evaluate the model on episodes the model has not been exposed to during the training phase. Our results show that the model is capable of correctly detecting whether an utterance is humorous 78% of the time and how long the audience’s laughter reaction should last with a mean absolute error of 600 milliseconds.

## introduction humor is a topic that has piqued interest of the computational creativity research community over the years. there are numerous systems that can generate humor using a variety of different methods @xcite @xcite . but just as important as it is to research generation from the computational creativity perspective, it is to study automatic assessment of humor. the role of humor is an important one for us humans as it has it's own social function @xcite . it helps us talk about difficult topics @xcite and relieves tension @xcite ). laughter has a role in building relationships @xcite and it has a positive effect on brain chemistry (gonot-schoupinsky and @xcite . humor and laughter are therefore an integral part of who we are as a species. humor is a phenomenon that requires surprise and coherence (see @xcite ); or incongruity and its resolution in other terms @xcite . however, what is surprising or incongruous, depends heavily on the context where humour is presented. quite indeed, something intended as a joke can be seen as a severe insult just by a change in context. therefore, we believe that a multimodal approach needs to be researched when when assessing humor automatically; something that thus far has been researched by focusing on the textual modality alone. annotated multimodal datasets are scarce, but an access to such a dataset is crucial for any computational attempt on humor detection and assessment. for this reason, we embrace a clever approach: we use episodes of the beloved american sitcom friends as our data source. the tv show has prerecorded audience laughter which provides us with an ultimate source for annotations. every time the audience laughs, we know that there was something humorous immediately before the laughter. a lack of laughter indicates no humor. in addition to this, the audience can laugh for a short or a long time, which allows us to gather data on how funny a given joke was. we propose a pipeline consisting of two neural models. one of them detects whether a sentence is humorous or not and the other rates how funny the sentence is in case it was deemed humorous. we evaluate our models on episodes of friends that were not used during training or validation. our work is a first step towards multimodal humor detection and assessment. we have also established several important data processing practices that make it possible for future research to automatically gather annotated multimodal data in a similar fashion as we did in this paper. the main contributions of our paper are as follows: ## related work there is a an extensive body of literature that focuses on humor generation @xcite @xcite and also a growing body of work that deals with multimodality in natural language understanding @xcite @xcite . however, in this section, we focus on some of the recent papers that deal with humor detection and analysis. in a recent work @xcite , the authors study humor detection in a context where there is a setup and a punchline. they use a gpt-2 model @xcite to assess uncertainty and surprise to determine if a setup-punchline pair is a joke or not. similar setup and punchline based approaches for humor detection have been widely studied in the past using different computational methodologies @xcite @xcite . however, such methods are very different from our approach as we do not expect our model to receive a setup and a punchline that are explicitly marked in the data. sentiment analysis has been used in humor detection @xcite . as many of the existing approaches, their approach also operates on a setup and a punchline. the authors found that sentiment conflict and transition between the setup and the punchline are useful in humor detection. the authors use an existing discourse parser (feng and hirst, 2012) combined with texblob foot_0 sentiment analysis and heuristic rules to detect humor. apart from humor detection, there is a line of work on assessing the humor value of a joke @xcite . the authors propose a model that does not detect humor, as it expects jokes as its input, but instead, the model rates how humorous a given input joke is. their model also expects a setup and a punchline division in the data. the authors train a transformer @xcite based model for the task. humorous headlines have been automatically ranked based on how funny they are @xcite . the authors use ridge regression and an lstm (long short-term memory) model with manually engineered features such as whether donald trump and his hair have been mentioned and the length of the headline. the authors conclude that a language model is simply not enough for assessing humor, but a wider context is needed to help the model understand humor. as we can see, most of the previous approaches on humor detection and assessment do either or. the models can either tell whether something is funny or not, or rate how funny a given joke is. in addition, many models seem to expect a clear division into a setup and a punchline, which makes it impossible to use them to detect humor in free formed speech or text. in addition, there are many types of humor, for instance sarcastic one (hämäläinen, 2016), that does not require an explicitly uttered setup in natural language, but rather the setup of the joke can be deduced from the context itself. our approach tries to tackle these shortcomings in the current state of humor detection. ## humor theories humor is an integral part of our social lives as humans and because of that, it has provoked the interest of many scholars in the past. some of the early theories of humor @xcite saw it as a question of superiority, where a superior person laughs at the misery of those inferior to them. while this explanation might be valid in the context of schadenfreude, more modern takes on humor theory reject it as it cannot explain humor as a whole. for @xcite , humor is a part of creativity together with other two components: discovery and art. what is seen as characteristic to humor, in his view, in comparison to the other two constituents of creativity, is that its emotional mood is of an aggressive nature. humor comes from bisociation which is a collision of two frames of reference happening in a comic way. @xcite presents a theory that is quite similar to the previously described one in the sense that in order for a linguistic expression to be humorous, it has to be compatible with two different scripts. the different scripts have to somehow oppose one another, for example in the sense that one script is a real situation and the other is hypothetical. @xcite sees humor to be consisting of six hierarchical resources of knowledge: language, narrative strategy, target, situation, logical mechanism and script opposition. similarly to the previous theories, the incongruity of two possible interpretations is considered to be an important aspect of humor. an interesting notion that sets this theory apart from others is that of target. according to the authors, it is not uncommon for a joke to have a target, such as an important political person or an ethnic group, to be made fun of. two requirements have been suggested in the past as components of humor in jokes: surprise and coherence (see @xcite ). a joke will then consist of a surprising element that will need to be coherent in the context of the joke. this is similar to having two incongruous scripts being simultaneously possible. @xcite discusses that the theories of @xcite and @xcite entail that people are forced into resolution of humor. he argues that humor should not be seen as resolution of incompatible scripts, but rather as a collaboration, where the listener willingly accepts the humorous interpretation of the joke. moreover, he argues that while incongruity contributes to humor, it does not alone constitute it. ## data construction we focus on the sitcom tv show friends. the show is one of the most popular american sitcoms ever produced and it aired from 1994 to 2004. our data consists of the entire show, i.e. 10 seasons and 236 episodes each of a duration around 20 minutes. all episodes had english well-aligned subtitles that correspond to what is said in the audio track of each episode. we have randomly sampled an episode from each season to assess the quality of the subtitles, and found no major errors or clear delays. while there are some multimodal annotated data for sarcasm detection @xcite , multimodal annotated data of humor in more general terms is very scarce. while several textual humor datasets exist @xcite , they are not suitable for our need as we are interested in multimodality. this is mainly due to the great subjectivity of what humans deem to be funny, and to the high amount of work and funds needed to manually annotate a dataset. to overcome this obstacle, we embrace an automatic approach for annotating humor in the tv show by recognizing laughter in the audience as described in the following subsection. ## assessing humor we present a pipeline of two neural models for assessing humor. the goal of the first model is to identify whether an utterance at the end of the segment is humorous or not given the rest of the segment as context, whereas the second model predicts the intensity of the humor once it has been detected. ## results and evaluation we run both, the text only and multimodally trained models, for humor detection on the test split. their performance is assessed using precision, recall, f1 and accuracy scores, which are given in the results indicate that the multimodal model clearly outperformed the text only model, by a 16% increase in detection accuracy. this suggests that audio cues were helpful in recognizing humor. for instance, sarcasm and irony are sometimes marked with clear intonations and tones. both of these phenomena are frequently used in sitcoms for humorous effect, which would aid the model in distinguishing when "yeah, right" is meant as a sincere confirmation or as a sarcastic remark for humorous effect. for humans and it is no surprise that computational models would struggle. as these models are tested on entirely novel contexts (i.e., new discourses that are not covered during the training phase), the performance achieved by them is impressive. to test the model for predicting laughter intensity, we compute the mean absolute error between the predicted intensity and the intensity of the laughter in the dataset. the average mean absolute error was only around 600 milliseconds. this means that the model can predict how long the audience will laugh after a given joke rather accurately, given that there is some flexibility in the duration of the laughter as it is not an absolute measurement of the humor of a joke. with these results, we can say that the model has learned to predict the intensity of humor in a joke well, given that for jokes provoking less laughter, the model predicts a short laughter, and for jokes provoking a lot of laughter, the model predicts a long laughter. even though the model is not quite accurate in knowing the exact duration of the laughter. ## discussion in this section, we discuss our work and the results obtained. as mentioned earlier, one crucial aspect for understanding humor is the context. we have defined the context in our work as a fixed 10 seconds but, in reality, the context might be wider than that. as another attempt to define context, we have split the tv show based on changes in the visual scene. this approach however was not very practical for our needs as it would mistakenly cut scenes based on changes in the camera angel. a potential solution to overcome the problem of fixed contexts is to observe semantic changes for discovering consecutive scenes that share the same topic. we keep this for future work. sitcom tv shows are a great resource for computational humor as there is a multitude of humor forms that they present. for this reason, it is challenging for a neural model to capture all humor forms (e.g., irony, sarcasm, satire, exaggeration, personification, silliness, pun and parody), whether they are expressed verbally or visually. our multimodal model develops its own understanding of what is funny based on the textual and audial features embedded by bert and hubert. thus, it is incapable of explaining the humorousness it perceives. in other words, the model can say whether something is funny or not, but it cannot say why something is funny. a future direction would be to break down these types of humor and feed them collectively to the model, which would enable it to recognize humor from different aesthetics and pinpoint the humor type. from the error analysis, it is evident that including video modality can help the model in understanding humor by situating what is said to the context presented in the video. however, it is not that clear to know which features would be needed. in figure 3 , we could see that it is the action that makes the humor more interpretable, whereas in figure 4 we saw that the facial expression was revealing of humor. if the humor occurs only in the visual modality as seen early on in figure 2 , it is in the silliness of how joey looks while wearing all those clothes. needless to say, a simple automatically extracted vector representation of the video such as video2vec @xcite is not capable of capturing all these different nuances expressed in the video. it might very well be that an entire tv show does not have enough data for the model to learn to use the video in a meaningful way. we did try to include video features in our models by obtaining textual descriptions of what is happening in a scene by using an existing state-ofthe-art video captioning model @xcite . the idea was that a captioning model could extract relevant information from the video into a textual format that could then be understood by a language model such as bert. in practice, this turned out to be an impossible task with the current models and the image datasets they were trained on. figure 5 shows the poor performance the model had on images from friends. in our experiments, we did not see a single correctly captioned image from the tv show while test data from the dataset the model was trained on produced decent results. we have also seen that the 10 second contextual window is not always enough to resolve whether something is humorous or not. the issue of increasing the context is that more context will also increase the amount of irrelevant context. we do not believe that simply increasing the context from 10 to 20 seconds, for example, is the most optimal way to go about it because some jokes require more context whereas others do not. perhaps the best way would be to introduce a third model to the pipeline that is trained to determine how much context is needed by identifying how far back in the dialog we can go and still stay in the same topic. a change in topic would indicate that the context goes too far away from what is needed to interpret the joke. ## conclusions this work has shown the first steps towards humor interpretation in a multimodal data. unlike the existing methods, our method does not rely on implicitly marked setup and punchline but can rather detect humor even in cases where the setup of the joke was not made explicit in the text. we have also trained a model that can rank the intensity of humor based on how long the audience laughed. the results are promising and our current research has a lot of potential for future research especially in studying how to deal with video and how much context one should include. the trained models can be incorporated in other computational creativity models for generating humor. for instance, a system for generating humorous transcripts could utilize our models for determining whether the plot is funny and which version of it would make the audience laugh the most. an interesting application of our approach would be to pipeline it with a laughter generator. the models presented in this paper could be used to identify where laughter should be inserted and with what intensity in a comedy show that does not have prerecorded laughter. this could save time in postediting if used in a professional setting. because friends has been translated into multiple languages, this makes it possible to rerun our experiments in different languages with a minimal effort. it also creates an ultimate test-bed for multilingual models where we can test whether a model learning humor from the data in all languages can learn a better representation.