Learning to Evaluate Humor in Memes Based on the Incongruity Theory (2022)

Abstract: Memes are a widely used means of communication on social media platforms, and are known for their ability to “go viral”. In prior works, researchers have aimed to develop an AI system to understand humor in memes. However, existing methods are limited by the reliability and consistency of the annotations in the dataset used to train the underlying models. Moreover, they do not explicitly take advantage of the incongruity between images and their captions, which is known to be an important element of humor in memes. In this study, we first gathered real-valued humor annotations of 7,500 memes through a crowdwork platform. Based on this data, we propose a refinement process to extract memes that are not influenced by interpersonal differences in the perception of humor and a method designed to extract and utilize incongruities between images and captions. The results of an experimental comparison with models using vision and language pretraining models show that our proposed approach outperformed other models in a binary classification task of evaluating whether a given meme was humorous.

## introduction humor is an essential element of human communication. studies have shown that humor helps to build relationships in work environments @xcite , facilitates smooth discussions on controversial topics @xcite , and helps motivate people to recognize and challenge misinformation @xcite . memes are a type of humor that has been prevalent in recent years, especially on social media. these images express multi-modal humor and often comprise a template image with superimposed upper and lower captions. in the example of a meme shown in figure 1 , the upper caption reads "join 1https ://www.funny-memes.org/2013/05/join-marinesthey-said-youll-be-hero.html figure 1 : an example of a meme from a meme-sharing website "best of funny memes"1. the marines they said" and the lower caption reads "you'll be a hero they said". both upper and lower captions are superimposed onto the template image of a marine in dress uniform holding an umbrella for the former president barack obama. the humor of memes can be explained by the incongruity theory @xcite . it is a well-established humor theory which states that a surprising contradiction or opposition to an expected situation or interpretation is a key element of any humor. for example, the meme shown in figure 1 has an incongruity between the caption and the template image; the caption explains the commonsense of viewers that a marine would be a hero in the battleground, but the image is showing the contradicting reality of a marine doing a boring job of holding an umbrella for the president. a study on incongruities in memes has shown that a large number of memes have image-caption incongruity @xcite . given the substantial impact of memes on online communication, such as their effectiveness in correcting misinformation @xcite @xcite , researchers have aimed to develop ai systems capable of understanding humor in memes. however, the evaluation of memes has proven a difficult task. humor is subjective, and interpersonal differences may affect the perception of humor for some memes based on viewers' cultural background and personality characteristics @xcite . hence, human-annotated datasets of humor in memes tend to exhibit inconsistent annotations. in addition, existing methods use vision-language pretraining models that do not explicitly extract semantic relationships between images and captions. as a result of this structure, utilizing image-caption incongruity is relatively difficult with such models, although incongruity was shown to be a significant element in humor of memes @xcite . in the present work, we addressed the inconsistency of annotations by first creating a meme dataset with a humor annotation of memes that is not influenced by interpersonal differences in the perception of humor. these reliable humor annotations were obtained via an annotation method called best-worst scaling (bws) @xcite . then, the annotations were refined by our proposed process to eliminate inconsistent examples. the consistency of each annotation was measured by quantifying the agreement of annotations between different annotators. based on this data, we propose a method that explicitly extracts and utilizes image-caption incongruities. our proposed method combines a visionlanguage transformer with a module designed to extract the features of image-caption similarity. we validated the performance of our proposed method by conducting experiments in which we used several models for comparison to classify whether a given meme (template image + caption) was humorous. the contributions of this study are summarized as follows. ## dataset construction to construct our reliable dataset, we first obtained a collection of memes from a meme-sharing website. then, the memes were annotated by crowdworkers. finally, the annotations were filtered and refined to eliminate inconsistent annotations. as a result, we compiled 1,450 memes with reliable and consistent annotations. ## meme evaluation model based on the incongruity theory studies have shown that many memes exhibit incongruities between images and their captions, which express humor @xcite . therefore, we hypothesized that a module designed explicitly to extract incongruities between an image and its caption would improve a model's ability to classify whether a given meme is humorous. to extract incongruities between image and text, we propose an incongruity extraction module consisting of clip image and text encoder @xcite , which is highlighted in orange in figure 6 . in this proposed method, a template image and caption are each fed into the corresponding pretrained clip encoder to obtain feature vectors of both the image (𝒗 @xmath0. since pretrained clip encoders are trained such that the encoded feature vectors of a similar image and caption are located close to each other in the same latent space, we hypothesized that a feature vector 𝒗 @xmath1. we considered this information encoded in 𝒗 clip to be useful in determining the level of incongruity between an image and its caption because imagetext incongruity can be considered as a type of semantic relationship between image and text. after obtaining 𝒗 clip as a feature representing incongruity between an image and a caption, 𝒗 clip is concatenated with the output features of a vilt encoder (𝒗 @xmath2, as shown in figure 6 . ## experiments 5.1 experimental setting to compare and evaluate the performance of the proposed method with the other models compared on the task of classifying humor in memes, we used 1,411 memes in the refined dataset with both upper and lower captions. to train and evaluate the models, we conducted a ten-fold cross-validation, in which 1,411 memes were randomly divided into ten subsamples such that all memes with the same template image belonged to the same subsample. the memes were distributed such that all subsamples had approximately the same amount of memes. the subsample with a minimum number of memes had 131 memes, and that with a maximum number had 153. to evaluate the models, each model was trained and evaluated ten times with the data divided into training, validation, and testing sets with a ratio of 8:1:1. for each evaluation step, the weight of the model that achieved the highest classification accuracy on the validation set was chosen to be evaluated on the testing set. we recorded classification accuracy scores calculated on ten different testing sets. in addition, to minimize the effect of random initialization of the mlp model on the results of the evaluation, we conducted ten-fold cross validation with eight runs over different random seeds. therefore, a total of 80 accuracy scores were obtained from each model, and the average accuracy score and standard error for each model were used for quantitative comparisons. ## conclusion constructing a computational system to evaluate humor in memes is difficult due to the lack of datasets of memes with reliable and consistent humor annotations and the complexity of searching and extracting cross-modal incongruities between images and their captions. to overcome these challenges, we first created a dataset of memes annotated using bws and proposed a refining process which was able to eliminate examples of memes affected by interpersonal differences in the perception of humor. then, we used the refined dataset to train and validate the effectiveness of the proposed method, which was designed to extract incongruities between images and their captions to accurately classify whether a given meme is humorous. the experimental results showed that the proposed model was able to extract and utilize incongruities between images and their associated captions to outperform other multi-modal models on the humor classification task. this demonstrates the importance of using features that represent incongruities when evaluating humor in memes. possible future work includes using the features representing incongruities not only to evaluate but also to generate new humorous memes from text or image input.