Content-Specific Humorous Image Captioning Using Incongruity Resolution Chain-of-Thought (2024)

Abstract: Although automated image captioning methods have benefited considerably from the development of large language models (LLMs), generating humorous captions is still a challenging task. Humorous captions generated by humans are unique to the image and reflect the content of the image. However, captions generated using previous captioning models tend to be generic. Therefore, we propose incongruity-resolution chain-of-thought (IRCoT) as a novel prompting framework that creates content-specific resolutions from fine details extracted from an image. Furthermore, we integrate logit bias and negative sampling to suppress the output of generic resolutions. The results of experiments with GPT4-V demonstrate that our proposed framework effectively generated humorous captions tailored to the content of specific input images.

## introduction humorous content comprising an image with an associated caption is universally popular in different communities. for example, imgflip 2 , bokete 3 , and the new yorker cartoon caption contest 4 all express different tastes in humor using images with text captions. this form of humorous content is important in human communication, such as by providing an effective way to lead others to challenge misinformation @xcite . while the topic of humorous image captions remains relatively unexplored, several studies have leveraged large-scale datasets of humorous combinations of images with captions from the internet to train image captioning models @xcite @xcite . however, previous research has shown that image 1 our project page is available at https: //kohtaro246.github.io/publication/ircot 2 https://imgflip.com/ 3 https://bokete.jp/ 4 https://www.newyorker.com/cartoons/contest captioning models trained using cross-entropy loss have a tendency to generate similar captions for different images @xcite . llms like gpt4 (openai, 2023a) have also been used to generate humorous captions using descriptions of the images provided by humans. however, existing methods are relatively limited, focusing on a single example and lacking in quantitative analysis @xcite . furthermore, the capabilities of large multimodal models (lmms) such as gpt4-v (openai, 2023b) to generate humorous captions have not been previously investigated. in this study, we found that gpt4-v also tends to produce generic captions in attempts at humor, and largely fails to capture the content-specific nuances in images found in humorous captions created by humans, as shown in figure 1 . inspired by the incongruity theory of humor, we introduce incongruity-resolution chain-of-thought (ircot) as an approach to generate humorous captions related to the content in input images. the incongruity-resolution theory is a well-established framework that describes how humor arises from an unexpected contradiction resolved through a cognitive rule that explains the content's incongruity @xcite . a study on incongruity in image macro memes, a form of humor comprising an image with an associated caption, suggests that most memes conform to the incongruity-resolution theory @xcite . ircot guides a machine learning model to identify and resolve incongruities in the content of input images as shown in figure 1 . we hypothesized that ircot could facilitate the creation of content-specific humorous captions for each image by generating resolutions based on intricate and unique details of the content depicted in the image, which can be recognized in the preceding steps. the results of experiments using gpt4 show that ircot improved the specificity of humorous captions generated by gpt4-v compared to prompting lmms. the framework is intended to generate humorous captions that are specific to the content in a given image based on the unique details of the image extracted by an "image description" module. in addition, we introduce logit bias and negative sampling to generate unique resolutions. this leads to the generation of humorous captions related to the content in a given input image. furthermore, we show that using logit bias and negative sampling fine-tuning during the resolution step enhanced the specificity of the generated captions without the need for training data provided by humans. these techniques penalize the model for generating generic resolutions that can resolve any incongruities. in addition to achieving content-specificity, we argue that ircot may reduce the risk of models generating offensive content compared to other data-driven approaches that use image-caption humor datasets that contain offensive content @xcite . intermediate explanations generated by ircot may improve an offensiveness detection model with challenging samples. this issue has become particularly pressing in light of increasing social demands for generative models to avoid harmful content, as outlined in the eu artificial intelligence act @xcite . the contributions of this study are summarized as follows: ## content-specificity of humorous captions generated by lmms in this section, we describe our analysis of the content-specificity of humorous captions generated by an lmm using gpt4-v. we selected gpt4-v for this analysis because it is considered a benchmark in lmms, and is commonly used to create training data for other models and evaluate their performance @xcite . additionally, prior research on the capabilities of computational models to assess humor identified gpt4 as the most proficient model among the three tested @xcite . ## content-specific humor generation in this section, we describe ircot, a novel prompting method that aims to improve the contentspecificity of humorous captions generated by lmms. ## detecting offensive content to explore the usage of ircot to detect offensiveness in generated humor captions, we prompted gpt4-v to rate the offensiveness of the captions generated by "llava ns+lb res". we compared qualitatively whether prompting with ircot intermediate steps would alter the rating generated by gpt4-v. figure 5 shows an example of a humorous caption which gpt4-v was only able to identify as possibly offensive when provided with ircot intermediate steps. this humor arises from the fact that there are some people who believe that drinking gin would alleviate the conditions of a cold. while the caption itself seems innocent pun, knowing the background of the pun can lead to new interpretations that could potentially be harmful. this example highlights the complexity of detecting the offensiveness of image-text humor, and the potential for ircot to aid in the detection of difficult-to-understand offensiveness. ## conclusion we demonstrated that using ircot with negative sampling and logit bias enables gpt4-v to generate humorous captions that are specific to input image content without the need for training data created by humans. the captions generated using ircot were considered more humorous compared to human-generated captions. this study is a pioneering effort to deepen our understanding of humor that appeals to humans.