Unveiling the Invisible: Captioning Videos with Metaphors (2024)

Abstract: Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent studies have shown that Vision-Language (VL) models cannot understand visual metaphors in memes and adverts. As of now, no probing studies have been done that involve complex language phenomena like metaphors with videos. Hence, we introduce a new VL task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a manually created dataset with 705 videos and 2115 human-written captions, along with a new metric called Average Concept Distance (ACD), to automatically evaluate the creativity of the metaphors generated. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task. We perform a comprehensive analysis of existing video language models on this task and publish our dataset, models, and benchmark results to enable further research.

## introduction metaphors are the most commonly used form of figurative language in literature @xcite . metaphors are a tool to colour the imagination of the reader by introducing unknown concepts in comparison to familiar concepts, thereby allowing them to be understood easily and powerfully. this trope is used in various creative fields like advertisements @xcite to convey information more effectively and includes modalities like text, images, and audio. figure 1 shows an example of using an image to creatively convey an idea. metaphors are also used in video 1 code, data, and models are available at https:// github.com/abisekrk/video-metaphor-captioning figure 1 : an example of a creative advertisement that shows the speed of the broadband by depicting a scene from the iconic movie 'titanic'. recently, chat assistants that can answer questions related to videos have shown good promise on standard video datasets @xcite @xcite . however, they struggle to understand videos that contain metaphors. to this effect, we introduce a novel task of 'video video explanation: the egg is so strong that is unbreakable with a hammer. the reason is that it was laid by a hen that was fed food from a fevicol (a glue) box. ## dataset no existing datasets have metaphor details available for videos. as advertisements have metaphorical representations in them to convey additional messages to viewers, we choose the pitt's ads dataset @xcite for constructing our dataset. the pitt's ads dataset consists of advertisement images and videos on a wide range of topics. the released dataset contained urls to 3, 477 videos out of which only 2063 videos are currently accessible. we annotate these videos with metaphor information for our experiments. additionally, we also query youtube with keywords like advertisements, creative advertisements, funny advertisements, etc. using the youtube search tool foot_0 . we filter videos that are less than 2 minutes and add them to our video metaphor captioning dataset (vmcd) if they have metaphors in them. ## our model existing video-language models are trained on video-text parallel data that do not contain much metaphor content. hence, in addition to analyzing their performance on our dataset, we introduce new models that are introduced to metaphors in the pretraining stage itself. in our model, the video representation is obtained through a pretrained video captioning model and prefixed with an instruction sequence to a large language model (llm). the llm generates the caption as a sequence of tokens conditioned on the video input and the instruction. figure 3 illustrates the model architecture. we sample 'k' frames from the input video 'v ', where k depends on the input restrictions of the video captioning model. we use the llava-13b-v1.5 @xcite ) model architecture for our experiments inspired by its success on many vision language tasks. we use the generative image text transformer model (git) @xcite as the video captioning model (c) for obtaining the video representation and vicuna @xcite as the llm. in all our experiments we freeze the weights of the git model and only finetune the mapping network and the llm. our model has two key advantages. since we train the mapping network to learn the mapping between the git decoder state to the embedding space of the llm, the mapping network maps git's understanding of the video in the form of its representation to the llm's embedding space, allowing the llm to directly generate output from the video. this also reduces the need to pretrain the model on a huge corpus of video-text parallel data which is resource intensive. ## experiments our experiments follow a two-step process. the models are first pretrained on the pretraining data built from mscoco and haivmet datasets and then finetuned on the vmc dataset. we discuss the experiment settings for both as follows. ## evaluation metrics we evaluate the performance of our model using a set of automated metrics and human evaluation. the n-gram overlap-based metrics-bleu @xcite , rouge @xcite are commonly used to compare the performance of the model in captioning tasks. as discussed in previous works on metaphor generation, the n-gram overlap based metrics cannot capture the quality of generated metaphors. this is because the same information can be conveyed through different comparisons. hence, we also report bertscore @xcite that compares the semantic similarity of the generated caption and the reference caption. in the task of video metaphor captioning, the model is trained to generate creative metaphors as output. previous works rely on manual evaluation to quantify the creativity and metaphoricity of the generated captions. as no existing metric can be used to evaluate the creativity of metaphors, we introduce a new and intuitive metric called-"average concept distance" (acd). it is calculated as: s-bert (reimers and gurevych, 2019) (allmpnet-base-v2) is used to obtain representations for pc and sc. for captions that do not contain either pc or sc, the similarity score is set as 1 to penalize the model. thus the model is evaluated based on how diverse comparison it can make for the object in question. figure 4 provides an overview of the acd metric computation. in addition to these automated metrics, we also manually evaluate models on four metrics-fluency, primary concept consistency, consistency, and creativity. ## results and analysis we evaluate the models based on both automatic metrics and human evaluation. ## conclusion and future work in this work, we proposed a novel vision-language (vl) task called video metaphor captioning that probes the language reasoning abilities of the video language models. we constructed and released a manually annotated dataset for the proposed task. we also released a new metric to automatically evaluate the creativity of the generated metaphor captions. our low-resource vl model that used a frozen video captioning model (git) with an llm decoder (vicuna) to generate metaphor captions showed comparable performance to sota video language models on the video metaphor captioning task. it was observed that all the video language models studied in the work lack a deeper understanding of video and language for a complex task like video metaphor captioning. we believe that our work will enable future research in this direction with our dataset and models being a strong benchmark for progress.