Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos (2022)

Abstract: We present the first openly available multimodal metaphor annotated corpus. The corpus consists of videos including audio and subtitles that have been annotated by experts. Furthermore, we present a method for detecting metaphors in the new dataset based on the textual content of the videos. The method achieves a high F1-score (62%) for metaphorical labels. We also experiment with other modalities and multimodal methods; however, these methods did not out-perform the text-based model. In our error analysis, we do identify that there are cases where video could help in disambiguating metaphors, however, the visual cues are too subtle for our model to capture. The data is available on Zenodo.

## introduction figurative language is a challenging topic for computational modeling as the meaning of a figurative expression is non-compositional and typically very context dependent (see @xcite . metaphor is one of the most important figures of language; it is constantly used in every day language @xcite to draw comparisons or to express something difficult and foreign in more familiar terms. metaphors can be conventional @xcite and they are often found in idioms, but at the same time metaphors are used to create something new (see @xcite . given its ubiquitous presence, understanding metaphors is integral in achieving true natural language understanding (nlu) in the real world. without their successful interpretation, our models are bound to make mistakes whenever anything is expressed in an indirect or creative fashion. metaphors are often very contextual and their successful detection and interpretation requires a wide range of contextual cues that would be captured in audio (e.g., prosody) and video (e.g., gestures and actions). therefore, we believe a multimodal dataset is a great contribution to metaphor research within and outside of the field of nlp. two important parts of a metaphor are a tenor and a vehicle (see @xcite . for example, in the metaphor life is a journey, life is the tenor and journey is the vehicle. how metaphors essentially operate is that a vehicle is used to give some of its attributes to the tenor. in the case above, journeys are long and full of adventure, which means that these properties are attributed to life in an indirect fashion. the meaning of a metaphor is never literal nor compositional, but rather calls for interpretation on the level of pragmatics (see @xcite . meanwhile, multimodality is becoming increasingly important for many tasks (see @xcite @xcite . we believe the availability of multimodal datasets for a variety of nlp tasks is lacking, and we hope to contribute to the community with our multimodal metaphor dataset. in this paper, we present the first fully open expert annotated multimodal dataset for metaphor detection 1 . in addition, we experiment with unimodal and multimodal methods for metaphor detection. our results indicate that the text-based model achieved the best performance. we discuss the results of our experiments and conduct an extensive error analysis to shed light on what was learned successfully by the model and its shortcomings. using cc by licensed videos in our corpus has been the primary design principle of our data collection so that we can release our corpus without any restrictions in its entirety. this, we believe, is more useful for research purposes than a corpus consisting of short video clips to compile with copyright laws such as the fair use law in the us. ## related work metaphors have, thus far, been computationally detected using only text. in this section, we describe some of the recent approaches for textual metaphor detection, the corpora used to achieve that and some of the multimodal research conducted on nlp tasks other than metaphor detection. there are several takes on metaphor interpretation @xcite @xcite and generation @xcite @xcite , but we do not describe them in detail as interpretation is a very different problem. there are two corpora currently used for metaphor detection, vu amsterdam (vua) metaphor corpus @xcite and corpus of non-native written english annotated for metaphor @xcite . unlike our corpus, both of these datasets contain textual modality only. for textual metaphor detection, gao et al. ( 2018 ) has used a bi-directional lstm (long short-term memory) based model with elmo embeddings. similarly, @xcite have used a bi-lstm model with bert and xlnet for the same task. not unlike the previous approaches, @xcite has also applied bi-lstm models comparing elmo and glove embeddings to bert embeddings with global and hierarchical attention models. traditional machine learning methods, logistic regression, linear svc (support vector classification) and random forest classifier, have been used recently with feature engineering to detect metaphors @xcite . in deepmet, proposed by @xcite , a siamese neural network have been utilized, where textual roberta @xcite embeddings are computed from the context, the token in question and its part-of-speech and fine-grained part-of-speech. deepmet was the best performing solution for detecting textual metaphors in the vua dataset, based on a recent shared task @xcite . there are several recent works on multimodal detection of a variety of linguistic phenomena. for example, svms (support vector machines) with word embeddings and feature extraction have been used for multimodal sarcasm detection @xcite . @xcite uses glove embeddings, features extracted from audio and facial recognition system output to predict emotion in a multimodal dataset. these multimodal features are fused using a memory fusion network (mfn) @xcite . similarly, @xcite detect emotion in a multimodal dataset by modeling the problem from the point of view of the quantum theory. while the field has seen increasing research on multimodal nlp @xcite @xcite , no data or model has been proposed for multimodal metaphor detection. ## our metaphor corpus in this section, we present our video, audio and textual corpus of manually annotated metaphorical language. our selection of the video clips includes only cc-by licensed videos on youtube that have human authored closed captions in english. the content of the videos presents mainly real people talking, which rules out animations and video game streams. the availability of human authored closed captioning is important as it speeds up our annotation time and provides us with subtitles that are already aligned with video and audio. the cc-by license was an important selection criterion because it makes it possible for us to release the dataset openly. we used the filters provided by youtube to limit our search to videos that were marked as cc-by and had closed captioning. however, the youtube filter does not distinguish between automatically generated closed captioning and a human authored one. fortunately, it is relatively easy to tell these two apart from each other. automated closed captioning tends to appear one word at a time, whereas human authored closed captioning is visualized more like traditional subtitles. these criteria greatly reduced the number of eligible videos to include in our corpus. apart from these criteria, we also filtered videos with sensitive and offensive languages. no further restrictions have been explicitly placed on the genres or types of videos, as we do not want to introduce biases for which types of contents are more likely to contain metaphors. therefore, the availability of the metaphors naturally occurring in the corpus is the result of the ubiquity of the metaphor in everyday language use. all youtube queries were conducted in incognito mode to avoid biased youtube suggestions based on our viewing habits. figure 1 shows real examples from our corpus where video can be useful in detecting metaphors. on the left, the woman wearing a gray shirt is sentence that you can use to really up @xmath0 our corpus consists of 27 youtube videos with a total duration of 3 hours, 53 minutes and 47 seconds of video. for comparison, a recently released multimodal dataset for sarcasm detection @xcite has the duration of 3 hours, 40 minutes and 47 seconds. the videos belong mostly to a start-up domain and many of them deal with issues of online visibility for a start-up company. this domain was a consequence of our selection criteria for videos. it turns out that youtube has plenty of high-quality human close-captioned videos released under the cc-by license that relate to this particular domain. our corpus provides linguistics researchers with the ability to study the use of metaphor in a multimodal setting, something that has gained attention in their field of science as well @xcite . this can, indeed, foster a wider interdisciplinary collaboration leading to a deeper understanding of the phenomenon. ## metaphor detection we experiment with uni-and multi-modal models for metaphor detection. in this section, we describe the preprocessing steps applied and the experimental setups conducted. ## results in this section, we follow the evaluation metrics commonly used for the metaphor detection task by reporting the precision, recall and f1 scores for the metaphorical label. regarding the textual models, we report three sets of results, which are for the models trained on: 1) vua corpus, 2) our corpus and 3) both the vua and our corpus. all the models predict metaphoricity on the token level. to ensure that our implementation of the deepmet approach is correct, we tested the first model on the vua test dataset of the metaphor detection shared task and achieved an f1-score of 0.68 and 0.73 on all pos and verb subsets of the data, respectively. these results are relatively close to the results reported by the authors. we believe that the huge differences between the first and second textual models, despite the larger size of vua's training dataset, are due to the differences in domains. the vua corpus contains academic texts, conversation, fiction, and news texts, whereas our corpus is dominated by conversations on the web and start-ups. it is evident that by exposing the model to general domains (i.e., vua's corpus) and, thereafter, concentrating it on the start-up domain, the model was able to identify the highest number of metaphorical usages. results from the other models (unimodal or multimodal) that involving audio and video showed that adding these modalities actually did not help improving the model -rather, they are detrimental to the model performance on metaphor detection. we extend two possible explanations for this failure. first, it is possible that because the visual and audio cues of metaphor are subtle, these models failed to learn from such a small amount of annotated data. second, it is unclear that the specific models we are using for audio and video modalities encode the information relevant for the metaphor detection task. for instance, whereas it is impossible to completely disentangle what exactly the wav2vec model is encoding, we can conjecture that it encodes information about phoneme identity considering it is optimized for the speech recognition task. therefore, it may not be entirely surprising that the wav2vec encoding is not useful for the metaphor detection task because it is adding redundant or irrelevant information to the model. it is our future work (or the future work for the community who utilizes this dataset) to refine our understanding of the multimodal encoding for the metaphor detection task (for instance, employing a model that more directly encodes information about speech prosody from the audio). ## discussion and conclusions in this work, we have only focused on metaphor as a strictly linguistic phenomenon and we have built a multimodal dataset where these linguistic metaphors have been tagged in terms of tenors and vehicles. however, it is apparent that metaphor is a phenomenon that occurs on a higher level of our cognitive capacities than mere language. there are several cases in our corpus, where we can evidence the existence of a metaphor but it is never expressed verbally. for example in figure 2 , money flying cannot be a metaphor when inspected purely from the point of view of language and its relation to the video when money is actually flying in the scene. however, it is a metaphor on a higher level in the sense that the entire scene where money was flying was to indicate someone becoming rich. in other words, stating a fact that is happening is not metaphorical if the fact is literally taking place, however the fact itself might be metaphorical. at the same time, as evidenced by our error analysis, there are certainly cases where video modality could help in disambiguating whether something is said metaphorically or not. for instance, talking about sprinkling in a kitchen environment (see figure 3 ) is a very strong sign that the word is potentially non-metaphorical. integrating these weak cues into a multimodal system is, however, not an easy task given that the current methods for video processing are limited in their coverage. therefore, in the future, it would be useful to annotate metaphors also in the other modalities. money flying can be a visual metaphor, and so can a sound effect, and they can exist independently from each other in different modalities. perhaps the reason why our multimodal attempts failed was that metaphor can be independent of the other modalities. producing such a dataset where these modal specific metaphors are also annotated for video and audio is definitely a huge undertaking that requires research in its own right. it is clear that our model can detect metaphors correctly, but also the mistakes it makes highlight that despite using a large roberta model, the meaning representation the model has cannot reach to such a nuanced level as to confidently detect metaphors. metaphor is a figurative device that cannot be explained by semantics, but rather requires pragmatic inspection. it is not clear based on our research and other contemporary approaches whether the current word or sentence embedding models are sufficient to navigate in the depths of pragmatics and subjective interpretation in any other way than learning some irrelevant co-occurring phenomena from a biased corpus. at the same time there is no such thing as an unbiased corpus, either, given that bias (and mostly heuristics causing it) is a fundamental part of our cognition as human beings. in this paper, we have presented a new open and multimodal dataset for metaphor detection. because we have focused strictly on cc-by licensed videos, we can make the entire dataset available on zenodo. in our current work, we have not taken the context widely into account when predicting metaphoricity, but rather resorted to a very local context. the fact that the videos can be published in full length makes it possible for any future work to explore different ways of including contextual cues freely.