Image Retrieval from Contextual Descriptions (2022)

Abstract: The ability to integrate context, including perceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utterance. In order to measure to what extent current vision-and-language models master this ability, we devise a new multimodal challenge, Image Retrieval from Contextual Descriptions (ImageCoDe). In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description. As such, each description contains only the details that help distinguish between images. Because of this, descriptions tend to be complex in terms of syntax and discourse and require drawing pragmatic inferences. Images are sourced from both static pictures and video frames. We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on ImageCoDe.Our results reveal that these models dramatically lag behind human performance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pictures, compared with 90.8 in humans. Furthermore, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. Our hope is that ImageCoDE will foster progress in grounded language understanding by encouraging models to focus on fine-grained visual differences.

## introduction natural languages are highly contextual @xcite : for a listener, recovering the speaker's intended meaning requires integrating information from different streams, such as grounding in perception @xcite , shared world knowledge, and temporal reasoning @xcite . these processes, more generally, fall under the umbrella term of pragmatics @xcite . despite recent progress in multimodal systems, it remains unclear to which extent they can handle settings where context plays a major role, such as in real-world communication. to this end, we present a new challenge that requires multimodal models to leverage context to retrieve images from text. in particular, given a contextual description and a set of minimally contrastive candidate images, i.e. differing only in some details, the model has to retrieve the target image. in order to discriminate between similar images, human annotators naturally produce highly nuanced and grammatically complex descriptions. an example of our new challenging dataset, image retrieval from contextual descriptions (imagecode), is shown in figure 1 . during the data collection process, sets of similar images are selected among static pictures from open images @xcite and (a larger portion) among video frames from diverse domains. including both types of images allows for diversifying the dataset while representing different degrees of visual similarity within each set. next, we crowdsource a contextual description of a target image (presented together with the rest of the set) that contains only differences relevant for retrieval. after a filtering phase involving human retrievers, we obtain a large-scale dataset with 94,020 images and 21,202 descriptions associated with image sets of size 10. as a result of this annotation protocol, successfully completing the task requires models to integrate several kinds of context: i) the image set, as the descriptions often only make sense in the context of several other images and are not suitable as stand-alone captions. in fact, aspects of the image that are very salient and that therefore would normally be emphasized are not useful in our proposed task. instead, the focus of our descriptions are finegrained details that help discriminate between images (see figure 1 ); ii) the speaker's intention. due to their high degree of image similarity, contextual descriptions may be literally true for multiple images; however, once the speaker's intention is taken into account, the correct image can be determined by virtue of pragmatics, i.e. grice's maxim of quality 2 (see figure 2 , figure 7 ); iii) temporal sequences: for video frames temporal reasoning is also required to compare different moments of an unfolding event. on our new dataset imagecode, we benchmark a series of vision-and-language models that achieve state-of-the-art performance on other multimodal tasks, specifically vilbert @xcite and uniter @xcite as two cross-encoder variants and clip as a strong biencoder @xcite . we report several findings. first, accuracy on static images is vastly superior than on video frames. therefore, the degree of similarity among the candidate images has an overwhelming impact on retrieval performance. second, all state-of-the-art models generally struggle with image retrieval from contextual descriptions, whereas humans consistently achieve high accuracy. hence, we propose model variants capable of better taking context into account: i) once an imagedescription pair is encoded, we refine this representation by attending to the other images in the set; 2 note: while we do not model pragmatics explicitly in our baselines, we find that the imagecode contains many examples suitable for pragmatic modeling ii) we augment image encodings with temporal embeddings. based on our results, models take advantage of this additional information fruitfully but only to a limited degree. because of its challenging nature, due to the minimally contrastive images and complex descriptions, we believe that imagecode will help make visio-linguistic models more context-aware and sensitive to fine-grained details. ## related work there is a long tradition of grounding language understanding on single images, in the form of visual question answering @xcite , visual dialogue @xcite , or visual entailment @xcite . recently, more and more focus has been directed to settings where the visual context consists of multiple images, either conventional static pictures @xcite @xcite @xcite @xcite @xcite , or video frames @xcite . while many of these benchmarks involve just two images, covr @xcite and isvqa @xcite provide more images, similar to our sets of 10 images. isvqa and spot-the-diff (jhamtani and berg-kirkpatrick, 2018a) are most similar to our dataset, imagecode. isvqa is based on several video frames that are synthetic and cover a restricted domain, with short questions for visual question answering. spot-the-diff provides two frames from surveillance video cameras and descriptions of all their differences. imagecode is unique as a) we cover a wider range of domains; b) we construct image sets that are maximally similar while being distinguishable through natural language (section 3) and c) we limit descriptions to relevant differences. this results in (a) diverse, (b) complex and (c) pragmatically informative descriptions. we do not claim to explicitly model pragmatics in this paper, i.e. with rational speech acts @xcite . instead we present a dataset that is naturally suitable for pragmatic reasoning @xcite as a listener has to consider the context, assume a gricean speaker and resolve ambiguities resulting from nuanced differences. the reasoning in our task and data collection is therefore also similar to referitgame and subsequent work @xcite where one crowdworker generates a referring expressing for an object in a single image and another worker picks an object based on the expression. ## data collection our data collection involves two steps with a human describer and retriever. the describer is given a set of 10 highly similar images @xmath0, i 2 , ..., i 10 ], one of them marked as the target image i t , and has to write a description d that clearly distinguishes i t from the other distractor images. in the second step, the retriever is given the same 10 images and the description from the first step and has to identify the target image based on the description. s and d are only added to our dataset if the retrieval is successful. below, we outline the main stages of data collection: first, the collection of similar, contrastive images in section 3.1. then, the crowdsourcing of contextual descriptions in section 3.2 and validation of the examples via image retrieval (section 3.3). the final imagecode dataset consists of 94,020 images (partitioned into 9,402 sets) and 21,202 contextual descriptions (16,594 in the train split, 2,302 and 2,306 in the validation and test split respectively). ## results in finally, all three models achieve the highest performance when fine-tuned with both visual and temporal context. adding temporal positional embeddings on top of the contextual module (+temporalembeddings) yields an accuracy of 29.9 for clip, 25.7 for uniter and 24.5 for vil-bert. crucially, even the best-performing models lag significantly behind the (micro-averaged) human accuracy of 90.8 (cf. pre-trained model. across all model variants and training regimes, clip consistently achieves higher accuracy than vilbert or uniter. this implies that a larger amount of parameters, pretraining examples or the contrastive objective are more beneficial than vilbert's or uniter's more expressive model architecture. thus, these results violate the expectations that attention between vision and language would be more suitable to jointly encode highly nuanced visual details and descriptions @xcite . additionally uniter slightly outperforms vilbert as its single-stream architecture might enable richer cross-modal interactions. video frames vs. static pictures. the highest accuracy on the subset of the data with video frames (20.9) is far lower than that for static pictures (59.4). this confirms that videos represent the main challenge in imagecode, both because of the higher similarity of images in a set and of the particular factors of variation that help differentiate among them (cf. section 4.3 and examples in appendix g). additionally, model performance on video frames seems to increase more consistently as more context (both visual and temporal) is provided, whereas there is no clear trend in the case of static pictures. error analysis. on a broad level, we have seen that video frames are much more challenging for models. next, to identify more fine-grained causes for the overall low performance of the vision-andlanguage models on imagecode, we compute the pearson's correlation between accuracy and a series of possible explanatory variables. in particular, we find a weak negative correlation with the number of tokens in the description (@xmath1.11) and a weak positive correlation with the average pair-wise euclidean distance between clip encodings of the images in a set (@xmath2.22), which represents visual similarity. by focusing on the 1000 annotated examples in ## conclusions and future work we created a new challenge, image retrieval from contextual descriptions (imagecode), which is designed to evaluate the ability of vision-andlanguage models to integrate visual, pragmatic, and temporal context into their predictions. in particular, given a complex and nuanced contextual description, a model is required to retrieve the corresponding image from a set of highly similar candidates. we benchmarked state-of-the-art biencoder and cross-encoder models, such as clip and vilbert. moreover, we proposed new variants of these models that are more suitable to solve this task, by augmenting them with a module to attend on the other images in a set and temporal embeddings. we found that imagecode is highly challenging for all variants: even the best model (28.9) lags behind human performance (90.8) dramatically. images sourced from video frames display the largest gap in performance. the most challenging phenomena in imagecode include pragmatics, negation, fine-grained distinctions between images, and occlusion among others.