Understanding Cross-modal Interactions in V & L Models that Generate Scene Descriptions (2022)

Abstract: Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state of the art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.

## introduction when humans view images, they can quickly capture their 'gist'. for example, it is immediately evident that figure 1 is a kitchen. such judgments are fast and are informed by expectations about which objects occur in typical scenes ('scene semantics') and their configuration ('syntax') @xcite @xcite . this knowledge affects the deployment of attentional resources @xcite @xcite . scene understanding and object recognition constrain the selection of attended locations in human visual attention @xcite . in this paper, we explore the implications of these findings for image captioning models. there are at least two levels at which an image can be appraised. an object-centric perspective focuses primarily on individual objects and actions (e.g. the example caption in fig 1 ). this has dominated captioning models (see @xcite , for an early, influential statement of this view) and has informed the design of widely-used datasets, which pair images with captions that explicitly mention at least some of the objects in a picture (e.g. @xcite @xcite @xcite . in contrast, a scene-level caption (e.g. 'a kitchen' for figure 1 ) contains less object-specific detail. such captions are less redundant with respect to the image they describe, but convey enough information to generate inferences about content and structure (e.g. kitchens typically contain cupboards, but not birds; etc). most image captioning datasets contain objectcentric captions and no currently available resource pairs both scene-level and object-centric captions with images. in this paper, we address this gap and ask (i) whether captioning models can be adapted both for object-centric and scene-level captioning and (ii) whether the two strategies rely on different types of interplay between the visual and linguistic modalities. addressing these questions can shed light on the ability of v&l models to reason about the relationship between scenes and their components. in addition, it is desirable for mod-els to generate scene-level descriptions as well as object-centric ones. in many communicative contexts, scene-level captions are informative and nonredundant, recalling the quality and the quantity discourse maxims defined by @xcite . we present a study of object-centric versus scene-level captioning. we focus on vinvl @xcite , a bert-based model in the oscar family @xcite of models, which have recently dominated the state of the art in image captioning. 1 our main contributions are: i) we introduce a novel dataset, hl-scenes (sec 3) extending part of the coco dataset @xcite with scene-level descriptions. the analysis is designed to thoroughly inspect object-scene relations by exploiting crossmodal attention (sec 5), coupled with probing (sec 7) and ablation studies (sec 6). ## related work datasets existing image-caption datasets emphasise object-centric captions (an early exception, using abstract scenes, is @xcite . this is also true of web-sourced datasets such as conceptual captions (cc; @xcite . for example, the cc filtering pipeline explicitly checks for overlaps between caption tokens and objects identified in the image. the nocaps benchmark @xcite tests models' ability to generalise to out-of-domain objects. there are several v&l datasets and tasks which introduce knowledge-rich annotations and address models' ability to reason with linguistic and visual cues @xcite @xcite @xcite . in this paper, we take this line of work further by introducing the novel hl-scenes dataset, which pairs object-centric and scene-level captions to images. models transformer-based v&l models are usually divided into single-stream @xcite @xcite and dual-stream @xcite @xcite architectures. it has been shown that single-and dual-stream models perform roughly at par under the same training settings @xcite . on the other hand @xcite showed that model performance is highly impacted by dataset curation, attention, and loss function definition. most v&l single-stream models are inspired by bert @xcite . they incorporate the visual modality in the form of features extracted using a visual backbone, typically a faster-rcnn @xcite pre-trained on an object labelling task such as imagenet @xcite . from the perspective of caption generation, the oscar @xcite single-stream architecture has emerged as an influential model. oscar enforces grounding between image-caption pairs by using object labels as anchor points (a strategy also adopted by @xcite . this makes it particularly suited to the goals of this paper, namely, in-depth analysis of the crossmodal interactions in the treatment of objects during generation. oscar and its successors, vinvl @xcite and lemon @xcite achieved sota performance on captioning tasks such as coco and nocaps. ## data we developed the new high level scenes (hlscenes) dataset, which is explicitly designed to pair images with both object-centric and scene-level captions. to this end, we sampled 15k images from the 2014 coco train split @xcite , with the constraint that each image depicts at least one person. captions in coco are highly objectcentric @xcite . we crowd-sourced three scene-level annotations per image on amazon mechanical turk foot_1 , from workers with at least an 85% approval rating. crowd workers saw an image and wrote a description in response to the question: where is the picture taken? annotators were encouraged to use their knowledge of typical scenes in writing their descriptions. finally, we paired our scene-level hl captions with the previously available coco @xcite captions. figure 2 shows an example of an image with the two types of captions. see appendix e for more examples. we collected a total of 14,997 imagecaption pairs, and we reserve 11,999 for training and 1,499 each for validation and testing. ## model vinvl @xcite ) is a single-stream bert-based model with a faster-rcnn @xcite visual backbone. it is an extension of oscar @xcite . vinvl implements a training strategy where object tags are used as anchor points between the visual and textual modality to facilitate cross-modal alignment. as pointed out by @xcite , this strategy is motivated by the fact that in the datasets used to pre-train multimodal coco reference: a close-up of a kitten looking at a dog laying in the background. generated: a cat and a dog sitting next to each other. ## how does attention to objects change from object-centric to scene-level generation? we first investigate the model's self-attention before and after fine-tuning on the scene-level caption generation task. ## conclusion in this paper, we addressed scene-level caption generation. taking a cue from prior work on scene semantics and syntax, our goal was to assess v&l models' ability to reason about the link between scenes and their components and exploit this to generate informative captions with less redundancy.