Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs (2023)

Abstract: Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special “SG Component” in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.

## introduction in recent years, vision and language models (vlms) such as clip @xcite have demonstrated impressive results and extraordinary zero-shot capabilities when trained on massive datasets containing image-text pairs (e.g., laion @xcite . however, recent empirical studies @xcite @xcite have shown that even the strongest vlms struggle to perform compositional scene understanding, including identifying object attributes and inter-object relations. our first step is to convert sgs into highly detailed captions. a naive approach would be to finetune vlms on these image-text pairs, however, we have found that this approach does not sufficiently improve performance. 2 this is also aligned with recent work @xcite , showing that contrastive learning approaches allow the model to concentrate mainly on object labels disregarding other important aspects, such as relations and attributes. to alleviate this issue, we take inspiration from these works and use the sg to generate hard-negative captions that highlight structural aspects. for example, if an sg contains an edge "dog-chasing-cat", then we can reverse that edge into "cat-chasing-dog" and generate a corresponding negative caption. next, we turn to introduce structure into the visual representation. inspired by prompt learning approaches @xcite @xcite , we incorporate into the image transformer encoder a set of "adaptive scene graph tokens", which interact with the patch and cls tokens via attention. by training these tokens to predict sg information, the encoder can capture better structured representations. however, the above task of predicting sg from the image transformer encoder is challenging, as it deviates from the initial vlm training objective. toward this end, we designed a technique tailored to the sg tokens, which parameterized the sg tokens independently from the patch tokens. in particular, we modify the transformer layers throughout the network by decomposing the parameters into two distinct sets: sg and patch tokens. we have found that this allows better learning of the graph prediction task while still maintaining zero-shot performance. we name our proposed approach sgvl (scene graphs for vision-language models). see figure 1 for an overview. to summarize, our main contributions are as follows: (i) we propose to exploit a small set of sg annotations that is rich with visual and textual information for enhancing compositional scene understanding in pretrained vlms; (ii) we introduce a new finetuning scheme that captures structure-related information using the visual and textual components when learning from sg labels. specifically, for the visual side, we incorporate special "adaptive sg tokens" in the image transformer encoder, and train these to predict sg information; (iii) our method shows improved performance on clip, blip, and blip2 on several benchmarks: winoground @xcite , vl-checklist @xcite , aro @xcite , and vsr @xcite , highlighting the effectiveness of our approach. text transformer encoder image transformer encoder man frisbee throwing graph-based negatives patch tokens adaptive sg tokens man frisbee man wearing shirt image-scene graph pairs wearing gray shirt gray shirt man throwing frisbee image-text loss ℒ !" scene graph loss ℒ 12 sg-to-text relationships matching objects matching gt classes and boxes predicted classes and boxes adapted with lora transformer layer with adaptive scene graph tokens patch tokens adaptive sg tokens 𝑄 < , 𝐾 < , 𝑉 < 𝑄 12 , 𝐾 12 , 𝑉 12 attention 𝑀𝐿𝑃 < 𝑀𝐿𝑃 12 explored prompt tuning in the context of multitask learning in nlp, followed by works in vision, and specifically in video domain @xcite , and vl @xcite . unlike these works, we add multiple prompts (which we refer to as adaptive sg tokens) to vlms in order to learn compositional scene information via the sg prediction task. we designed a technique tailored to the sg tokens, which decomposes the parameters throughout the transformer layers into two distinct sets: sg and patch tokens. this allows better learning of the graph prediction task while still maintaining zero-shot capabilities. learning structured representations. structured representations have been shown to be beneficial in many applications: video understanding @xcite @xcite , relational reasoning @xcite @xcite , vl @xcite @xcite , human-object interactions @xcite , and even image & video generation @xcite . in particular, sgs @xcite @xcite have been extensively used to provide semantic representations in a wide range of applications @xcite @xcite . unlike these previous works, here we propose a novel architecture design that utilizes scene graph data, demonstrating that a small amount of scene-graph data can be used to improve pretrained vlms via fine-tuning. finally, sg data has been recently used to evaluate for compositional understanding in vlms @xcite . this works differs from our approach, as this work only proposed a benchmark, and not a new training method, while our approach proposes a method for fine-tuning pretrained vlms using scene graphs. ## scene graphs for vl models we begin by describing the standard vl transformer architecture and the scene graph annotations (section 3.1). we then introduce our structural components for both language (section 3.2) and vision (section 3.3), and the training losses (section 3.4). our method is illustrated in figure 2 . ## experiments and results we apply our sgvl approach to three popular vlms: clip, blip, and blip2. 5 we evaluate on several vl compositional datasets following the standard protocol for each model. additional results and ablations are in supp. section a. ## conclusions structured understanding of complex scenes is a key element of human perception, but its modeling still remains a challenge. in this work, we propose a new approach for incorporating structured information into pretrained vlms from sg data to improve scene understanding. we demonstrate improved performance on four benchmarks probing compositional scene understanding with only a mild degradation in zs performance. our findings suggest that only a small amount of high-quality annotations may be sufficient to improve such mod-els. we hope these findings will encourage future work to generalize our approach beyond the vl regime or using other types of dense annotations, such as segmentation maps and depth maps.