Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (2023)

Abstract: Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relating entities from image regions with visually masked contexts. When applied to strong baselines pretrained on large amounts of Web data, zero-shot evaluations on both coarse-grained and fine-grained tasks show the efficacy of our methods in learning multimodal representations from weakly-supervised relations data.

## introduction current vision-and-language models (vlms) are pretrained on large amounts of image-text pairs collected from the web, and shown to perform remarkably on a variety of downstream applications (e.g., @xcite @xcite @xcite . nonetheless, recent work has highlighted their limitations in fine-grained tasks, where precise understanding of both modalities is required to correctly select a positive match against a negative one. examples of such tasks include verb understanding (sitting vs. standing; hendricks and nematzadeh 2021), word order (water in bottle vs. bottle in water; @xcite , spatial relations (above vs. below; @xcite ) and other linguistic phenomena @xcite @xcite . recent work @xcite @xcite shows that leveraging entity localisation, such as bounding boxes, from supervised data improves performance on downstream tasks like visual qa @xcite and visual reasoning @xcite . interestingly, modelling visual locations is also crucial to learn fine-grained image-text mappings @xcite . motivated by this promising research thread, in this work, we further explore the benefits of supervised data in multimodal pretraining by leveraging structured visual information in the form of relations in scene graphs @xcite @xcite . a scene graph is a data structure that describes the content of a visual scene by expressing its entities (e.g., helmet), their attributes (e.g., red), and their relationships (e.g., person wear red helmet; see figure 1 ). while a large body of work has focused on generating scene graphs @xcite @xcite @xcite @xcite , inter alia), they have also been used in other applications, such as im-age retrieval @xcite , image generation @xcite and image captioning @xcite . however, the use of scene graphs in vision-andlanguage (v&l) pretraining has received limited attention. in contrast to prior work @xcite ) that relied on relations inferred from captions through masked language modelling, we use a small dataset of humanannotated scene graphs, which go beyond the salient ones that are referred in a caption. to make full use of this rich learning signal, we introduce (i) a novel objective and (ii) a new method for datato-text generation, which both explicitly aim to induce structure in the image and its connection in text. our results show that modelling a limited amount of scene graph data in addition to millions of web-crawled image-text pairs further improves coarse-and fine-grained skills in vlms. contributions. in this work, 1) we aim at improving fine-grained understanding in vlms by modelling the structure of visual scenes during pretraining. in particular, we rely on a small amount of human-annotated scene graphs, 1 and propose two novel pretraining approaches: verbalised scene graphs (vsg) and masked relation classification @xcite . when compared to strong baselines, 2) we show their effectiveness during pretraining on both fine-and coarse-grained zero-shot tasks. our models achieve overall better fine-grained abilities, such as state-of-the-art in visual spatial reasoning, whilst keeping competitive performance on coarse-grained retrieval. 3) we shed light on the individual contributions and interactions of our proposed methods. we find, for instance, that vsg enables dense caption understanding, and that, at scale, modelling relations can be more effective than modelling entity locations for fine-grained understanding. finally, 4) we revisit the standard practice of selecting the last checkpoint in v&l pretraining, showing that coco dev tr@1 leads to better models, especially on coarse-grained tasks. ## learning visual relations recent work in modelling spatial layout of entities in an image @xcite has been shown to be effective for fine-grained v&l understanding @xcite ). yet, the information of a visual scene goes beyond individual entities, and understanding their semantic relationships is a key step towards better vlms. nevertheless, this is an under-explored area of research. we hence investigate two approaches to better impose the structure of visual scenes from scene graphs (see figure 2 ). the first approach, verbalised scene graphs (vsg), provides a different view of an image by associating it with a description of the relationships between entities in the image. our second approach, masked relation classification (mrc), predicts the relation between two entities in the same image when their cross-modal representations are obtained from visually masked contexts. ## experimental setup we validate the effectiveness of our approaches by enhancing two strong vlms on four, diverse fine-grained and two coarse-grained benchmarks. app. a provides details to reproduce our work. our models can be accessed and verified online. 2 models. we focus our analysis on four models: albef and x-vlm, and their corresponding relation-enhanced models (realbef and rex-vlm, respectively). for reference, we also test two strong systems: clip @xcite , a popular dual-encoder; and blip-2 @xcite , a vlm with frozen large image and text models. albef @xcite is a widely used vlm that achieves strong downstream performance by effectively combining key components for v&l learning, such as a contrastive objective and crossattention, in its design. in albef, an image and a caption are first independently encoded with a vision (vit; @xcite and a text (bert; @xcite transformer @xcite , respectively; and then fused in a dual-stream crossmodal transformer @xcite . the model is pretrained with three objectives (cf., section 2.1): masked language modelling, image-text contrastive learning, and image-text matching. x-vlm @xcite uses the same components and objectives as albef, but additionally learns to locate visual concepts in the image given the associated texts. it does so by predicting an entity's bounding box (bbox) coordinates given the visually grounded representation of its label (e.g., 'black swan'). moreover, @xcite showed that x-vlm also learns to ground an object label by applying albef's losses to a visually-masked image, which they collectively refer to as the visually-masked albef (vma) loss. these objectives allow it to acquire strong fine-grained understanding abilities, outperforming larger models, such as flamingo @xcite and blip-2 @xcite , on these tasks. ## results it is interesting to note that both our models further increase their performance gains when moving from 3m to 13m images, despite now only having 0.6% of the images annotated with scene graphs. improved fine-grained understanding. in addition to vsr, rex-vlm 13m performs best across all the fine-grained tasks, which test models for a much broader range of fine-grained skills. it gains +1.7pp on valse and +0.8pp on svo-probes, and rex-vlm 3m gains +1.4pp on valse. these results confirm that visual relations can provide useful signal towards fine-grained understanding, even when only available for a tiny percentage of pretraining data. on the other hand, realbef models are on par with their baselines. recall that they model relations between entities without explicitly learning about the entities themselves. that is, it is harder for albef models to learn relations without doing localisation. moreover, comparing x-vlm and realbef on valse and svo-probes, we see that modelling objects (x-vlm) on top of albef is slightly better than solely modelling relations (realbef). substantially better fine-grained understanding on dense captions. captions with two main predicates than one. we thus consider testing our models for the ability to understand long, fine-grained descriptions of images on the task of zero-shot image-paragraph retrieval. in fact, paragraphs are longer, more informative, and more linguistically complex than sentence-level captions. rex-vlm 13m achieves 89.3 tr@1 and 88.8 ir@1 (+9.0pp and +12.0pp compared to x-vlm 13m ). such high performance is largely preserved when training on 3m images (87.4pp and 87.8pp), and it carries over to real-bef models as well. overall, relation-enhanced models gain from +5.6pp to +12.8pp on this task. competitive coarse-grained retrieval. finally, we evaluate our relation-enhanced models on zeroshot image-text retrieval tasks to verify that their gains on fine-grained tasks do not hamper performance on coarse-grained tasks. ## analysis of learning dynamics recent work on v&l pretraining typically trains for a fixed number of steps, and then selects the last checkpoint to report performance on different tasks. however, @xcite showed that current, strong models achieve peak performance on different fine-grained tasks at different stages of pretraining. this motivates us to study the pretraining dynamics of our models, and to reassess the current practice of choosing the last checkpoint by investigating how different checkpoint selection strategies affect the performance on our tasks. convergence rates. performance for x-vlm models pretrained on 13m images is shown in figure 6 . 6 we find that our rex-vlm model requires longer training to achieve peak performance across fine-grained tasks. in fact, while our baseline's performance starts degrading after 250k steps, rex-vlm continues improving over time, reaching its best yet results at 500k steps. 7 we can also see that, by the end of our training, relation-enhanced models typically achieve better performance than the best results given by our baselines, confirming the validity of our results from the previous section. likewise, the evaluation curves show that our models and baselines can achieve comparable coarse-grained retrieval performance, and that longer training can help relation-enhanced models close the gap with the baselines (see figure 8 in app. b.2). given a fixed number of steps, we leave it for future work to investigate pretraining schedules that better balance coarse-and fine-grained tasks so as to obtain a single checkpoint that performs well on both kinds of skills. fine-grained vlms. while the vast majority of vlms are solely pretrained on large-scale data collected from the web (e.g., @xcite @xcite @xcite @xcite , a recent line of work investigates the challenge of learning fine-grained image-text mappings. filip @xcite , loupe @xcite , regionclip @xcite , pyra-midclip @xcite and hiclip @xcite propose different fine-grained alignment methods for dual-encoder networks. on the other hand, glip @xcite , fiber @xcite , pevl @xcite , mvptr @xcite , x-vlm @xcite and pali @xcite show the benefits of learning cross-modal representations from additional supervised object detection data. finally, there is increasing interest in training vlms that perform well on a range of coarse-and fine-grained vision and language tasks @xcite @xcite @xcite . scene graphs and multimodality. the structural representations of scene graphs has been explored in the context of different v&l tasks, such as image-text retrieval @xcite @xcite , image captioning @xcite , and visual qa @xcite @xcite . only two studies have, however, investigated the role of scene graphs in v&l pretraining. ernie-vil @xcite first extracts scene graphs from the captions with an off-the-shelf model, and then proposes mlm-based object, attribute, and relationship prediction tasks to learn cross-modal detailed semantic alignments. @xcite , in addition to extracting subject-relation-object triplets from captions with an off-the-shelf model, also generate paired visual features based on the entities output by an object detection model and their co-occurrences in the vg dataset @xcite . unlike them, we rely on a small sample of human-annotated scene graphs, and propose two methods for relation prediction in v&l pretraining. furthermore, we are the first to show the benefits of modelling scene graphs towards acquiring better fine-grained skills during multimodal pretraining. ## conclusion previous work in multimodal pretraining has shown the importance of modelling objects (using localisation data) in improving the performance of both coarse-and fine-grained tasks. in this paper, we investigate if supervision from relational data-by modelling relations between objects in a visual scene-can improve performance on these tasks. in particular, we rely on scene graph annotations, an under-explored data structure for multimodal pretraining, and propose two approaches for leveraging relations between entities in an image: 1) mrc, a pretraining objective that predicts the relation between the objects in two image regions; and 2) vsg, a versatile data-to-text generation recipe that converts scene graphs into captions, that can then be fed to any vlm. when applied to strong vlms, we find that our methods improve their finegrained understanding, with rex-vlm achieving state-of-the-art spatial reasoning abilities, as well as strong performance on other tasks too. we hope that our work motivates further research in improving fine-grained understanding in vlms. given the promise of our results with a few annotated images, an interesting future direction is to study how to best scale up our approaches with machine generated data, e.g., by generating pseudolabels from off-the-shelf scene graph generators from either images or captions, or both.