Semantic Aligned Multi-modal Transformer for Vision- L anguage U nderstanding: A Preliminary Study on Visual QA (2021)

Abstract: Recent vision-language understanding approaches adopt a multi-modal transformer pre-training and finetuning paradigm. Prior work learns representations of text tokens and visual features with cross-attention mechanisms and captures the alignment solely based on indirect signals. In this work, we propose to enhance the alignment mechanism by incorporating image scene graph structures as the bridge between the two modalities, and learning with new contrastive objectives. In our preliminary study on the challenging compositional visual question answering task, we show the proposed approach achieves improved results, demonstrating potentials to enhance vision-language understanding.

## introduction vision-language tasks, such as image captioning @xcite , visual question answering @xcite , and visual commonsense reasoning @xcite , serve as rich test-beds for evaluating the reasoning capabilities of visually informed systems. these tasks require joint understanding of visual contents, language semantics, and cross-modal alignments. in particular, beyond simply detecting what objects are present, models have to understand comprehensively the semantic information in an image, such as objects, attributes, relationships, actions, and intentions, and how all of these are referred to in natural language. inspired by the success of bert @xcite on a variety of nlp tasks, there has been a surge of building pretrained models for visionlanguage tasks, such as vilbert @xcite , vl-bert @xcite , and uniter @xcite . despite the impressive performance on several vision-language tasks, these models suffer from fundamental difficulties in learning effective visually grounded representations, as they * equal contribution figure 1 : a visual question-answering example illustrating the effectiveness of using scene graph as the bridge for cross-modal alignment rely solely on cross-attention mechanisms to capture the alignment between image and text features, and learn from indirect signals without any explicit supervisions. recently, oscar @xcite introduced object tags detected in images as anchor points to ease the learning of semantic alignments between image regions and word sequences. however, individual object tags in isolation ignore the rich visual information, such as attributes and relationships between objects. without such information as contextual cues, the core challenge of ambiguity in visual grounding remains difficult to solve. as figure 1 shows, in order to answer the question correctly, the model needs to reason about object relationships. without the relation "on" between "cup" and "table", the model mistakenly thinks the "cup" is on the "tray". this work tackles the above challenges by introducing visual scene graphs as the bridge to align vision-language semantics. extracted from the image using modern scene graph generators, a visual scene graph effectively depicts salient objects and their relationships. the visually-grounded intermediate abstraction permits more effective vision language cross attention for disambiguation and finer-grained alignment. specifically, we propose samformer (semantic aligned multi-modal trans-former) that learns the alignment between the modalities of text, image, and graphical structure. for each of object-relation labels in the scene graph, the model can easily find the referring text segments in natural language, and then learn to align to the image regions already associated with the scene graph. on the basis of the visually-grounded graph, we apply a contrastive loss and a masked language model loss that explicitly encourage image-text alignment. furthermore, we propose a per-triplet (object, relation, subject) contrastive loss to align object and relation representations across the two modalities respectively. we adopt a set of datasets, including microsoft coco captions dataset @xcite , visual genome @xcite , vqa @xcite , gqa @xcite , flicker 30k @xcite , sbu @xcite , and conceptual caption @xcite to pre-train our model and fine-tune it on visual compositional question answering (gqa) @xcite . our preliminary analyses show improved performance and demonstrate the potential of the proposed approach on broader visual-language applications. ## semantic aligned vision and language transformer this section presents the proposed semantic aligned multi-modal transformer (samformer) for visionlanguage pre-training. figure 2 provides an overall architectural view of the method. given a pair of an image i and a text sequence w describing the image, the goal of vision-language pre-training is to learn a joint representation of the pair which captures the alignment between the words and image regions and can be adapted to assist downstream tasks. same as the previous vision-language models @xcite , the proposed samformer first separately encodes each modality into singular embedding features, and then employs a multi-layer self-attention transformer to align the features and obtain a crossmodal contextualized representation. samformer differs critically from previous meth-ods in that we incorporate the visual scene graph extracted from the image to enhance the cross-modal representation learning. the structured, visuallygrounded graph encodes rich semantic information (e.g., objects, relationships), which, compared to isolated object tags @xcite and bare image text singular features @xcite @xcite , offers valuable cues to resolve ambiguity and bridge together text and visual semantics. we describe in details how visual scene graph is integrated to interplay with the text and image modalities for better alignment (section 2.1), and on this basis how contrastive learning strategies are devised for fine-grained alignment supervisions (section 2.2). ## conclusion and future work in this work, we propose samformer, a novel semantic aligned multi-modal transformer model for vision-language pre-training. we explicitly align the visual scene graphs and text using triplet tags as anchors as well as a contrastive loss between each triplet tags and its paired visual features. we show improved preliminary results on gqa. as shown in the empirical study, the performance is to some extent capped by the rather limited relations and object categories that can be extracted from off-the-shelf pre-trained scene graph models and object detectors. for future work, we plan to jointly train with scene graph models to more effectively learn from limited labeled data and weak supervision signals from paired text.