Probing the Role of Positional Information in Vision-Language Models (2022)

Abstract: In most Vision-Language models (VL), the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we probe the use of the PI in the representation and study its effect on Visual Question Answering. We show that the model is not capable of leveraging the PI for the image-text matching task on a challenge set where only position differs. Yet, our experiments with probing confirm that the PI is indeed present in the representation. We introduce two strategies to tackle this: (i) Positional Information Pre-training and (ii) Contrastive Learning on PI using Cross-Modality Matching. Doing so, the model can correctly classify if images with detailed PI statements match. Additionally to the 2D information from bounding boxes, we introduce the object’s depth as new feature for a better object localization in the space. Even though we were able to improve the model properties as defined by our probes, it only has a negligible effect on the downstream performance. Our results thus highlight an important issue of multimodal modeling: the mere presence of information detectable by a probing classifier is not a guarantee that the information is available in a cross-modal setup.

## introduction pre-trained vision-language models @xcite @xcite reached strong performance in many multimodal tasks such as visual question answering @xcite @xcite or visual inference @xcite . all these models use the transformer architecture @xcite and make use of several pre-training strategies like masked cross-modality language modeling (mm) and cross-modality matching (cmm) similar to masked language modeling and next sentence prediction @xcite in nlp. because the attention mechanism treats its inputs as unordered sets, transformer-based nlp models need to use position encodings to represent the mutual position of the tokens, so that the models can grasp the sentence structure. the mutual position of objects is equally important for understanding the structure of an image. vl models differ in how they represent objects in the image, typically represented as sets of object features and pi. therefore, object detectors are used to obtain bounding box information for all objects. in many models, the upper left and lower right corners of the object's bounding box are used as 2d information to create a learnable positional encoding. in addition to the spatial but flat 2d values, we determine the depth of the objects in the image and make it available as an additional feature. until now, vl models recognize the objects on a flat map but not in the real three-dimensional context. we found that the current lxmert model is capable of forwarding pi through the model but is not capable to use it to solve image-text matching tasks where positional keywords are replaced by their counterparts. introducing two new pretraining strategies, we target these unimodal and multimodal evaluation schemes and improve probing results. yet, we did not get any perfomance increase on the downstream tasks. this is most likely due to the small fraction of position-related text in the pre-training corpus and suboptimal results of the object detector. regarding pi type, it seems to be sufficient to input object center values which is far less than most vl model input today. ## positional information in vl models in nlp, the importance of word order is given great attention @xcite . different methods exist, including analytical position encodings @xcite , learnable additive embeddings @xcite or the relative the attention query @xcite . there is no equivalent research that would specifically approach pi in vl models. however, the position of the objects is considered in almost all common transformer-based approaches. in lxmert @xcite the upper left and lower right corners of the object are used to encode its position. the same is true for m4c @xcite . other models also use the relative area fraction of the objects as an additional feature. although the network should be able to determine this feature, it is explicitly added, as in case of vilbert @xcite , unicoder-vl @xcite , and ernie-vil @xcite . uniter @xcite uses -in addition to the objects' corners -the absolute object area and the image width and height. oscar @xcite uses bounding box and image height and width. only clip @xcite does not use pi, although they use another pre-training concept. see current models use only 2d object information. by introducing depth as a new feature, we represent objects in the 3d space. this is not only important to be able to define the distances between objects but also to have a more meaningful understanding of the object sizes. using the area of the bounding box without depth information does not add the actual object size information since the sizes depend on the depth localization of the object. ## evaluation of positional information to determine the capability of current models with regard to pi, we experiment with three evaluation methods. firstly, we perform an intrinsic evaluation to determine whether the pi passes through the model. secondly, we test if the models are capable of utilizing pi using the cmm task. lastly, we report extrinsic results for gqa downstream task @xcite on different data subsets. we report the results of the probing experiment in section 5. for our experiments, we use four types of pi. an empty set (∅) acts as a baseline. object center values (x, y) act as a coarse identification of where the object is located. moreover, we evaluate x1, y1, x2, y2, which is the standard representation of bounding boxes and is also often used in vl models. this pi description contains information about object width, height, and area. therefore, we ignore further settings that add these types to the input in our evaluation. since we are also interested in analyzing depth, we investigate the setting x1, y1, x2, y2, d as well. ## probing results this section shows the results of the experiments described in section 3. ## from probing to pre-training in the previous section, we evaluated the role of pi in pre-trained lxmert. in this section, we use the probing tasks as a part of model pre-training to improve weaknesses that we identified in the previous section. alongside the established strategies, we add two tasks to learn mutual positions and fine-grained pi details in captions utilizing the cmm task. these strategies are elaborated below. ## conclusions current vl models make use of different pi inputs without evaluating their impact. in our work, we inspect the effect of such pi input types and investigate depth as a new input extension. in the original setting, the model is able to forward the positional information through the whole transformer layer stack, but it cannot utilize it in the contrastive evaluation and only marginally in the downstream task. overall, having any type of pi is helpful, though object-center values are often sufficient. however, object features f j are already good proxies for where objects are located. because this can be based on spurious correlations, we propose pre-training methods that should make the model rely on pi directly. we introduced two new pre-training strategies. firstly, positional information pre-training to ensure that data is passed through the model properly and does not need to rely on feature correlations. this operates on visual component only and increases performance on the corresponding intrinsic evaluation task. moreover, we introduce contrastive learning on pi using cmm. in doing so, we connect pi in the textual and visual modality. as a result, the model is now able to succeed in the contrastive evaluation task. however, these improvements do not affect the downstream performance on gqa. it is not enough to add different features unchecked, trusting they are properly utilized by the transformer. in line with bertology @xcite @xcite , studies are important to understand better what a model is capable. the same is true for pre-training strategies. it is not sufficient to add new pre-training strategies, although they look promising. with our probing experiments, we tried to receive a better understanding of the inner workings of lxmert. we see the importance to investigate differences between general concepts and impact on a downstream task. we see two major issues for pi in vl models. firstly, the pre-training data contains too little fraction of sentences with pi content. hence, especially the cl pre-training strategy has not enough samples to learn from. secondly, the used object detector is not very powerful (see predictions in figure 5 ). newer detection models like vinvl @xcite might help to have a better image representation, which consequently leverages performance regarding pi context.