Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers (2024)

Abstract: An effective method for combining frozen large language models (LLM) and visual encoders involves a resampler module that creates a ‘visual prompt’ which is provided to the LLM, along with the textual prompt. While this approach has enabled impressive performance across many coarse-grained tasks like image captioning and visual question answering, more fine-grained tasks that require spatial understanding have not been thoroughly examined. In this paper, we use diagnostic classifiers to measure the extent to which the visual prompt produced by the resampler encodes spatial information. Our results show that this information is largely absent from the resampler output when kept frozen during training of the classifiers. However, when the resampler and classifier are trained jointly, we observe a significant performance boost. This shows that the compression achieved by the resamplers can in principle encode the requisite spatial information, but that more object-aware objectives are needed at the pretraining stage to facilitate this capability.

## introduction recent approaches for developing vision and language (v&l) models leverage existing vision @xcite , and language experts @xcite @xcite and try to learn a mapping between them @xcite @xcite @xcite . in most cases, the experts are kept frozen while the only learnable component is the mapping between the visual and the language expert. the simplest approach uses a linear projection layer that matches the dimensionality of the visual and textual embeddings before feeding them to the llm @xcite . a more sophisticated 1 code available here ❄resampler probe the phrase 'strawberries and cream in an fruit/snacks tray' refers to the top left part of the image. ## related work resamplers the idea of the resampler is inspired primarily by computer vision, where an attention mechanism is used to compress visual features into learnable queries (often referred to as slots) @xcite @xcite . more recently, resamplers have been applied to more multimodal tasks. flamingo @xcite and subsequent open-source variants @xcite are based on the perceiver resampler @xcite , with cross-attention between the latent queries and the visual embeddings followed by a stack of selfattention blocks that operate on the latent queries. in the q-former @xcite , the latent queries are also informed by the input text and, therefore, create a more 'linguistically informed' visual prompt. probing probing is a class of methods for interpreting neural models by assessing whether the model representations encode specific kinds of information at different processing stages @xcite . the concept of probing is straightforward; we extract representations from a model that is already trained on some task(s), and use a lightweight diagnostic classifier on top of these representations to solve a probing task that reflects the information that we seek to find. the classifier's performance is then taken to correlate with the extent to which that information is encoded by the model @xcite . many within (multimodal) nlp have thus adopted probing to interpret model behavior @xcite @xcite . ## experiments is spatial understanding a property of v&l resamplers? we experiment with three different spatial understanding tasks. in refcocog @xcite , the objective is to predict the coordinates of the region that is described by the input phrase. secondly, we use the 'random split' from the vsr dataset @xcite , where the model has to assess the validity of a caption describing a spatial relationship between two entities. finally, we introduce the region cell matching (rcm) task, which follows the vsr formulation but is designed to test for a more rudimentary form of spatial understanding regarding the location of one entity in the image. inspired by captchas, an image is divided into a 3x3 grid, and each grid cell is assigned a location description (such as top left or middle). we generate synthetic captions by combining refcocog descriptions with the cell location as shown in the implicit probing example of figure 1 . to ensure that performance is not influenced by frequency biases, we balanced the distribution of positive and negative examples. appendix a contains further details about the dataset. in our experiments, we use the q-former from the first pretraining stage of blip2 @xcite and instructblip @xcite . to probe the resamplers, we follow past work @xcite and use a single linear layer after flattening the embeddings of the query tokens. for ref-cocog, the linear layer predicts the normalized coordinates of the region that matches the referring expression. we use the bounding box loss from (m)detr @xcite : a weighted sum of the generalised iou and l1 losses. similarly, for vsr and the rcm task, we use a linear layer that predicts the probability that the query matches the image trained using binary cross entropy. we tune the learning rate, number of epochs, and loss weights (only for ref-cocog) using bayesian hyperparameter optimization @xcite for at least ten iterations. for further implementation details, see appendix b. in all cases, we evaluate the best model in terms of validation performance. we compare the two resamplers against similarly-sized models that employ patch representations. we avoid comparison against models with object-centric visual encoding because the task of visual grounding is significantly easier in these models as they need to select the correct can1 2 3 4 5 6 7 8 9 10 11 layer 45 50 55 60 65 70 accuracy performance on vsr using intermediate layers frozen q-former frozen iblip q-former (a) p e rs o n a n im a l v e h ic le e le c tr o n ic a p p li a n c e in d o o r k it c h e n fu rn it u re fo o d a c c e s s o ry o u td o o r s p o rt s super-category 0 20 40 60 80 100 accuracy@iou0 is spatial information encoded in earlier layers but discarded in deeper layers? we previously observed that resamplers have poor performance in spatial understanding tasks when using representations from the last layer. next, we examine if the representations from intermediate layers better encode spatial information. intuitively, representations from earlier layers could lead to greater probing performance as they are closer to the visual encoder's output. figure 2a shows the results on vsr after probing representations from intermediate layers. overall, intermediate layer representations do not provide performance gains. there is a clear upward trend regarding the performance of the q-former from blip2, whereas for instruct-blip we observe fluctuations within a small range across layers. a similar trend is observed in the re-fcocog results which are included in appendix c. scaling the probing classifier additionally, we experiment with scaling the probe classifier by introducing non-linearities. in particular, we use 2-layer and 4-layer classifiers with swiglu activation functions. we refrain from using more complex classifiers because they may infer features that are not actually used by the underlying model @xcite . for training, we used the same setup as with our previous experiments. ## conclusion in this paper, we explored to what degree multimodal resamplers preserve spatial information. while previous work has demonstrated the effectiveness of resamplers across a variety of v&l tasks, our investigation revealed their limitations when applied to spatial understanding tasks. in particular, we probed two resamplers and showcased that grounding natural language descriptions in image regions is not an inherent ability of these modules. furthermore, probing experiments showed limited spatial understanding in two easier settings. these involved image-text matching with captions referencing the absolute location of an entity, or spatial relationships between two entities. nevertheless, our results showcased that when the resampler is fine-tuned, the compression of the visual encoding induced by the resampler can be effective. we believe that this is due to the lack of an object-aware pretraining objective that would encourage the resamplers to encode spatial information. future work should build upon our findings and design objectives that incentivize disentangled representations @xcite .