A Subspace-Based Analysis of Structured and Unstructured Representations in Image-Text Retrieval (2022)

Abstract: In this paper, we specifically look at the image-text retrieval problem. Recent multimodal frameworks have shown that structured inputs and fine-tuning lead to consistent performance improvement. However, this paradigm has been challenged recently with newer Transformer-based models that can reach zero-shot state-of-the-art results despite not explicitly using structured data during pre-training. Since such strategies lead to increased computational resources, we seek to better understand their role in image-text retrieval by analyzing visual and text representations extracted with three multimodal frameworks – SGM, UNITER, and CLIP. To perform such analysis, we represent a single image or text as low-dimensional linear subspaces and perform retrieval based on subspace similarity. We chose this representation as subspaces give us the flexibility to model an entity based on feature sets, allowing us to observe how integrating or reducing information changes the representation of each entity. We analyze the performance of the selected models’ features on two standard benchmark datasets. Our results indicate that heavily pre-training models can already lead to features with critical information representing each entity, with zero-shot UNITER features performing consistently better than fine-tuned features. Furthermore, while models can benefit from structured inputs, learning representations for objects and relationships separately, such as in SGM, likely causes a loss of crucial contextual information needed to obtain a compact cluster that can effectively represent a single entity.

## introduction the integration of techniques from natural language processing (nlp) and computer vision (cv) has led to the development of multimodal approaches, which have quickly attracted the scientific community's attention. examples include tasks such as image captioning @xcite , machine translation @xcite , word sense disambiguation @xcite , and visual question answering @xcite . great progress in these tasks has been made by using massive amounts of training data with deeper models, leading to rapidly increasing computational costs. in this paper, we specifically look at the imagetext retrieval task, where the goal is to retrieve an image from a text query (image retrieval) or a text from an image query (text retrieval) from a database containing images and texts. in this context, we see a line of works encoding local and global structures to learn representations for both modalities, extracted using object detectors @xcite and large pre-trained language models @xcite . to further understand the relationship between such structures, several works also encoded visual @xcite and textual @xcite scene-graphs or designed their pipelines to learn such graphs @xcite . a more recent trend has been to use transformerbased models to learn the representations for each modality and to model their interaction @xcite , also making use of such structured data @xcite . while these frameworks have resulted in state-ofthe-art performance in multiple downstream tasks, including image-text retrieval, the inference is computationally expensive for this task as it requires a forward pass of each image-text pair in the database to perform retrieval. although structured inputs and fine-tuning have shown consistent performance improvement across all the aforementioned models, this paradigm has been challenged recently with newer transformerbased models, such as clip @xcite . this model, for example, can not only reduce the computational inference overhead by allowing the images and texts to be processed individually, but it also achieves zero-shot state-of-the-art results for image-text retrieval despite not explicitly using structured data during its pre-training. in light of these issues, this paper analyzes visual and text representations produced by several multimodal frameworks in the task of image-text retrieval. we are particularly interested in studying the ability of these models in encoding relevant information to perform retrieval in a variety of scenarios, including model fine-tuning versus zero-shot performance for models that require pre-training, as well as how the addition or removal of structure information from images (e.g., scene-graphs) and texts (e.g., semantic triplets), affects such representations. we find it pivotal to understand the role of such strategies as their integration ultimately leads to increased computational resources. to perform such an analysis, we set a common ground by looking at subspace representations in the context of image-text retrieval. in the subspace setting, the idea is to represent a single entity, e.g., an image or a sentence, as a low-dimensional linear subspace in the original high-dimensional feature space and to perform retrieval based on subspace similarity. such representation is based on the empirical evidence that patterns of the same entity (e.g., pictures of the same person) tend to cluster in high-dimensional space @xcite . we expect features from the same entity learned by such multimodal frameworks also form these compact clusters, and therefore their distribution can be represented by linear subspaces. furthermore, as most image-text retrieval frameworks rely on the cosine similarity between feature vectors to compare two entities, the subspace similarity comes in handy as it is equivalent to cosine similarity when we have onedimensional subspaces (i.e., a single vector representing an entity). finally, subspaces give us the flexibility to model an entity based on a set of vectors, e.g., a set of object embeddings in an image or set of entities in a sentence, allowing us to observe how integrating more information by fine-tuning or adding structure data, changes the representation of each entity. this paper focuses on frameworks that explicitly incorporate or capture structured inputs, either from the visual or textual side. concretely, we evaluate and compare the text-image retrieval performance using the subspace representation of features extracted using three frameworks: sgm @xcite , uniter @xcite , and clip @xcite . we chose these three models based on the distinct way they treat multimodal data: sgm, a scene graph-based model, heavily relies on structured data, generating object-level and relationship-level cross-modal features; uniter, a pre-trained transformer-based model that generates joint visual and textual embeddings relying on objects detected on the input images; and clip, a pre-trained contrastive model which is trained by simply pairing whole images with complete sentences and without making explicit use of structure, which also allows us to extract of image and text embeddings individually in a zero-shot fashion, overcoming the limitations of previous models such as uniter. we analyze the performance of feature subspaces of selected models on two standard benchmark datasets, coco @xcite and flickr30k @xcite , focusing on the tasks of image-to-text and text-to-image retrieval. furthermore, we observe how results change when modeling pre-trained and fine-tuned features from uniter and introducing or removing structure information from sgm and clip features. our results indicate that uniter's pre-training leads to features with critical information representing each entity during pre-training, with zero-shot features performing consistently better than fine-tuned features. moreover, we observed that learning representations for objects and relationships separately, such as in sgm, likely causes a loss of crucial contextual information needed to effectively represent a single entity, whereas using only sgm's object representations led to better performance. this result might explain why clip features can better characterize entities when features are extracted based on global features, where we observed that explicitly considering local structure information harms retrieval performance. ## experimental framework we experimented with image-text retrieval on two datasets, flickr30k @xcite (flickr30k) and coco @xcite . both datasets contain 5 captions (i.e., text passages) for each image. however, they differ in one order of magnitude regarding the number of examples (approx. 300k images on cocoand approx. 30k on flickr30k). because of this reason, in order to keep computational costs within our budget, we used flickr30k to extensively study multiple settings and selected only the best configurations for our experiments with coco. in all cases, each image and caption is represented by a single or several feature vectors, and retrieval is performed using sm and msm as defined earlier. our evaluation is performed based on the r@k metric, the percentage of queries whose groundtruth is ranked within the top k, which is the standard for the task. we experimented using different subspaces' dimensions and report the best results. below, we give details about how our multi-modal features are extracted for each model. sgm with this model, we are particularly interested in understanding how considering objects and their relationships affects retrieval. to extract the features, we use the model checkpoints trained on both datasets provided by the authors. each image is represented by one set of visual object features @xmath0, and one set of visual relation features @xmath1. each caption is represented by one set of textual object features @xmath2. considering we have two sets of features representing each visual and textual entity, we followed the same strategy taken by sgm when performing retrieval by calculating s o and s r based on subspace similarity, and then summing both to achieve the final similarity for the pair s o,r . to better understand the role of each set of features in representing an entity, we also performed retrieval based only on s o , only on s r , and on s g , which represents each entity by the concatenation of the object and relation features. uniter as uniter's excellent performance is due mostly to its extensive pre-training and finetuning, we are interested in comparing the retrieval performance of pre-trained features versus finetuned features in image-text retrieval. we feed positive image-caption pairs through the model to obtain their joint representations (i.e., sequence of vectors). we split each sequence to obtain one set of features @xmath3, and one set of features @xmath4. for the captions, we disregarded the representation for the [sep] token. while we understand that processing only positive image-caption pairs is not the ideal approach to perform retrieval, we reckon this is a limitation of uniter, as it requires an image and a text passage to be fed simultaneously. ideally, we would like to be able to forward each image and text only once and perform a ranking on top of the obtained representations. we performed preliminary experiments feeding only captions and only images, but the results showed that this approach does not create meaningful representations. therefore, since we want to observe the effects of fine-tuning on the multi-modal representations, we primarily focus on the performance difference between them rather than the actual numbers. we use the pre-trained uniter released by the authors and test it on three different settings: zero-shot (zs) where we directly use the pretrained uniter to extract our representations; fine-tuned (ft), where we further train the pretrained model on the downstream dataset with the default sampling strategy; and another fine-tuned model where the final training is performed using an improved technique for hard negative example mining (ft hn ). we note that the latter strategy has resulted in the best retrieval performance for the original model. ## conclusions and future work the main goal of this paper was to better understand the role of structured inputs and fine-tuning in image-text retrieval. we analyzed visual and text representations extracted with sgm, uniter, and clip by representing a single image or text as low-dimensional linear subspaces and performing retrieval based on subspace similarity. we analyzed how the performance of the selected models' features changed when considering fine-tuning versus zero-shot performance for models that require pretraining, as well as the addition or removal of structure information from images (e.g., scene-graphs) and texts (e.g., semantic triplets). our results indicate that uniter's pre-training leads to features with critical information representing each entity during pre-training, with zero-shot features performing consistently better than finetuned features. moreover, we observed that using only sgm's object representations led to better performance than when considering the relationship representations. finally, considering structure information with clip does not improve the retrieval results. however, we could observe that global information from the text side seems more critical than text local information. a natural progression of this work is to analyze these features from a geometrical perspective, using the well-established literature on subspace representation.