Probing Cross-modal Semantics Alignment Capability from the Textual Perspective (2022)

Abstract: In recent years, vision and language pre-training (VLP) models have advanced the state-of-the-art results in a variety of cross-modal downstream tasks. Aligning cross-modal semantics is claimed to be one of the essential capabilities of VLP models. However, it still remains unclear about the inner working mechanism of alignment in VLP models. In this paper, we propose a new probing method that is based on image captioning to first empirically study the cross-modal semantics alignment of VLP models. Our probing method is built upon the fact that given an image-caption pair, the VLP models will give a score, indicating how well two modalities are aligned; maximizing such scores will generate sentences that VLP models believe are of good alignment. Analyzing these sentences thus will reveal in what way different modalities are aligned and how well these alignments are in VLP models. We apply our probing method to five popular VLP models, including UNITER, ROSITA, ViLBERT, CLIP, and LXMERT, and provide a comprehensive analysis of the generated captions guided by these models. Our results show that VLP models (1) focus more on just aligning objects with visual words, while neglecting global semantics; (2) prefer fixed sentence patterns, thus ignoring more important textual information including fluency and grammar; and (3) deem the captions with more visual words are better aligned with images. These findings indicate that VLP models still have weaknesses in cross-modal semantics alignment and we hope this work will draw researchers’ attention to such problems when designing a new VLP model.

## introduction which caption matches the image better? (a) a bunch of bananas on a table (b) a bunch of the bananas table (c) a closeup image that is a a bunch but a a bananas but a a table (d) bunch that the bananas at the table (e) a image image of a bunch near a bananas near a table (f) a bunch of bananas of a table models choice uniter (b) rosita (c) vilbert (d) clip (e) lxmert (f) human (a) although vlp models have excellent performance on a large panoply of cross-modal tasks, it remains unclear which information vlp models have aligned between different modalities. some studies have investigated this problem and designed a series of probing tasks to explore it @xcite . however, these probing tasks almost probe vlp models based on the classification task, such as classifying or counting objects and recognizing consistencies between regions and phrases. we argue that the simple classified probing tasks fail to explicitly explore vlp models' inner alignment mechanism, as they only require models to use partial information, such as a region, a phrase, or both, to make a decision. to illustrate this point, in without doubt, it is too difficult and nearly impossible to find out all unreasonable captions the models favor, by first generating some manually and then testing them, as in fact, we do not know the model's preferences beforehand. however, if we can use the matching score from vlp models as a signal to train a cross-modal generated model, then maximizing such scores will amplify the models' preferences and reflect them in the generated sentences. inspired by this idea, in this paper we present the first work of using a novel probing method to empirically study cross-modal alignment via the image captioning task. the generated captions from our probing method have higher matching scores, which are considered to fit images better and contain important alignment information that the vlp models focus on. we then apply our probing method to five powerful and representative vlp models, including uniter, rosita, vilbert, clip, and lxmert, which cover two mainstream architectures and two mainstream alignment tasks. our approach explicitly maps the alignment information to the generated captions. through analyzing these captions, we discover that objects and visual words tend to receive more attention in vlp models' inner alignment mechanism, and the more nouns the captions contain, the more vlp models deem they match images. it indicates these vlp models are overwhelmingly dependent on partial information (objects in images and visual words in captions), rather than the whole semantics of images and captions when they judge whether image-caption pairs are aligned. moreover, we find that these models favor certain sentence patterns (details in section 4). captions that follow these patterns will be considered more consistent with images. it suggests when deciding whether image-caption pairs are aligned, vlp models ignore more important textual information, such as fluency and grammar. ## related work there has been an increasing number of papers studying how different modalities are aligned in vlp models. we now review recent representative studies. @xcite report the dominance of textual modality with value, a comprehensive framework they introduce including multiple probing tasks. @xcite design three probing tasks to analyze linguistic properties of multi-modal embeddings, such as estimating the number of object instances in the image. all these tasks employ a simple neural network classifier to probe the ability of vlp models in certain aspects or reveal the importance of textual compared with visual information. @xcite evaluate vlp models on count, a task requiring the model to correctly predict the number of objects in an image, and find several prevailing vlp models that fail to identify entities in an image. moreover, @xcite propose valse to test vlp models for their vision-linguistic grounding capabilities on specific linguistic phenomena and find that vlp models struggle to ground their interdependence and relationships in visual scenes when forced to respect linguistic indicators. it is also suggested that more targeted investigations are needed to probe the cross-modal alignment capacity of vlp models. instead of the previous simple classification-based method, in this work, we introduce a new probing method based on the image captioning model, targeting for the cross-modal alignment. as we will discuss in the following sections, our approach is able to generate sentences that enable a more transparent analysis towards the alignment of visual and textual modality. ## limitations of current vlp models the above issues reveal that vlp models tend to prefer the captions with fixed sentence patterns and more visual words. in this section, we design a set of experiments to verify this phenomenon and further discuss the problems of current vlp models. we argue that these limitations have potentially hindered vlp models from aligning visual and textual modalities authentically. ## conclusion in this paper, we empirically study the cross-modal semantics alignment capability of vlp models, using a newly proposed probing method via an image captioning model. by analyzing the issues of generated captioning guided by the vlp models, we find that vlp models have particular weaknesses in cross-modal semantics alignment, including paying more attention to aligning objects and visual words, while neglecting global semantics; preferring fixed sentence patterns; and considering captions with more visual words are better aligned. we hope our work sheds light on promoting better architecture or pre-training tasks for crossmodal semantics alignment to overcome these limitations. researchers can also use our probing method to discover potential problems when designing new vlp models.