Finding Structural Knowledge in Multimodal- BERT (2022)

Abstract: In this work, we investigate the knowledge learned in the embeddings of multimodal-BERT models. More specifically, we probe their capabilities of storing the grammatical structure of linguistic data and the structure learned over objects in visual data. To reach that goal, we first make the inherent structure of language and visuals explicit by a dependency parse of the sentences that describe the image and by the dependencies between the object regions in the image, respectively. We call this explicit visual structure the scene tree, that is based on the dependency tree of the language description. Extensive probing experiments show that the multimodal-BERT models do not encode these scene trees.

## introduction in recent years, contextualized embeddings have become increasingly important. embeddings created by the bert model and its variants have been used to get state-of-the-art performance in many tasks @xcite @xcite @xcite . several multimodal-bert models have been developed that learn multimodal contextual embeddings through training jointly on linguistic data and visual data @xcite @xcite . they achieve state-ofthe-art results across many tasks and benchmarks, such as visual question answering @xcite , image and text retrieval @xcite , and visual commonsense reasoning @xcite . 1 bert and multimodal-berts are blackbox models that are not easily interpretable. it is not trivial to know what knowledge is encoded in the models and their embeddings. a common method for getting insight into the embeddings of both textual and visual content is probing. language utterances have an inherent grammatical structure that contributes to their meaning. natural images have a characteristic spatial structure that likewise allows humans to interpret their meaning. in this paper we hypothesize that the textual and visual embeddings learned from images that are paired with their descriptions encode structural knowledge of both the language and the visual data. our goal is to reveal this structural knowledge with the use of probing. more specifically, in order to perform this probing, we first make the inherent structure of language and visuals explicit by a mapping between a dependency parse of the sentences that describe the image and by the dependency between the object regions in the image, respectively. because the language truthfully describes the image, and inspired by @xcite , we define a visual structure that correlates with the dependency tree structure and that arranges object regions in the image in a tree structure. we call this visual dependency tree the scene tree. an example of this mapping to the scene tree is visualized in figure 1 . the aligned dependency tree and scene tree allow us to conduct a large set of experiments aimed at discovering encoded structures in neural representations obtained from multimodal-berts. by making use of the structural probes proposed by @xcite , we compare the dependency trees learned by models with or without provided image features. furthermore, we investigate if scene trees are learned in the object region embeddings. research questions in this study, we aim to answer the following research questions. in a broader framework this study might contribute to better representation learning inspired by how humans acquire language in a perceptual context. it stimulates the learning of representations that are compositional in nature and are jointly influenced by the structure of language and the corresponding structure of objects in visuals. ## related work probing studies several studies have been performed that aim at analyzing bert and multimodal-berts. for bert, probes are designed that explore gender bias @xcite , relational knowledge @xcite , linguistic knowledge for downstream tasks @xcite , part-of-speech knowledge @xcite , and for sentence and dependency structures @xcite . these studies have shown that bert latently learns to encode linguistic structures in its textual embeddings. @xcite made a first attempt at converting the probes to the visual modality and evaluated the information stored in the features created by visual models trained with self-supervision. for multimodal-berts, one study by @xcite investigates how well these models learn to count objects in images and how well they generalize to new quantities. they found that the multimodal-berts overfit the dataset bias and fail to generalize to out-of-distribution quantities. @xcite found that visual information is much more used for textual tasks than textual information is used for visual tasks when using multimodal models. these findings suggest more needed research into other capabilities of and knowledge in multimodal-bert embeddings. we build on this line of work but aim to discover structures encoded in the textual and visual embeddings learned with multimodal-berts. this is a first step towards finding an aligned structure between text and images. future work could exploit this to make textual information more useful for visual tasks. ## results and discussion this discussion is based on the results from the test split. the results on the validation split (see appendix b), lead to the same observations. rq 1: do the textual embeddings trained with a multimodal-bert retain their structural knowledge? to answer rq 1, we report the results for both structural probes on the ptb3 dataset. here we only use the textual embeddings, since no visual features are available. the results for the depth probe are in figure 2 , and for the distance probe in figure 3 . the results of both multimodal-berts (figures 2c and 3c for vilbert and figures 2b and 3b for uniter) in terms of nspr. and root acc are very comparable showing similar curves and scores. for both, the seventh layer is the best performing one. the shape of the curves across the layers is similar to those for the bert model in figures 2a and 3a . however, the scores of the multimodal-berts drop significantly. while the multimodal-berts were initialized with weights from bert, they were trained longer on additional multimodal data with a different multimodal objective. this shows that the multimodal training hampers the storing of grammatical structural knowledge in the resulting embeddings. sub-rq 1.1: to what extent does the joint training in a multimodal-bert influence the structures learned in the textual embeddings? for this experiment, we compare the effect of having visual features present when using the structural probes on the textual embeddings. we run the probes on flickr30k. the results for the depth probe are in figure 4 , and for the distance probe in figure 5 . first, we see that for all models (bert and multimodal-berts) the scores increase compared to the results on the ptb3 dataset (see discussion of rq 1), but still follow a similar trend across the layers. the latter is most likely due to the complexity of the sentences and language of the ptb3 dataset, which is simpler for the captions. for vilbert, there is a drop in performance for the earlier layers. we believe this is caused by the early stopping method firing early with these settings. another explanation is that it is more difficult for the dualstream model to use the additional parameters. bert outperforms the multimodal-berts on ptb3, however, this is not the case on flickr30k. for the depth probe (figure 4 ) and the uuas metric on the distance probe (figure 5 ), the results obtained on these two datasets are almost equal. this can be due to the additional pretraining of the multimodal-berts on similar captioning sentences. another explanation is that, during such pretraining, the models learned to store relevant information in the visual embeddings. we run an additional experiment where we use the pretrained multimodal-bert, but while probing we only provide the sentence to the model, and mask out the image. the results for the depth probe are in figure 6 , and for the distance probe in figure 7 . here we can see that the results are almost identical to when we provide the model with the visual embeddings. this indicates that the model does not have any benefit from the visual data when predicting the structures for textual embeddings, and it seems that the model uses the extra parameters of the vision layers to store knowledge about the text. rq 2: do the visual embeddings trained with a multimodal-bert learn to encode a scene tree? we aim to find the layer with the most structural knowledge learned when applied to multimodal data. see the results in figures 8 and 9 . regarding the results for the depth probe (figure 8 ), the scores between layers fluctuate inconsistently. the scores do improve slightly over the dspr. ## conclusion and future work we made a first attempt at investigating whether the current multimodal-bert models encode structural grammatical knowledge in their textual embeddings, in a similar way as text-only bert models encode this knowledge. furthermore, we were the first to investigate the existence of encoded structural compositional knowledge of the object regions in image embeddings. for this purpose, we created a novel scene tree structure that is mapped from the textual dependency tree of the paired caption. we discovered that the multimodal-berts encode less structural grammatical knowledge than bert. however, with image features present, it is still possible to achieve similar results. the cause for this requires more research. while tree depths from the scene tree are not natively present in the features, we found that this could be a potential method of finding connections and distances between regions, already decently predicted with the faster r-cnn features. the multimodal-bert models are currently trained with an objective that does not enforce the learning or storing of these types of structural information. hence we assume that the models learn to encode more abstract knowledge in their features. our work opens possibilities to further research on scene trees as a joint representation of object compositions in an image and the grammatical structure of its caption. furthermore, we recommend investigating the training of multimodal-berts with objectives that enforce the encoding of structural knowledge.