Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? (2022)

Abstract: Recent advances in vision-and-language modeling have seen the development of Transformer architectures that achieve remarkable performance on multimodal reasoning tasks.Yet, the exact capabilities of these black-box models are still poorly understood. While much of previous work has focused on studying their ability to learn meaning at the word-level, their ability to track syntactic dependencies between words has received less attention.We take a first step in closing this gap by creating a new multimodal task targeted at evaluating understanding of predicate-noun dependencies in a controlled setup.We evaluate a range of state-of-the-art models and find that their performance on the task varies considerably, with some models performing relatively well and others at chance level. In an effort to explain this variability, our analyses indicate that the quality (and not only sheer quantity) of pretraining data is essential. Additionally, the best performing models leverage fine-grained multimodal pretraining objectives in addition to the standard image-text matching objectives.This study highlights that targeted and controlled evaluations are a crucial step for a precise and rigorous test of the multimodal knowledge of vision-and-language models.

## introduction vision-and-language (v&l) models have recently shown substantial improvement on a range of multimodal reasoning tasks. taking inspiration from successes in text-only natural language processing @xcite , stateof-the-art v&l models are usually composed of a transformer-based architecture pre-trained in a self-supervised manner on large-scale data, and then fine-tuned on downstream tasks. while these models show remarkable performance on a range of tasks, more controlled and systematic analyses are necessary in order to obtain target sentence: a man is wearing a hat. ## conclusion this work examines whether state-of-the-art v&l models learn multimodal syntactic dependencies, by focusing on a case study on simple predicatenoun dependencies. our controlled experiments and analyses on a range of recent models reveal that their capability track such dependencies is variable, with some models (e.g., lxmert and uniter) show performance above chance level and others (e.g., clip) performing even below chance. in contrast to the recent trend in the field focused on increasing pretraining data and using simple general-purpose pretraining objectives @xcite , here we observe that best performance is achieved, rather, with highquality pretraining data, and more fine-grained pretraining objectives. more specifically, our results suggest that multimodal pretraining objectives have a major impact on the model's learning of grounded predicatenoun dependencies. models that include more targeted objectives such as visual question answering and word region alignment in addition to the general image-text matching objective show better performance. in addition, having highly descriptive pretraining datasets seems to help with learning fine-grained multimodal dependencies. in comparison, models trained on larger, web-scraped datasets do not perform well. in the future, the proposed highly-controlled evaluation protocol can be used to conduct more targeted studies regarding the role of model architecture, pretraining objectives, as well as training data quality and quantity in order to build v&l models that are better at learning grounded predicate-noun dependencies, and possibly also other, move advanced multimodal reasoning tasks.