An Examination of the Compositionality of Large Generative Vision-Language Models (2024)

Abstract: With the success of Large Language Models (LLMs), many Generative Vision-Language Models (GVLMs) have been constructed via multimodal instruction tuning. However, the performance of GVLMs in multimodal compositional reasoning remains under-explored. In this paper, we examine both the evaluation metrics ( VisualGPTScore, etc.) and current benchmarks for evaluating the compositionality of GVLMs. We identify the syntactical bias in current benchmarks, which is exploited by the linguistic capability of GVLMs. The bias renders VisualGPTScore an insufficient metric for assessing GVLMs. To combat this, we first introduce a **SyntaxBias Score**, leveraging LLMs to quantify such bias for mitigation. A challenging new task is subsequently added to evaluate the robustness of GVLMs against inherent inclination toward syntactical correctness. Using the bias-mitigated datasets and the new task, we propose a novel benchmark, namely **S**ynt**A**ctically **DE**-biased benchmark (SADE). Our study provides an unbiased benchmark for the compositionality of GVLMs, facilitating future research in this direction. Code and dataset are available at https://github.com/TeleeMa/SADE.

## introduction a surge of research on vision-language models (vlms) has demonstrated success in a wide range of tasks, including zero-shot visual recognition @xcite @xcite , visual question answering @xcite , and image-to-text retrieval @xcite . previous vision-language models (vlms) have predominantly been developed using image-text contrastive (itc) learning @xcite @xcite and image-text matching (itm) (tan and bansal, 2019; chen et al., 2020; gan et al., 2020; li et al., 2021; zhang et al., 1 code and dataset are available at https://github.com/ teleema/sade . 2021; @xcite frameworks, a category we term encoder-based vision-language models (evlms). with the advent of large language models (llms) like chatgpt, @xcite and llama @xcite , recent studies have extended the decoder-only architecture to multimodal settings, which is named generative vlms (gvlms) @xcite @xcite @xcite @xcite . the gvlms deviate from the evlms in projecting visual features into the latent lexical space of llms, and leveraging the auto-regressive generative capacity to solve vision-language tasks. in the training process, most work follows the recipe of freezing the main body of visual encoders and llms, only updating the negligible parameters of projecting layers, which is also called "bridge architecture" @xcite . despite the emergence of research on gvlms, the understanding of compositionality in gvlms has remained an enigmatic black box, with no thorough investigations conducted thus far. previous research studies @xcite @xcite @xcite in multimodal compositionality focus on establishing retrieval-based benchmarks for evaluating evlms on object relations and attribute understanding, order sensitiveness of sentence elements, and atom-level understanding. the evlms have demonstrated abilities to discriminate positive captions from negative ones based on the image-text similarity, where the disparities between the positive and negative captions are relatively subtle, such as "an old person kisses a young person" and "a young person kisses an old person" @xcite . however, we observe there exists an underlying bias towards the llm part of gvlms in the evaluation of the aforementioned benchmarks. during the evaluation, the log-likelihood-based scores are widely adopted to evaluate the generative models @xcite @xcite to estimate the conditional probabilities of specific generations. following @xcite , we alias the log-likelihood score as visualgptscore. we examine the current benchmarks for evaluating gvlms with visual-gptscore and find that: based on these observations, our contributions include: ## experimental setup we introduce the configurations of experiments for the syntactical bias examination in this section. ## evaluation metric examination visualgptscore measures the probability of generating specific references conditioned on the given images, as defined in eqn. 3. the generative evaluation method is based on the inherent attribute of gvlms and used in image-text retrieval @xcite @xcite . since current benchmarks on vl compositions consists of image-text pairs, we follow @xcite to utilize visualgptscore for evaluating the vl compositionality of gvlms. in this section, our primary focus is to examine the bias of using visu-algptscore in current benchmarks. ## benchmarks examination from above, we know current benchmarks are curated for evaluating evlms based on similarity score originally. hence, we examine the impact of using these datasets for evaluating gvlms with visualgptscore, and uncover the bias of existing datasets. ## mitigate the bias in benchmarks in this section, we propose a strategy to modify the benchmarks and mitigate the syntactical bias to provide a better evaluation of gvlms. specifically, we filter current datasets leveraging llms and add a novel challenge to evaluate visual content understanding. we name the new benchmark as syntactical de-biased benchmark, abbreviated as sade. in the following, we describe the filtering details of each dataset and the new challenge. then we show human evaluation to show the effectiveness of sade. ## conclusion in this work, we evaluate the compositionality of "bridge-architecture" generative vlms via generative multimodal score, visualgptscore. we examine both the visualgptscore and current benchmarks for evaluating the multimodal compositional understanding of gvlms. based on the examinations, we identify the syntactical bias that exists in current datasets for gvlms, and define the bias with syntaxbias score quantitatively. we then propose a sade benchmark that mitigates the syntactical bias and provides a better content understanding evaluation for gvlms. we report the results of multiple gvlms on our proposed sade benchmark and uncover new findings of the gvlms' capabilities.