C ounter C urate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples (2024)

Abstract: We propose CounterCurate, a framework to comprehensively improve the visio-linguistic compositional reasoning capability for both contrastive and generative multimodal models. In particular, we identify two critical under- explored problems: the neglect of physically grounded reasoning (counting and position understanding) and the potential of using highly capable text and image generation models for semantic counterfactual fine-tuning. Our work pioneers an approach in addressing these gaps.We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning. We then apply simple data augmentation using the grounded image generation model GLIGEN to generate fine-tuning data, resulting in significant performance improvements: +33% and +37% for CLIP and LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark. Moreover, we exploit the capabilities of high-performing text generation and image generation models, specifically GPT-4V and DALLE-3, to curate challenging semantic counterfactuals, thereby further enhancing compositional reasoning capabilities on benchmarks such as SugarCrepe, where CounterCurate outperforms GPT-4V.To facilitate future research, we release ourcode, dataset, benchmark, and checkpoints at https://countercurate.github.io/

## introduction large language models such as chatgpt (openai, 2023a), gpt4 @xcite , and llama @xcite have demonstrated remarkable knowledge and reasoning abilities. recently, large multimodal models (lmms) @xcite further leverage large-scale imagetext pairs for improving visio-linguistic understanding capabilities. however, these models exhibit ## experiments we utilize the proposed three datasets, flickr30k-positions, flickr30k-counting, and flickr30k-attributes, created via countercurate, to fine-tune both contrastive and generative models. specifically, we select two common multimodal models, vit-b/32 from @xcite openclip pretrained on laion-2b @xcite and llava-1.5 @xcite as base models. an overview of our results is shown in figure 5 . ## conclusion in conclusion, countercurate significantly enhances the visio-linguistic compositional reasoning capabilities of multimodal contrastive and generative models. this is achieved by addressing the neglect of physically grounded reasoning and exploiting the potential of using text and image generation models for semantic counterfactual fine-tuning. we believe our contributions can pave the way for further research in compositional reasoning. all in all, the new description must meet all of these requirements: here are some examples whose output format you should follow: original caption: a child in a pink dress is helping a baby in a blue dress climb up a set of stairs in an entry way. enhanced caption: [/en#1/people a child] in [/en#2/clothing a pink blouse] helping [/en#3/people a baby] in [/en#4/clothing a blue dress] climb up [/en#5/other a set of stairs] in [/en#6/scene an entry way]. bounding boxes: #1: