CELLO : Causal Evaluation of Large Vision-Language Models (2024)

Abstract: Causal reasoning is fundamental to human intelligence and crucial for effective decision-making in real-world environments. Despite recent advancements in large vision-language models (LVLMs), their ability to comprehend causality remains unclear. Previous work typically focuses on commonsense causality between events and/or actions, which is insufficient for applications like embodied agents and lacks the explicitly defined causal graphs required for formal causal reasoning. To overcome these limitations, we introduce a fine-grained and unified definition of causality involving interactions between humans and/or objects. Building on the definition, we construct a novel dataset, CELLO, consisting of 14,094 causal questions across all four levels of causality: discovery, association, intervention, and counterfactual. This dataset surpasses traditional commonsense causality by including explicit causal graphs that detail the interactions between humans and objects. Extensive experiments on CELLO reveal that current LVLMs still struggle with causal reasoning tasks, but they can benefit significantly from our proposed CELLO-CoT, a causally inspired chain-of-thought prompting strategy. Both quantitative and qualitative analyses from this study provide valuable insights for future research. Our project page is at https://github.com/OpenCausaLab/CELLO.

## introduction causal reasoning is recognized as a fundamental component of human intelligence @xcite . recent advances in large language models (llms) have promoted a surge of research successfully adapting llms to vision-language tasks, resulting in powerful large causal relations question: a child needs to reach something high. can you move this chair for her to use? yes, i can help. ## causality in vision-language context we introduce a fine-grained and unified definition of causality in the vision-language context, inspired by the concept of causal dispositions @xcite . we propose that a causal relation between entities exists when one entity influences the state of another. to be specific, a causal effect is present if one entity causes another to sustain its current state. this can be further explicated through counterfactual reasoning: if the "cause" entity were absent, the "effect" entity would not continue in its current state. as shown in figure 2 , we identify three distinct categories of causal relations in a scene: object-object, human-object, and human-human causal relations. object-object causal relation. this represents interactions between objects, such as "the stick holding the balloon." without the stick, the balloon would not be in its current position (attached to the stick). hence, the stick causes the balloon to maintain its current state. identifying these relationships is crucial for understanding the physical interactions and dependencies within a scene. human-object causal relation. this involves interactions between humans and objects, such as "the woman and child holding the stick." without the woman and child, the stick would fall. thus, both the woman and child cause the stick to sustain its current state. recognizing these relations helps in comprehending human actions and their interactions with the surrounding environment. human-human causal relation. this denotes interactions between humans, such as "the woman holding the child." without the woman, the child would not be held. therefore, the woman causes the child to remain held. understanding these relationships is essential for interpreting social interactions and human behaviors in a scene. the causal graph depicted in figure 2 shows how entities in the scene are interconnected via causal relationships. understanding these causal relations facilitates more precise and significant interpretations of complex scenes. for example, in embodied artificial intelligence @xcite and autonomous driving systems @xcite , robots or vehicles should make decisions based on the causal relationships between entities within their environments. ## the cello dataset in this section, we elaborate on the dataset construction process based on the definition of causality as discussed in section 3. as shown in figure 3 , this process consists of three main steps: causal graph extraction, causal task selection, and causal question construction. ## the cello-cot strategy to enhance the capability of lvlms in accurately responding to the questions in cello, we propose cello-cot, a causally inspired chain-of-thought prompting strategy. it decomposes each causal question into multiple clear and manageable steps, enabling a sequentially structured analysis that supports effective problem-solving. given a causal question q with a corresponding image i, we provide lvlms with a series of instructions ℓ := (s 1 , . . . , s 4 ), including detailed descriptions of the four steps s 1 , . . . , s 4 depicted in figure 5 . this structured approach includes 1) extracting core entities from the question text; 2) identifying the causal graph structure represented in the image; 3) determining the type of causal task, and 4) compiling knowledge of causal inference relevant to the current task (e.g., the core concepts about "confounder" in figure 5 ). the model f lvlms : s @xmath0, . . . , r 4 corresponding to these steps. the final answer output will consider all these reasoning processes. compared to the standard strategy of directly posing questions to models, cello-cot imposes an inductive bias @xcite on lvlms, providing an effective solution to tackle causal reasoning problems. baselines. we evaluate ten leading lvlms in a zero-shot fashion, including four limited-access lvlms: claude-3-sonnet, claude-3-opus (anthropic, 2024), gemini-1.5-pro @xcite , and @xcite , and six open-source lvlms: blip-2 (6.7b) @xcite , llava-mistral (7b), bakllava (7b), llava-vicuna (13b) @xcite , minicpm-llama3-v-2.5 (8b) @xcite , and qwen-vl(7b) @xcite . details on these models are provided in appendix c. for consistent evaluation, we use standard accuracy metrics for all the models and tasks. performance is also benchmarked against a random baseline (i.e., 0.5 for binary and 0.25 for multiple-choice questions). ## related work causal evaluation on language models. several works have evaluated causality-related skills for nlp tasks. for example, @xcite investigate commonsense causality through "if-then" statements, while @xcite introduce reasoning tasks that consist of a series of steps towards a high-level goal. @xcite and @xcite focus on identifying cause-effect pairs to extract causal relations from document-level context. with the increasing focus on llms and causality, numerous studies have aimed to evaluate the causal reasoning abilities of large language models (llms) @xcite @xcite @xcite @xcite . unlike these studies, our research focuses on causal relations within the vision-language context. ## conclusion in this paper, we introduce a fine-grained and unified definition of causality involving humans and objects. building on the definition, we construct a novel dataset, cello, to assess the causal reasoning abilities of lvlms. to elicit causal reasoning in lvlms, we propose cello-cot, a causally inspired chain-of-thought prompting strategy, enabling lvlms to tackle challenging causal tasks in cello. extensive experimental results, as well as further quantitative and qualitative analyses on cello, provide insights for future work.