ECH o: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning (2023)

Abstract: We introduce ECHo (Event Causality Inference via Human-Centric Reasoning), a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios. ECHo employs real-world human-centric deductive information building on a television crime drama. ECHo requires the Theory-of-Mind (ToM) ability to understand and reason about social interactions based on multimodal information. Using ECHo, we propose a unified Chain-of-Thought (CoT) framework to assess the reasoning capability of current AI systems. Our ToM-enhanced CoT pipeline accommodates various large foundation models in both zero-shot and few-shot visio-linguistic reasoning. We use this framework to scrutinize recent large foundation models such as InstructGPT and MiniGPT-4 on three diagnostic human-centric tasks. Further analysis demonstrates ECHo as a challenging dataset to expose imperfections and inconsistencies in reasoning. Our data and code are publicly available at [https://github.com/YuxiXie/ECHo](https://github.com/YuxiXie/ECHo).

## introduction social intelligence refers to the ability to understand, navigate, and respond effectively in social scenarios. as a widely investigated concept in psychology and social sciences, it has become a prominent facet of artificial intelligence @xcite @xcite . in the development of social intelligence, humans gain the crucial cognitive capacity of understanding and reasoning about the mental states (i.e., beliefs, desires, intentions, emotions, and thoughts) of individuals. this pivotal form of competence is commonly denoted as theory-of-mind (tom) @xcite @xcite . as a fundamental ability of social commonsense reasoning @xcite , tom is important for machine reasoning to achieve artificial general intelli- ## related work echo takes a further step towards social intelligence on human-centric inference in visiolinguistic scenarios, probing the tom capacity of large foundation models via cot reasoning. visio-linguistic reasoning. datasets and tasks in visio-linguistic reasoning span widely from descriptive information extraction @xcite @xcite , physical relation inference @xcite , to complex and deep reasoning on the event and human factors @xcite @xcite . echo follows this trend to enhance the reasoning depth towards human-specific facets. unlike recent works @xcite @xcite of human-centric reasoning, echo is integrated with rigorous tom annotations, supporting the final inferences via cot reasoning for better consistency. specifically, there are long-standing arguments on the development and assessment of tom in both human psychology and machine intelligence @xcite @xcite . measurement of tom is usually based on false belief tasks @xcite , assessing the ability to distinguish one's own (true) belief and others' (false) belief, given the information and experience asymmetry among different individuals. constructed on crime drama, echo contains an abundance of such cases of false belief to probe tom ability. ## the echo corpus echo contains 2k inference instances collected via our tom-enhanced cot scheme. to facilitate the approximation of authentic social interactions, we ground echo in csi: crime scene investigation, an american procedural forensics crime series in english (wikipedia contributors, 2023). with visual evidence and scenes in frames and utterances and narrations in screenplays, csi represents a rich multimodal source, spanning widely factual, relational, and inferential data. as shown in ## diagnose human-centric reasoning in visio-linguistic inference using echo echo centers on rigorous human-centric information that supports tom-enhanced cot reasoning in visio-linguistic scenarios. we detail the formulation of our three sequential tasks to diagnose the human-centric reasoning ability next. notation. each echo instance consists of a sequence of visual frames @xmath0, f 2 , • • • , f n ] = f 1:n , a textual screenplay t . the frames are manually selected following the role and emotion identification in annotation. we gather frames featuring different characters together to represent the visual content of each clip. we designate each utterance or narration in the text to be an event e i for the causality inference, as there is a key character c to focus on in each instance. task one: role identification (cf. annotation round two, stage 1). the psychoanalysis of a person's role in social interactions indicates their identity, helping to infer their intentions, actions, and relations with others @xcite . therefore, we test the ability of role identification to probe the fundamental humancentric understanding in tom reasoning. given frame(s) f i of the key character c and the corresponding screenplay t , we prompt the model to generate the role r of c. the role can be defined by age, occupation, or relations with others, as these attributes can exhibit a strong correlation with the human mental states for tom reasoning. task two: emotion interpretation (cf. annotation round two, stage 2). emotions convey clues of mental states beyond verbal messages @xcite . they bridge fundamental understanding (e.g., role identification) and further inference (e.g., intent prediction) in human-centric reasoning. we thus propose emotion interpretation as our second diagnostic task. we formulate this task as multi-choice question answering and test the alignment of model and human predictions on 13 candidate emotions, adapted from the taxonomy of @xcite to the crime data. task three: event causality inference (cf. annotation round two, stage 3). despite the practical success of large foundation models on a wide range of reasoning tasks, there is a debate as to whether they genuinely execute causal reasoning or just reproduce memorized patterns @xcite . furthermore, these models still produce imperfections such as erroneous logic and human-factor understanding @xcite @xcite . hence, we formulate a subsequent task as event causality inference to assess the causality reasoning capacity among sociallygrounded events. we also utilize the tom infer- holly gribbs: pretty good. gil grissom: good. okay, look, this was a routine robbery. dust for prints, check the videotape take lots of fun photos. i'll be back in about an hour to pick you up. holly gribbs: okay. (holly gets out of the car.) gil grissom: if you get done early use channel seven on your walkie. (grissom pulls out of the parking lot.) (holly heads inside the liquor store.) [causality question] what has caused/enabled the event that gil grissom says: if you get done early use channel seven on your walkie? 🤖 .m. a man and woman sitting in a car at night. a man in a black shirt and a woman in a white shirt. the man's eyes are wide and he is staring off into space. ## tom-enhanced cot reasoning given the inputs of visual frames v and textual screenplay t , our objective is to make humancentric inference i. we follow @xcite to break down the process into intermediate steps r and thus accommodate the three tasks in a unified framework to assess large foundation models. as illustrated in figure 4 , our framework follows the vision + llm paradigm @xcite @xcite to facilitate multimodal understanding using the reasoning ability inherently grown in language models. ## experiments we assess the social intelligence of existing large foundation models using the unified framework in section 5 on our diagnostic tasks in section 4. we ablate components in our framework on different backboned models and evaluate their effect on both ## conclusion we introduce a visio-linguistic dataset echo to probe human-centric social intelligence. with our tom-enhanced cot framework, we diagnose the reasoning ability of large foundation models. experiment results and further analysis demonstrate the deficiency of current ai systems and potential bias in echo for efficient, correct, and consistent reasoning. we foresee follow-up work on both model and data facets to develop faithful reasoning across a broader range of social scenarios.