CLEAR : Can Language Models Really Understand Causal Graphs? (2024)

Abstract: Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models’ understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models’ behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings. Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains.

## introduction causal reasoning is fundamental to how humans understand the world and solve challenges @xcite . the ability to reason causally allows us to explain phenomenon and predict the future @xcite @xcite . there are various causal models used to investigate and represent causation, including mathematical equations, logical statements, and causal graphs @xcite . among them, causal graph gains widespread adoption due to its intuitive and concise representation of complex causal relationships @xcite . ## the clear benchmark to explore the question: can language models really understand causal graphs? we propose clear, the first benchmark dedicated to causal graph understanding. we ensure dataset diversity by accounting for various factors: the size, type, and density of causal graphs, as well as the richness of tasks and question types. ## conclusion this paper provides a comprehensive and in-depth exploration on the question: can language models really understand causal graphs? we define a practical framework for accessing a model's understanding. we introduce clear, a novel benchmark designed to evaluate a model's understanding of causal graphs, filling a significant gap in existing research. we validate our framework through extensive experiments and conclude five insightful findings.