CR e TIHC : Designing Causal Reasoning Tasks about Temporal Interventions and Hallucinated Confoundings (2023)

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their ability to establish causal relationships, particularly in the context of temporal interventions and language hallucinations, remains challenging. This paper presents CReTIHC , a novel dataset designed to test and enhance the causal reasoning abilities of LLMs. The dataset is constructed using a unique approach that incorporates elements of verbal hallucinations and temporal interventions through the reengineering of existing causal inference datasets. This transformation creates complex scenarios that push LLMs to critically evaluate the information presented and identify cause-and-effect relationships. The CReTIHC dataset serves as a pioneering tool for improving LLM’s causal inference capabilities, paving the way for a more nuanced understanding of causal relationships in natural language processing (NLP) tasks. The whole dataset is publicly accessible at: (https://github.com/ChangwooChun/CReTIHC)

## introduction llms have emerged as a powerful tool in natural language processing, with the ability to generate human-like text by exploiting extensive prior knowledge. despite these advancements, llms are based on the transformer structure @xcite and have exhibited shortcomings in establishing causal relationships in their outputs. this stems from their reliance on empirical reasoning and statistical patterns from training data @xcite . specifically, llms face challenges in disentangling true causal relationships from confounding biases introduced by language hallucinations and temporal interventions @xcite . ## related works causal reasoning, the process of identifying and understanding the cause of certain outcomes and the influence of changes in these causes, has been a long-standing focus in various fields @xcite @xcite . in the domain of deep learning-based natural language processing, efforts have been made to incorporate and interpret causal reasoning to discern how causal factors contribute to the outcomes generated by language models @xcite @xcite @xcite . in the era of deep learning, causal reasoning has emerged as a crucial capability to ensure that models generalize well and produce accurate outcomes when faced with new data that may not conform to the training data distribution @xcite . although the scaling laws have facilitated the emergence of models with human-like generalization capabilities @xcite @xcite , such as gpt-3, gpt-4 foot_0 , and llama @xcite , by leveraging extensive data and large architectures, these models often struggle to establish causality. pre-training on extensive data does not suffice in ensuring causal inference as causal relationships are often obscured by interventions, confounding biases, and language hallucinations @xcite . recently, to enhance causal reasoning in models, methods such as adversarial training and data augmentation techniques rooted in linguistic theories of causal connectives have been introduced @xcite . notably, these methods have achieved statistically significant improvements on the copa dataset, including its balanced version designed to avoid surface-level cues. recently, there have been efforts in creating tasks, frameworks, and datasets focusing on commonsense causal reasoning, such as copa @xcite , αnli @xcite , rock @xcite , and e-care @xcite . these aim at assessing causal inference abilities independently of the sheer volume of knowledge. however, existing datasets and tasks have limitations in measuring and evaluating the dynamic nature of causal relationships, especially in scenarios involving sequential events leading to a particular outcome. furthermore, these datasets don't address confounding biases introduced through temporal signals or language hallucinations stemming from word co-occurrences to address these limitations, our study restructures the e-care foot_1 dataset to present tasks centered around temporal interventions and hallucinated confoundings. these tasks are designed to be more challenging and representative of real-world scenarios. this approach aims to provide a foundation for more robust and fine-grained analysis of causality in natural language processing, particularly in the era following the advent of llms. ## proposed method our key idea is to investigate changes in the causal relationship between two events when a new event appears. we designed the cretihc dataset by adding temporal interventions and hallucinated confoundings to the e-care, an existing natural language commonsense causal reasoning dataset. grounding prior studies regarding causality @xcite @xcite , we have noticed that unseen, unobserved interventions can heavily impact the causal relationship between events. also, previous causality research shows that causal relationships guarantee antecedent relationships, but antecedent relationships do not guarantee causal relationships @xcite @xcite . to examine whether llms can identify temporal order and actual causal relationships, both temporal order relationships confused with actual causal relationships are needed. however, the original e-care dataset, which is not arranged chronologically, consists of premises, hypotheses, and explanations that make the hypotheses plausible. we restructured this dataset to create a chronologically ordered sequence of events. specifically, when the premise and hypothesis are in an effect relationship, we reordered them so that the premise is the event that occurred first and the hypothesis is the event that occurred second. conversely, when the premise and hypothesis are in a cause relationship, we reordered them so that the hypothesis is the event that occurred first and the premise is the event that occurred second. the preceding event is denoted as s 1 , and the succeeding event is denoted as s 2 . in the next phase, we employed llms to manipulate the causal relationships in the restructured dataset, presented as a series of consecutive events. we used temporal interventions (task 1) and introduced confounders through linguistic manipulation (task 2), with the llm generating sentences that vary the causal relationship between s 1 and s 2 . hallucinated confounders, adding new elements to make spurious causal relationships, were utilized to create confusion between word co-occurrence and actual causality. a event that changes the causal relationship between s 1 and s 2 are denoted by z. task 1: temporal interventions task 1 uses a temporal intervention to manipulate the causal relationship between event s 1 and outcome s 2 based on the timing of the causal event s 1 . if s 1 and s 2 have a plausible causal relationship, we intervene with an event that could occur between s 1 and s 2 to weaken the causal relationship. if s 1 and s 2 do not have a causal relationship with each other, an intervention can strengthen the causal relationship between the two events. the event z is necessarily positioned between the two events, forming a chain that alters the relationship between the causally related events. depending on its temporal placement, an intervention can relatively transform the strength of causality between the two events s 1 and s 2 . an example of a prompt used for task 1 is in appendix b. task 2: hallucinated confoundings task 2 examines cases where no causal relationship exists between event s 1 and s 2 , but a confounding event z obscures this fact. the task involves two structures: a collider and fork. in the collider structure, the confusing event z is causally related to both s 2 and s 1 . here, no direct causal relationship exists between s 2 and s 1 , but event s 1 is changed to imply a causal relationship. in the fork structure, both s 2 and s 1 are the result of the confusion event z, and event s 1 is changed to imply a causal relationship that does not exist. this creates confusion that there is a causal relationship between s 1 and s 2 , even though there is no direct causal relationship between them. an example of a prompt used for task 2 is in appendix c. after rearranging the example from the e-care dataset into a sequence of chronological order and applying the suggested operations, the sentence can be changed to the following. an example from e-care dataset: ## experiments our proposed cretihc dataset introduces the task of identifying all three factors(enhance, weaken, irrelevance) involved in a causal relationship. when appearing an event between two chronologically ordered events, s1 and s2, it is a matter of determining how that event affects the existing causal relationship between s1 and s2. a total of 5 line of sentences are given in one sample, the s1 and s2 events are fixed, and the task is to determine how each of the 3 sentences, a1 to a3, affects the causal relationship between the previous two events. to exclude the influence of sentence order of prompts, the positions of enhance, weaken, and irrelevance were randomly mixed. ## conclusions the significance of causal reasoning in language understanding cannot be overstated. our research has sought to reflect the complexity of real-world causal reasoning by enhancing an existing commonsense causal reasoning dataset with temporal interventions and hallucinated confoundings. the cretihc dataset, rich with continuous event sequences and confounding variables, can offers a more rigorous and holistic exploration of causal relationships in language. our approach, which mirrors real-world causal reasoning scenarios, has leveraged large language models (llms) to generate comprehensive datasets. this not only underscores the potential of llms in enhancing various tasks in natural language processing (nlp) but also highlights the need for models that can efficiently learn from small amounts of data and understand causal relationships as adeptly as humans do. while we acknowledge that a language model trained on our dataset may not fully resolve causal inferences, we believe that our work with the cretihc benchmark performance measurements contributes to the ongoing journey toward full causal understanding by machines. in conclusion, our research underscores the importance of causal reasoning in language understanding and paves the way for future research in this critical area.