Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need (2023)

Abstract: A scientific claim typically begins with the formulation of a research question or hypothesis, which is a tentative statement or proposition about a phenomenon or relationship between variables. Within the realm of scientific claim verification, considerable research efforts have been dedicated to attention architectures and leveraging the text comprehension capabilities of Pre-trained Language Models (PLMs), yielding promising performances. However, these models overlook the causal structure information inherent in scientific claims, thereby failing to establish a comprehensive chain of causal inference. This paper delves into the exploration to highlight the crucial role of qualitative causal structure in characterizing and verifying scientific claims based on evidence. We organize the qualitative causal structure into a heterogeneous graph and propose a novel attention-based graph neural network model to facilitate causal reasoning across relevant causally-potent factors. Our experiments demonstrate that by solely utilizing the qualitative causal structure, the proposed model achieves comparable performance to PLM-based models. Furthermore, by incorporating semantic features, our model outperforms state-of-the-art approaches comprehensively.

## introduction a scientific claim refers to a statement or proposition that puts forward a particular hypothesis, theory, or discovery within the realm of science. the process of verifying a scientific claim typically entails thorough examination and assessment to ascertain its validity, relying on evidences that can support, refute, or remain neutral regarding the claim. in order to accomplish this, researchers must diligently scrutinize and characterize the specific features, qualities, or attributes of the claim. they are tasked with identifying the essential elements, variables, or factors involved and furnishing a comprehensive comprehension of the claim's essence and extent. in the field of scientific claim verification, the majority of research efforts have been devoted to dedicating attention architectures @xcite and leveraging the text comprehension capabilities of pre-trained language models (plms) @xcite to improve performance. these studies follow the end-to-end paradigm and employ transformers as the foundational framework, enabling the model to focus on relevant parts of the claim and capture the essential elements and relationships within the text. however, despite their proficiency to capture key elements and their interactions, these models neglected the causal structure information inherent in the claims that should be characterized during the process of scientific claim verification, let alone establish a comprehensive chain of causal inference between the claim and the evidence. following the schema proposed in sciclaim @xcite ), we provide an example of qualitative causal structure in a scientific claim in figure 1 . the claim describes the relationship between 'temperature' and 'covid-19 transmission', which we identify as the key factors (highlighted in green). the core meaning conveyed in the claim is the causal connection between these factors, expressed by the term 'change' indicating an association (highlighted in red in figure 1 ). the arrows from association to factors indicate that 'temperature' serves as the cause, while 'covid-19 transmission' is effected by this cause. the yellow spans represent modifiers of the association, denoting its likelihood, extent, or direction, constraining the applicability or scope of the claim. as shown in figure 1 , there could also exist a qualitative causal structure in the evidence. unlike factoid claim verification, a cognitive sys-temperatures significantly changes covid-19 transmission in (sub)tropical cities of brazil. tem developed for verifying scientific claims is anticipated to make sense of specialized descriptive language and employ causal reasoning concerning interactions between quantities or entities @xcite . of utmost importance is the understanding of the qualitative causal structure, as we hypothesize that it can characterize a scientific claim and facilitate the traversal of causal reasoning among relevant causally-potent factors for verification. for instance, the evidence-based verification of the claim in figure 1 can be achieved by exploring the relevant key factors and the associations that aggregate features from factors. moreover, the modifiers play a crucial role in determining the direction of traversal, e.g., replacing 'significantly' with 'hardly' or 'never' could completely change the meaning of the claim. with the hypothesis, the objective of this paper is to examine the role of qualitative causal structure to characterizing and verifying scientific claims. to achieve this goal, we employ the schema introduced in sciclaim @xcite to construct simplified qualitative causal structures for claim-evidence pairs in the datasets healthver @xcite and scinli @xcite . these qualitative causal structures are then organized as heterogeneous graphs to enable reasoning. further, we propose a novel attention-based graph neural network model that performs causal reasoning between claim and evidences. considering that compound claims need to be divided into several atomic subclaims, we introduce a graph decomposition algorithm and compute representations for sub-graphs, which are then aggregated using a weighted-sum operation to obtain final representations for the entire graph. the weight distribution of sub-graphs also provides insights into which subclaims are supported by evidence and which are not. through extensive experiments, we demonstrate that our proposed model, utilizing only the qualitative causal structure information, achieves comparable performance to pre-trained language model (plm)-based models. additionally, by incorporating the semantic features of the original sentences, our model achieves state-of-the-art performance. ## qualitative causal schema sciclaim @xcite proposes a fine-grained graph schema for scientific claims. we simplified the schema by removing certain entity and relation types that are not essential for the task as follows. entities are labeled text spans. we aggregate six entity types into three: factor, association, and modifier, furthermore, to conduct aggregation, we design a novel node type root. factor remains unchanged as variables tested or asserted within a claim, typically nouns or noun-phrases, such as 'cats' and 'covid-19' in figure 2 . association are the same in the original schema, which is explicit phrases associating one or more factors that describe the structural effect of the entity, such as 'pass' in figure 2 . modifier is a new entity type, which aggregates the original entity types magnitude, evidence, epistemics, and qualifiers since they perform similarly in scientific claim verification as modifiers of the association. root is designed to aggregate tree representations into a single vector representing the whole scientific sentence. relations are directed edges between labeled entities, and we preserve the original arg0 and arg1 relation types, which relate associations to their cause or effect, respectively. we also retain the modifier edge with a new name modifying to distinguish the entity modifier. modifying relates associations to their modifying components (modifiers figure 3 : the entities in red, yellow and green are associations, modifiers and factors repectively. we illustrate a typical entity-overlapped annotation modification with the text: patients with greater baseline lap volume are more likely to benefit from statin therapy. in our schema), and comp_to which represents comparison in scientific claim verification. however, we remove other high-order edges such as q+ and q-, since these relationships are contained in the semantics of associations. in addition to modifying the schema, we propose the introduction of a nested tree structure to replace the annotation setting that allows for entity overlapping. the original schema permits the overlapping of two entity spans, posing a challenge for models to accurately identify and differentiate between distinct entities. we present a typical example of entity overlapping in figure 3a , where the entire span 'patients with greater baseline lap volume' is annotated as a factor, despite the overlapping association 'greater'. furthermore, the schema does not enable the subsequent model to concentrate on specific scientific entities within the sentence, such as 'baseline lap volume' and 'statin therapy', as these concepts are not extracted as entities. therefore, we propose the nested design to decompose the overlapped entity into smaller, non-overlapping sub-trees as we present in figure 3b , the span 'patients with greater baseline lap volume' in figure 3a is decomposed into a sub-tree rooted by 'greater' in figure 3b . the tree structure enhances the model's comprehension of hierarchical relationships between entities and their modifiers, thereby facilitating more precise identification of relevant entities and relations for claim verification. following the schema, we annotate the structure of each sentence in healthver dataset @xcite , which is a complex scientific claim verification dataset. we proceed to train a joint entity and relation extraction model for extracting the sentence structures. additionally, we utilize the extraction model to conduct experiments on another dataset scinli @xcite . ## proposed approach in this section, we introduce the proposed model for scientific claim verification by utilizing the qualitative causal structure, also referred to as verqcs. we explain the big picture of verqcs here. verqcs utilizes causal graphs within claims and evidence. the graph structures are extracted by our dedicated extraction model. with both the graphs and the textual content, verqcs performs reasoning and ultimately predicts the relationship. modules like factor cross-attention, causal graph decomposition, and causal graph reasoning are introduced to capture the structural characteristics of claims and evidence, which are subsequently employed to predict the relationships. prior to introducing the model, two properties of the graph structure are introduced: first, the associations always serve as the root nodes of the trees. in figure 2 , both sub-claims have 'pass' as their root, which is an association; second, if we designate the tree directly connected to the root as the root tree, we observe that root tree corresponds to the semantic span of sub-claims or sub-evidence. in figure 3b , the subtree rooted by 'greater' corresponds to the nounproperty phrase 'patients with greater baseline lap volume' which does not provide any assertive information that can be argued or serve as evidence. we have observed that only the root tree corresponds to the semantic span, which is the tree rooted by 'benefit from' in the figure. the properties we observe reveal the intrinsic connections between structure representations and semantic meaning. sub-claim or sub-evidence can be extracted via identifying the root tree. consequently, we employ structural reasoning on their respective root trees to facilitate reasoning between sub-claims and sub-evidence. the tree representation learning module, which we further detail in section 4.5, encodes a tree structure using an embedding vector and employs a re- gnn graph aggregation graph homogenization modifying graph gnn factor graph factor cross-attention causal graph decomposition 𝐸𝑚𝑏 𝑇 𝐶𝚓 𝐸𝑚𝑏 𝑇 𝐶𝚔 𝐸𝑚𝑏 𝑇 𝐸𝚓 𝐸𝑚𝑏 𝑇 𝐸𝚔 𝐸𝑚𝑏 𝐺 𝐶 𝐸𝑚𝑏 𝐺 𝐸 g e 𝑻 decomposition algorithm extraction model entity encoder entity encoder label prediction causal graph reasoning cross-attention aggregation extraction model semantic representation semantic representation input claim: the authors found the close association between frailty and diabetic…… input evidence: in middle east, the recovery rate and mortality rate showed a correlation …… figure 4 : the framework of verqcs with an input example of a claim-evidence pair. the nodes in red, yellow and green represent associations, modifiers and factors respectively. ## experiment in this study, we perform two distinct experiments: structure extraction and scientific claim verification, respectively. we present the experiment of scientific claim verification in the section, more details about structure extraction are presented in appendix a.1. ## conclusions this paper introduces the utilization of qualitative causal structure in scientific claim verification. to this end, we propose the novel model verqcs, which performs structural reasoning on the causal graph. the framework has been demonstrated to be effective even without semantic features enhancing. additionally, verqcs demonstrates its capability of selecting sub-claims and sub-evidence by assigning weights to the sub-graphs, which provides insights of how verqcs conducts reasoning. in conclusion, we have substantiated the efficacy of qualitative causal structures in claim verification. moreover, we hypothesize that the factor entities within causal graphs can be seamlessly linked to external knowledge graphs, thus offering opportunities for further exploration.