Pipeline for modeling causal beliefs from natural language (2023)

Abstract: We present a causal language analysis pipeline that leverages a Large Language Model to identify causal claims made in natural language documents, and aggregates claims across a corpus to produce a causal claim network. The pipeline then applies a clustering algorithm that groups causal claims based on their semantic topics. We demonstrate the pipeline by modeling causal belief systems surrounding the Covid-19 vaccine from tweets.

## introduction causal information facilitates learning @xcite @xcite , and is crucial to how humans use and represent language @xcite @xcite . causal relations are also ubiquitous in higher-level reasoning: they underlie our rich and flexible categories @xcite , shape our explanatory preferences @xcite , and structure our memories of events @xcite . beliefs about causal relations can also have pernicious outcomes. for example, beliefs that vaccines cause autism are central to antivaccination attitudes @xcite , and the belief that liberal politicians have causal influence over the outcome of climate science research motivates climate change denialism @xcite . because misinformation in online environments can spread rapidly to encourage these attitudes @xcite , new data science methods are necessary to combat these trends. however, data science algorithms generally struggle to advance a rigorous scientific understanding of psychological processes, as they provide correlational evidence that does not isolate cognitive mechanisms. methodologists should aim to develop natural language processing (nlp) algorithms that produce cognitively plausible data representations that researchers can utilize to guide explanatory understanding and motivate future interventions. because causal relations are the backbone of most higher-level reasoning processes in humans and are central to how we use language, developing systems that can isolate people's causal representations from language data is a natural place to start. however, nlp has historically struggled to identify instances of psychological causality (what a speaker thinks causes what) @xcite . this is because the variety of ways people communicate causality is immense @xcite @xcite , with most causal information latent in language inputs @xcite . previously, methods that relied on hand labeling causal constructions to relate linguistic features to components of causal relations were extremely brittle and struggled to generalize to outof-sample data @xcite . however, large language models may help overcome this shortcoming as these models utilize rich semantic representations of language and sub-word tokenization that can help them identify instances of causal language not expressed in training @xcite @xcite . in addition to simply identifying instances of causal language, data representations should account for the breadth of people's conceptual systems in which a causal claim is made. for example, causal beliefs are not held in isolation; instead, people have rich interlocking belief systems that span multiple topics that shape the integration of evidence @xcite @xcite . previ-ous methods for producing representations of people's belief systems rely on experiments and are slow to develop and may not generalize outside the lab @xcite . because it is important to understand the full context of people's belief systems to reliably predict how people will interpret evidence and make decisions, tools must be designed that can identify the vast web of beliefs that people use to interpret information in the wild. nlp tools can take advantage of the proliferation of social data online to build these representations @xcite . to this end, we introduce a pipeline based on the large language model, roberta-xl @xcite , that detects causal claims in text documents, and aggregates claims made across a corpus to produce a network of interlocking causal claims, or a causal claim network foot_0 . causal claim networks can be used to approximate the beliefs and causal relations composing people's conceptual understanding of the entities and events discussed in a corpus. to guide future research, we host a pipeline that produces interactive visualizations of people's causal belief networks. we demonstrate this software by building causal belief systems surrounding covid-19 vaccines from tweets. the pipeline for extracting causal claim networks follows three main steps (see figure 1 ). first, text documents are fed to a large language model, a roberta-xl transformer model @xcite , trained to extract causal claims made in a document (sentence to a paragraph in length). second, the entities that compose causal claims are clustered according to their embeddings, clusters proxy causal topics @xcite . third, claims made across the corpus are coreferenced and strung together to make a network of cause and effect claims to be displayed in an interactive visualization. we will now describe how a user could use our pipeline to build a causal claim network to visualize the causal claims made in a corpus of text documents. as shown in figure 2 , this follows two steps. first, a user uploads a .csv file containing the documents they wish to analyze. documents should be a sentence to a paragraph in length and can range from tweets, journal entries, or news headlines. next, the user selects which column in the dataframe contains the texts to be analyzed. a user can also specify if they want the pipeline to preprocess the documents and cluster the entities. it is worth noting that entity clustering works best when there are an abundance of causal claims about semantically distinct entities. if a user chooses to cluster claims and the pipeline does not produce an output, it does not mean that there are no causal claims present, but rather that there are no clear semantic clusters. in these cases, the users should deselected 'cluster entities' and rerun the pipeline. as seen in figure 2 , we analyze a data set of tweets about the covid-19 vaccine with the file name covid_tweets.csv, and the column containing the tweet texts is titled tweets. we provide access to this dataset on the tool interface, which can be downloaded to replicate this tutorial. once the document file is uploaded and the user presses submit, the job is queued and causal claims will be extracted. as shown in figure 3 , a job status window will be populated and the user will be updated on the degree of completion. as a rough reference, extracting causal claims from about 6000 tweets takes about a minute to complete once the job begins. once the job is completed, the screen will be populated with the causal claim network, like the one in figure 4 . there are a few things worth highlighting here. first, each edge represents a single extracted causal claim in the corpus, and nodes are colored by their causal cluster, or topic (see figure 5 ). clusters proxy topics in the data set and can be interpreted as central causal topics in the data set. in the next section, we describe how we calculate clusters. the causal claim network produced by the pipeline is interactive. a user can click on an edge to see the document and extracted causal claim that constitutes that edge (see figure 6 ). furthermore, as shown in figure 7 , a user can simplify the network by selecting to collapse the edges between the nodes. the edge thickness is proportional to the number of documents between those two clusters. to facilitate downstream analysis of causal claims (e.g., by analyzing sentiment or stance of causal claims), a user can download the edge list that produced the network as a .csv file. columns in this .csv file include: cause word span, cause cluster, effect word span, effect cluster, text, and figure 1 : high-level schematic describing how text documents become a causal claim network. raw text documents are first fed to a roberta-xl transformer fine-tuned on causal language, which follows a bio tagging scheme (see section 3.1) to return a list of tuples encoding the expressed cause and effect relationships. these tuples are co-referenced across the corpus to produce a causal claim network. figure 2 : uploading text data for causal claim analysis follows two steps. first select the .csv file you wish to analyze, then select which column in the dataframe contains the text documents to be analyzed. document id. the pipeline allows users to specify different parameters for the model (see figure 8 ). while the pipeline uncovers clusters automatically, users can specify the number of clusters to uncover in the corpus (this will equate to the number of nodes in the causal claim network). users can also specify the n_gram range to be used during preprocessing and can specify the number of top words used to describe each causal cluster/topic. ## what's happening under the hood in this section, we describe how the pipeline builds causal claim networks. this follows three steps: (1) causal claims are extracted using a roberta-xl transformer model that identifies which words belong to cause and effect events @xcite , (2) claims are clustered based on their semantic topics, and (3) a causal claim network is built by combining the claims stated in the corpus. ## related work although our approach is domain-general (documents do not need to belong to a single issue or topic for the pipeline to return a set of causal this system identifies claims and arguments made in the corpus, and sources additional wikidata information to put the claims in a richer content. mining causal claim networks requires isolating causal claims which oftentimes constitute arguments expressed in a text document: a reasoner makes a causal claim when explaining a mechanism @xcite or argument. claim detection is an active area of research @xcite , as is the detection of the components of arguments in text @xcite . because causal reasoning is central to the way people construct arguments @xcite , understanding how people posit causal claims can shed light on the types of arguments people will endorse related to that issue. most claim detection algorithms work on the level of single documents, but approaches such as those of @xcite propose corpus-wide claim detection. our pipeline utilizes a mixing of the two: claims are detected within a document and then aggregated across the documents in the corpus to provide a corpus-level representation. ## conclusions and future work interactive data visualization is an effective way for people to make sense of complex data @xcite and can be an effective tool to guide scientific thinking @xcite . our pipeline is designed to help researchers explore the causal claims expressed in a corpus through interactive exploration. there are therefore many applications of this tool to the study of human reasoning and belief change, and future work will test the efficacy of these use cases. for example, researchers in cognitive science have worked on developing methods to measure people's rich conceptual systems about vaccines @xcite . these methods often require the development of surveys that measure people's attitudes toward a variety of related issues. causal claim networks can give researchers a starting place to know what measures they should include in these surveys. in future work, we will work on expanding the visualization tool to include features that allow for richer forms of interaction. for example, by allowing users to build subnetworks based on another data attribute (e.g., stance of the document, expressed sentiment), to allow for comparisons across networks. related to this, future work will also develop quantitative measures of divergence across networks.