Learning Faithful Representations of Causal Graphs (2021)

Abstract: Learning contextual text embeddings that represent causal graphs has been useful in improving the performance of downstream tasks like causal treatment effect estimation. However, existing causal embeddings which are trained to predict direct causal links, fail to capture other indirect causal links of the graph, thus leading to spurious correlations in downstream tasks. In this paper, we define the faithfulness property of contextual embeddings to capture geometric distance-based properties of directed acyclic causal graphs. By incorporating these faithfulness properties, we learn text embeddings that are 31.3% more faithful to human validated causal graphs with about 800K and 200K causal links and achieve 21.1% better Precision-Recall AUC in a link prediction fine-tuning task. Further, in a crowdsourced causal question-answering task on Yahoo! Answers with questions of the form “What causes X?”, our faithful embeddings achieved a precision of the first ranked answer (P@1) of 41.07%, outperforming the existing baseline by 10.2%.

## introduction learning distributed word representations that capture causal relationships are useful for real-world natural language processing tasks @xcite @xcite . approximating the notion of causality with a similarity-based distance metric using separate vector representations for cause and effect tokens has led to significant improvement in the performance of downstream tasks like question answering, but can be too restrictive to generalize over unobserved edges in larger causal graphs @xcite . in downstream causal reasoning based tasks like dialog systems @xcite , explanation generation @xcite , question answering @xcite , it is important to align the models with the corresponding causal graph. however, words that have low cosine similarity capture various semantic similarities, like relatedness, synonyms, replaceability, or complementarity, but not directionality @xcite . hence, any symmetric distance in an embedding space cannot convey the directed causal semantics for a downstream task @xcite . in this paper, we overcome these two shortcomings and propose to optimize for directed faithfulness @xcite that word embeddings have to satisfy towards a causal graph. prior work on capturing sufficient information for causal inference tasks from embeddings aims to directly use them for average treatment effect estimation @xcite . we are, however, interested in a complementary question: "can we learn word embeddings based on a distance measure that maps the directed distance between nodes in a causal graph to that in the embedding space?". unlike prior work, which aims to learn a causal aware embedding restricted to direct link prediction @xcite , we propose faithfulness constraints so that causal word embeddings aims to preserve the partial ordering over pairwise distances in the directed causal graph. in this paper, to achieve the goal of learning faithful word embeddings with a vocabulary of more than 100k tokens, we minimize faithfulness violations over pairwise samples of nodes in the causal graph. through this constrained optimization, we learn an embedding that can be applied directly for causal inference tasks but also generalizes to emergent causal links. it has been shown that nlp models need to understand such causal links that persist in the real world for safe deployment @xcite . embeddings that violate the faithfulness property, can lead to spurious correlations based on co-location in the embedding space. for example, in a yahoo! causal question-answering task's example: "what causes nosebleed?": the answers were "dry air", "heavy dust", "damaged nasal cells" and "liver problems". if we were to only rely on an undirected association based embeddings, the causes "dry air" and "liver problems" might be nearby (with distance of 2), but would be appropriately placed far in a directed causality based embedding space. to capture such asymmetric properties, we aim to preserve alignment with the causal graph by mapping causal links to an asymmetric quasi-pseudo distance measure during training to capture directionality of the causal graph as per figure 1 . since human validated causal graphs can be used directly to answer questions of the type "what causes x?", we demonstrate the utility of learning faithful representations by using our distance-based features to solve the yahoo! causal question-answering (qa) task. a causal qa task, unlike a standard qa task, can directly benefit from incorporating a causal graph into word embeddings to answer anti-causal queries. our key contributions are: ## conclusion we show that the faithfulness of text embeddings to a causal graph is important for causal inferencealigned downstream tasks. by incorporating the three faithfulness properties of neighborhood, uniformity, and distance correlation through regularization constraints while learning embeddings, we improve the precision of the first ranked answer in the causal qa task by 10.2%. we show that this is due to causal re-alignment of embeddings as per an asymmetric pseudo-distance metric.