D ouble L ingo: Causal Estimation with Large Language Models (2024)

Abstract: Estimating causal effects from non-randomized data requires assumptions about the underlying data-generating process. To achieve unbiased estimates of the causal effect of a treatment on an outcome, we typically adjust for any confounding variables that influence both treatment and outcome. When such confounders include text data, existing causal inference methods struggle due to the high dimensionality of the text. The simple statistical models which have sufficient convergence criteria for causal estimation are not well-equipped to handle noisy unstructured text, but flexible large language models that excel at predictive tasks with text data do not meet the statistical assumptions necessary for causal estimation. Our method enables theoretically consistent estimation of causal effects using LLM-based nuisance models by incorporating them within the framework of Double Machine Learning. On the best available dataset for evaluating such methods, we obtain a 10.4% reduction in the relative absolute error for the estimated causal effect over existing methods.

## introduction a common goal of scientific research is the analysis of causal relationships @xcite @xcite . consider the following motivating example, where a pharmaceutical company wants to estimate the causal effect of the prescription of antibiotics (treatment) on the patient's disease progression (outcome). the causal effect is defined as the expected change in disease progression across two counterfactual worlds which only differ in whether the patient is given antibiotics @xcite . when randomization is impossible or unethical, we estimate causal effects from observational data using assumptions about the underlying data distribution. confounders -variables affecting both the treatment and outcome -introduce potential bias that must be addressed. when data is low-dimensional, confounding can be controlled for using various methods from the literature @xcite . however, several challenges arise in the case of high-dimensional confounders. suppose the pharmaceutical company has free-text clinical notes that may include information about patients' histories, diagnoses, or relationships with their doctors @xcite . if these variables appear nowhere else in the patients' records, then account for potential confounding should use text-based causal methods @xcite . since text is high-dimensional, it requires sophisticated modeling that captures semantic meaning. existing models often utilize overly simplified representations of the text @xcite , such as a bag-of-words (bow) representation. while such representations combined with simple estimation models allow for consistent foot_0 estimation, they may fail to capture the true complexity of the text's underlying relationships. the use of large language models (llms) in causal estimation has only recently been studied @xcite , and many researchers suggest the need for more sophisticated natural language processing (nlp) techniques @xcite @xcite . however, while llms excel at predictive tasks, they do not meet the necessary statistical assumptions for a consistent causal estimation. we present doublelingo, combining double machine learning with llm-based nuisance models to enable a theoretically consistent estimation of causal effects with text-based confounding. we test our model on a novel dataset @xcite , obtaining the best causal effect estimates reported thus far. in particular, our relative absolute error is over 10% lower than the best current models. ## causal inference background while causal inference is a broad and diverse field @xcite , we provide a brief introduction here. for recent surveys of causal inference and natural language processing, see @xcite or @xcite . ## model we now introduce notation to formalize our proposed method to use llms to replace a simplistic bow text representation. consider this partially linear model corresponding to figure 1 : here θ 0 is the true at e we hope to estimate, η 0 = (m 0 , g 0 ) are nuisance parameters, and u, v are our error terms. following keith et al. ( 2023 ), 2 we similarly assume the causal effect @xmath0. any estimator θ 0 of θ 0 must be both unbiased and consistent such that: 3 while llms have drastically changed the field of nlp @xcite , they are not consistent estimators of causal parameters due to both explicit and implicit regularization @xcite . thus, a naive approach of using an llm such as bert @xcite to learn the propensity p(a | t ) in equation ( 7 ) would be biased. ## causal dataset & experiment unlike supervised learning models, which can be evaluated on held-out test sets with ground-truth labels, causal estimation methods require evaluations with counterfactual ground-truth, which is impossible to measure from observed data @xcite . researchers often turn to (semi-)synthetic data, for which there is a tension between generating realistic text and maintaining full knowledge of the underlying data-generating process (dgp) @xcite . most current datasets fail to accomplish both, either fully specifying the dgp but with unrealistic text @xcite , or using real-world text inside a semi-synthetic dgp (veitch et al., 2020).