Conceptualizing Treatment Leakage in Text-based Causal Inference (2022)

Abstract: Causal inference methods that control for text-based confounders are becoming increasingly important in the social sciences and other disciplines where text is readily available. However, these methods rely on a critical assumption that there is no treatment leakage: that is, the text only contains information about the confounder and no information about treatment assignment. When this assumption does not hold, methods that control for text to adjust for confounders face the problem of post-treatment (collider) bias. However, the assumption that there is no treatment leakage may be unrealistic in real-world situations involving text, as human language is rich and flexible. Language appearing in a public policy document or health records may refer to the future and the past simultaneously, and thereby reveal information about the treatment assignment. In this article, we define the treatment-leakage problem, and discuss the identification as well as the estimation challenges it raises. Second, we delineate the conditions under which leakage can be addressed by removing the treatment-related signal from the text in a pre-processing step we define as text distillation. Lastly, using simulation, we show how treatment leakage introduces a bias in estimates of the average treatment effect (ATE) and how text distillation can mitigate this bias.

## introduction in observational settings, scholars need to collect information about potential confounders in order to estimate the causal effect (τ ) of a treatment on an outcome @xcite ). if we observed the set of confounders directly, we could condition on those quantities to recover unbiased causal effects. yet, because some confounders u are difficult to measure directly, scholars are turning to alternative data sources, such as medical records, policy documents, or social media posts, to indirectly measure (proxy) confounders @xcite . recent methodological frameworks supply ways of integrating high-dimensional text data into causal estimation @xcite @xcite . however, prior literature has primarily assumed that documents only contain information about the confounder, but not about the treatmentsomething we term the no-treatment-leakage assumption. here, "contain information" means that the text is caused by the treatment (or the confounder) directly or indirectly. when treatment leakage occurs after treatment assignment, its bias is equivalent to a post-treatment bias @xcite . treatment leakage leads to an identification challenge. the challenge is that w is both necessary for adjusting (as it is a proxy) yet it is also a posttreatment variable. without treatment leakage, w would not be a post-treatment variable, as it does not harbour information about the treatment assignment. but because of leakage, scholars would have to accept bias arising from either adjusting on a post-treatment variable (arising from the part of w influenced by the treatment) or bias arising from not adjusting for unobserved confounding. although several methodological studies develop and adapt causal-inference methods for text data @xcite , almost no studies examine the biasing influence of treatment leakage and how to counter this bias. our work investigates the treatment-leakage challenge. it shows that if w is the only available text representing u and there exists a distillation method, f , that has the ability to transform (e.g. partition) w into its post-treatment w t and proxy textual-components w u , then adjusting on w u is the best one can do in identifying τ . as w u is not post-treatment, we can adjust for it to reduce the bias when estimating τ . these f functions can represent a human annotator, identifying and removing parts of text (e.g., words, sentences) that belong to w t and curating w u ; or, under additional assumptions, f can be based on supervised or unsupervised machine learning models that transform the text or its representation @xcite . in this paper, we define key assumptions and demonstrate the mechanics of text distillation in a simulated experiment. using a language model, we generate synthetic documents w so that they contain information about the treatment assignment, t , and the unobserved confounding, u , imprinted paragraph by paragraph. because we control which paragraph is affected by t (injecting post-treatment bias) or by u (infusing knowledge about the confounder), we have an oracle distillation function, f , that mimics human coding. this oracle method perfectly distills w , and supplies w u . then, when using w u in our causal model, we reduce bias of τ markedly. although our oracle is idealized, it deepens intuition, and in future work, we will investigate the conditions under which automated methods can be applied to obtain f . by conceptualizing the problem of treatment leakage in text data and investigating its impact, scholars developing causal methods can be better positioned to tailor their frameworks to reduce bias; domain scholars can better calibrate their data collection procedure to account for this leakage. ## treatment leakage in text data while the literature on dealing with confounding in observational studies is established @xcite , recent advances have been made in the analysis of text-based causal inference. indeed, text w is widely available in the health and social sciences @xcite , and can be used to proxy for some confounders, u , that would otherwise remain unobserved @xcite . if the text only contains information about u and no other factors, then w is a faithful representation of u and we denote it as w u . however, text, by its nature as a medium of creativity, rarely has fixed boundaries, and can contain information not only about confounders, but also leak information about the treatment assignment and its effects. the future-and backward-looking nature of text can exacerbate treatment leakage. documents that often contain backward looking temporally (e.g. in much of journalism) or has an unknown production date, will like contain information about the treatment and its effects. using these docu-ments directly for causal inference would inject post-treatment bias. conversely, documents that reference the future (e.g., many public-policy documents in the economy and polity) may also lead to unfavorable rmse if they predict the future well (see §2.1.2). as a result, a substantial amount of real-world text containing rich information about confounding factors might be affected by that language can reference the future, post-treatment state. ## text distillation as preprocessing text distillation is a form of text preprocessing. it has to target any text (e.g., tone, words, sentences) that belongs to w t , and remove it from w . thus, distillation ensures that the treatment signal is negated. as figure 1 , panel b. shows, if distillation is perfectly successful, it results in cutting the red arrow (from t to w ). the arrow is cut, because the distillation function has removed w t from w , supplying w u for causal analysis. ## experimental setup we use simulation to illustrate the dynamics of text distillation and build on the framework for evaluating text-based causal inference methods introduced by @xcite . we generate numerical covariates from the model in figure 1 ; the general procedure is described in §a, with implementation details in §b. parameters are selected so that ate estimates τ are biased if the estimator does not account for the unobserved confounder u . following wood-doughty et al. (2021), we generate documents, w , by sampling from an englishlanguage gpt-2 model @xcite . in contrast to their approach, text generation is conditioned not only on u but also on t . as described in detail in §a, we define paragraph-level topics, where some topics are associated with u , some with t , and some with a residual topic related only to other background variables (r in figure 1 ). for a given paragraph topic, we define a number of prompts and a distribution shift that increases the probability of generating topic-related keywords. as we simulate and record which paragraphs are affected by t and by u , our distillator f has oracle properties. we can then use f to investigate three idealized distillation scenarios. the first is when a distillator was not applied or the distillator failed to do any distillation f (w ) = w . it outputs the same corpus. the second is when it perfectly distills w , excluding all paragraphs affected by t . that is, apply f (w ) = w * such that w * = w u . the third scenario is when f was overly aggressive and accidentally removed not only paragraphs related to t but also those related to u , resulting in w * * . this corpus violates the proxy-faithfulness assumption that w * * fully measures u . then, we use the three corpora, one at a time, for causal inference. we use an inverse propensity weighting (ipw) estimator, fully described in appendix c. ## experiments and results based on the setting described in §4, our analysis produces six estimates, three based on distillation and three based on facts about the data-generating process. figure 2 shows all estimates. adjusting with the true treatment propensity τ 6 adjusting for confounder with distilled text τ 5 adjusting for confounder with over-distilled text τ 4 adjusting for confounder with non-distilled text τ 3 unadjusting for true confounder τ 2 adjusting for true confounder τ 1 -10 -5 0 5 estimated treatment effect the first estimate, τ1 = 5.5, is the baseline where all information is known to the outcome model, including u . because this linear model adjusting for u and x is equivalent to the data-generating model, and the estimated effect would be equal to the true value of 5 without sampling noise. the bootstrapped 95% confidence interval (ci) is 3.4 to 7.6. the second estimate, τ2 = -2.3, is obtained when u is omitted from the model to induce omitted variable bias (ci: -4.2, -0.1). the third estimate, τ3 , uses ipw to estimate the ate (see §c). here, we use the non-disitilled documents, w , to estimate propensities. as figure 2 shows, in the absence of distillation, the bias increases compared to conditioning on x alone, producing τ3 = -7.0 (ci: -9.4, -4.6). the fourth estimate, τ4 , applies overly aggressive distillation. this approach gives a result similar to the unadjusted estimate: τ4 = -2.9 (ci: -5.1, -0.6). the fifth estimate, τ5 , applies oracle distillation by removing the paragraphs we know were affected by t . using w * , the bias is reduced substantially, yielding an estimate τ5 = 3.5 (ci: 1.2, 5.8). as the ci of this τ includes the true τ = 5, we conclude that distillation successfully recovers τ . however, we note that this recovery is not perfect and will be affected by sampling and modeling parameters. the sixth estimate, τ6 , demonstrates the impact of model selection for the propensity estimator. using the true (simulated) propensity, the ipw estimate is τ6 = 4.9 (ci: 2.2, 7.6). this result shows that further gains could be made by careful model selection @xcite . figure 3 shows distributions of propensity values for τ3 , τ5 , and τ6 . without distillation (red), the estimated propensities cluster near 0 and 1. t is predicted almost perfectly, as mentioned in §2.1.1, causing the ipw estimate to to be similar to the unweighted one. conversely, with distillation, the predicted probabilities are now similar to the datagenerating propensities, and thereby, the resulting causal estimate is improved. ## discussion this paper shows the critical role of the notreatment-leakage assumption when using text for causal inference. while text is becoming an established data source, it may harbour valuable information about a confounder but also contaminating information about post-treatment effects. this issue has seen little discussion in text-based causal inference literature @xcite @xcite ), but has the potential to severely bias causal estimates, potentially leading to false discoveries or invalid policy recommendations in social and health settings @xcite @xcite . before discussing the implication of treatment leakage, three limitations should be considered. first, more work is required to show how the notreatment-leakage assumption operates under different covariance structures (i.e., different datagenerating processes). second, a larger simulation framework is needed to decompose estimator bias and variance. third, all results are based on simulated data, and more research is needed to generalize our insights to real data. although simulate data are idealized, they provide a benefit by allowing us to analyze the mechanics of treatment leakage and text distillation in a controlled environment. based on our simulated data, our analysis shows that when the no-treatment-leakage assumption is violated, effect estimates will be severely biased. in the presence of treatment leakage, scholars may be better off abstaining from using a non-distilled text to adjust for confounding. although, in theory, the best solution is to use a text distillation that removes all treatment leakage, in practice, using distillation can be difficult to achieve. therefore, one critical extension of our work is to develop methods that estimates the amount of treatment leakage in text. this estimate will enable applied researchers to make an informed decision about whether to adjusting for text-based confounding or abstain from it when leakage is high, and when text distillation is not an option. a second extensions is to develop a generalized framework that accounts for when the adjusted text represents multiple nodes in a dag (in combination with the confounding and the treatment or without them). while our article focuses on treatment leakage, there are other types of leakage when a single document is a function of combinations of dag nodes such as the outcome, confounder, treatment, mediator, or instrument. thus, a generalization of the no-treatment-leakage assumption is the no-node-leakage assumption. such methods will benefit from insights established in the literature on causal inference with proxies @xcite @xcite . a third extension is to develop a variety of text distillation methods, suitable for different application settings. researchers need alternative frameworks when human partitioning of text is not possible to achieve manually, because of corpus size or language complexities. automatic distillation could be attempted with additional assumptions, perhaps building from the literature on removing sensitive information in text representations @xcite ). treatment t i . for each paragraph j in the document, we draw a paragraph topic z ij , depending on the values of u i and t i , and then a prompt w 0 ij depending on the value of z ij . finally, we sample from the gpt-2 language model foot_0 to generate the paragraph text w ij , starting from the prompt w 0 ij and with a vocabulary distribution shift defined by z ij . algorithm 1 shows the pseudocode. algorithm 1 generation of synthetic data. in the pseudocode above, the functions f x , f u , f t , and f y define the distributions of the observed confounders, unobserved confounder, treatment and outcome, respectively. on the paragraph level, the function f z defines a categorical distribution over paragraph topics, and f w 0 a categorical distribution over prompts. similarly to @xcite , we use two mechanisms to condition the generation of a paragraph on a topic z: a prompt and a vocabulary distribution shift. the distribution shift is designed to promote a set of keywords related to the topic and we implement it by multiplying the language model probabilities by a topic-specific vector θ z of scale factors: