Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates (2020)

Abstract: Many applications of computational social science aim to infer causal conclusions from non-experimental data. Such observational data often contains confounders, variables that influence both potential causes and potential effects. Unmeasured or latent confounders can bias causal estimates, and this has motivated interest in measuring potential confounders from observed text. For example, an individual’s entire history of social media posts or the content of a news article could provide a rich measurement of multiple confounders. Yet, methods and applications for this problem are scattered across different communities and evaluation practices are inconsistent. This review is the first to gather and categorize these examples and provide a guide to data-processing and evaluation decisions. Despite increased attention on adjusting for confounding using text, there are still many open problems, which we highlight in this paper.

## introduction in contrast to descriptive or predictive tasks, causal inference aims to understand how intervening on one variable affects another variable @xcite @xcite @xcite . specifically, many applied researchers aim to estimate the size of a specific causal effect, the effect of a single treatment variable on an outcome variable. however, a major challenge in causal inference is addressing confounders, variables that influence both treatment and outcome. for example, consider estimating the size of the causal effect of smoking (treatment) on life expectancy (outcome). occupation is a potential confounder that may influence both the propensity to smoke and life expectancy. estimating the effect of treatment on outcome without accounting for this confounding could result in figure 1 : left: a causal diagram for text that encodes causal confounders, the setting that is focus of this review paper. the major assumption is that latent confounders can be measured from text and those confounder measurements can be used in causal adjustments. right: an example application in which practitioner does not have access to the confounding variable, occupation, in structured form but can measure confounders from unstructured text (e.g. an individual's social media posts). to eliminate confounding bias, one approach is to perform randomized controlled trials (rcts) in which researchers randomly assign treatment. yet, in many research areas such as healthcare, education, or economics, randomly assigning treatment is either infeasible or unethical. for instance, in our running example, one cannot ethically randomly assign participants to smoke since this could expose them to major health risks. in such cases, researchers instead use observational data and adjust for the confounding bias statistically with methods such as matching, propensity score weighting, or regression adjustment ( §5). in causal research about human behavior and society, there are potentially many latent confounding variables that can be measured from unstructured text data. text data could either (a) serve as a surrogate for potential confounders; or (b) the language of text itself could be a confounder. our running example is an instance of text as a surrogate: a researcher may not have a record of an individual's occupation but could attempt to measure this variable from the individual's entire history of social media posts (see fig. 1 ). an example of text as a direct confounder: the linguistic content of social media posts could influence censorship (treatment) and future posting rates (outcome) @xcite . a challenging aspect of this research design is the high-dimensional nature of text. other work has explored general methods for adjusting for highdimensional confounders @xcite @xcite @xcite . however, text data differ from other high-dimensional data-types because intermediate confounding adjustments can be read and evaluated by humans ( §6) and designing meaningful representations of text is still an open research question. 1 even when applying simple adjustment methods, a practitioner must first transform text into a lower-dimensional representation via, for example, filtered word counts, lexicon indicators, topic models, or embeddings ( §4). an additional challenge is that empirical evaluation in causal inference is still an open research area @xcite ) and text adds to the difficulty of this evaluation ( §7). we narrow the scope of this paper to review methods and applications with text data as a causal confounder. in the broader area of text and causal inference, work has examined text as a mediator @xcite ), text as treatment @xcite @xcite ), text as outcome @xcite , causal discovery from text @xcite , and predictive (granger) causality with text @xcite @xcite . outside of this prior work, there has been relatively little interaction between natural language processing (nlp) research and causal inference. nlp has a rich history of applied modeling and diagnostic pipelines that causal inference could draw upon. because applications and methods for text 1 for instance, there have been four workshops on representation learning at major nlp conferences in the last four years @xcite @xcite . as a confounder have been scattered across many different communities, this review paper aims to gather and unify existing approaches and to concurrently serve three different types of researchers and their respective goals: ## applications in paper treatment outcome(s) confounder text data text rep. adjustment method johansson et al. (2016) viewing device (mobile or desktop) reader's experience news content news word counts causal-driven rep. learning de choudhury et al. (2016) word use in mental health community user transitions to post in suicide community previous text written in a forum social media (reddit) word counts stratified propensity score matching de choudhury and kiciman (2017) language of comments user transitions to post in suicide community user's previous posts and comments received social media (reddit) unigrams and bigrams stratified propensity score matching falavarjani et al. (2017) exercise (foursquare checkins) shift in topical interest on twitter pre-treatment topical interest shift social media (twitter, foursquare) topic models matching olteanu et al. (2017) current word use future word use past word use social media (twitter) top unigrams and bigrams stratified propensity score matching pham and shen (2017) group vs. individual loan requests time until borrowers get funded loan description microloans (kiva) pre-trained embeddings + neural networks a-iptw, tmle kiciman et al. (2018) alcohol mentions college success (e.g. study habits, risky behaviors, emotions) previous posts social media (twitter) word counts stratified propensity score matching sridhar et al. (2018) exercise mood mood triggers users' text on mood logging apps word counts propensity score matching saha et al. (2019) self-reported usage of psychiatric medication mood, cognition, depression, anxiety, psychosis, and suicidal ideation users' previous posts social media (twitter) word counts + lexicons + supervised classifiers stratified propensity score matching sridhar and getoor (2019) tone of replies changes in sentiment speaker's political ideology debate transcripts topic models + lexicons regression adjustment, iptw, a-iptw veitch et al. (2019) presence of a theorem rate of acceptance subject of the article scientific articles bert causal-driven rep. learning + regression adjustment, tmle roberts et al. (2020) perceived gender of author number of citations content of article international relations articles topic models + propensity score coarsened exact matching roberts et al. (2020) censorship subsequent censorship and posting rate content of posts social media (weibo) topic models + propensity score coarsened exact matching the twitter accounts of college-age students and extract alcohol mentions and indicators of college success (e.g., study habits, risky behaviors, and emotions) from their twitter posts. they condition on an individual's previous posts (temporally previous to measurements of treatment and outcome) as confounding variables since they do not have demographic data. they represent text as word counts and use stratified propensity score matching to adjust for the confounding bias. the study finds the effects of alcohol use include decreased mentions of study habits and positive emotions and increased mentions of potentially risky behaviors. text as a surrogate for confounders. traditionally, causal research that uses human subjects as the unit of analysis would infer demographics via surveys. however, with the proliferation of the web and social media, social research now includes large-scale observational data that would be challenging to obtain using surveys @xcite . this type of data typically lacks demographic information but may contain large amounts of text written by participants from which demographics can be extracted. in this space, some researchers are specific about the confounders they want to extract such as an individual's ideology @xcite or mood @xcite . other researchers condition on all the text they have avail-able and assume that low-dimensional summaries capture all possible confounders. for example, researchers might assume that text encodes all possible confounders between alcohol use and college success @xcite or psychiatric medication and anxiety @xcite . we dissect and comment on this assumption in section 8. open problems: nlp systems have been shown to be inaccurate for low-resource languages @xcite , and exhibit racial and gender disparity @xcite . furthermore, the ethics of predicting psychological indicators, such as mental health status, from text are questionable @xcite . it is unclear how to mitigate these disparities when trying to condition on demographics from text and how nlp errors will propagate to causal estimates. language as confounders. there is growing interest in measuring language itself (e.g. the sentiment or topical content of text) as causal confounders. for example, @xcite examine how the perceived gender of an author affects the number of citations that an article receives. however, an article's topics (the confounders) are likely to influence the perceived gender of its author (reflecting an expectation that women write about certain topics) and the number of citations of that article ("hotter" topics will receive more citations). other domains that analyze language as a confounder include news @xcite , social media @xcite , and loan descriptions @xcite . see section 4 for more discussion on the challenges and open problems of inferring these latent aspects of language. ## estimating causal effects two predominant causal inference frameworks are structural causal models (scm) @xcite and potential outcomes @xcite , which are complementary and theoretically connected @xcite @xcite . while their respective goals substantially overlap, methods from structural causal models tend to emphasize conceptualizing, expressing, and reasoning about the effects of possible causal relationships among variables, while methods from potential outcomes tend to emphasize estimating the size or strength of causal effects. ## measuring confounders via text after drawing the causal graph, the next step is to use available text data to recover latent confounders. some approaches pre-specify the confounders of interest and measure them from text, p (z | x). others learn confounders inductively and use a low-dimensional representation of text as the confounding variable z in subsequent causal adjustments. pre-specified confounders. when a practitioner can specify confounders they want to measure from text (e.g., extracting "occupation" from text in our smoking example), they can use either (1) lexicons or (2) trained supervised classifiers as the instrument of measurement. lexicons are word lists that can either be hand-crafted by researchers or taken off-the-shelf. 2019 ) also build machine learning classifiers for users' mental states (e.g., depression and anxiety) and apply these classifiers on twitter posts that are temporally prior to treatment. if these classifiers accurately recover mental states and there are no additional latent confounders, then conditioning on the measured mental states renders treatment independent of potential outcomes. open problems: since nlp methods are still far from perfectly accurate, how can one mitigate error that arises from approximating confounding variables? closely related to this question is effect restoration which addresses error from using proxy variables (e.g., a father's occupation) in place of true confounders (e.g, socioeconomic status) @xcite . wood-6 see morgan and winship (2015) pgs. 33-34 on both the necessity and difficulty of specifying a causal graph for applied social research. time-ordering can be particularly helpful when encoding causal relationships (for instance, there cannot be an arrow pointing from variable a to variable b if b preceded a in time). @xcite build upon effect restoration for causal inference with text classifiers, but there are still open problems in accounting for error arising from other text representations and issues of calibration @xcite and prevalence estimation @xcite in conjunction with nlp. ideas from the large literature on measurement error models may also be helpful @xcite @xcite . ## adjusting for confounding bias given a set of variables z that satisfy the backdoor criterion ( §3.2), one can use the backdoor adjustment to estimate the causal quantity of interest, conditioning on all confounders is often impractical in high-dimensional settings such as those found in natural language. we provide an overview of methods used by applications in this review that approximate such conditioning, leading to unbiased estimates of treatment effect; however, we acknowledge this is not an exhaustive list of methods and direct readers to more extensive guides @xcite . open problems: causal studies typically make an assumption of overlap, also known as common support or positivity, meaning that any individual has a non-zero probability of assignment to each treatment condition for all possible values of the covariates: ∀z, @xcite show that as the dimensionality of covariates grows, strict overlap converges to zero. what are the implications of these results for high-dimensional text data? ## human evaluation of intermediate steps text data has the advantage of being interpretablematched pairs and some low-dimensional representations of text can be read by humans to evaluate their quality. when possible, we suggest practitioners use (1) interpretable balance metrics and/or (2) human judgements of treatment propensity to evaluate intermediate steps of the causal estimation pipeline. ## evaluation of causal methods because the true causal effects in real-world causal inference are typically unknown, causal evaluation is a difficult and open research question. as algorithmic complexity grows, the expected performance of causal methods can be difficult to estimate theoretically @xcite . other causal evaluations involve synthetic data. however, as @xcite discuss, synthetic data has no "unknown unknowns" and many researcher degrees of freedom, which limits their effectiveness. thus, we encourage researchers to evaluate with constructed observational studies or semi-synthetic datasets, although measuring latent confounders from text increases the difficulty of creating realistic datasets that can be used for empirical evaluation of causal methods. ## discussion and conclusion computational social science is an exciting, rapidly expanding discipline. with greater availability of text data, alongside improved natural language processing models, there is enormous opportunity to conduct new and more accurate causal observational studies by controlling for latent confounders in text. while text data ought to be as useful for measurement and inference as "traditional" lowdimensional social-scientific variables, combining nlp with causal inference methods requires tackling major open research questions. unlike predictive applications, causal applications have no ground truth and so it is difficult distinguish modeling errors and forking paths from the true causal effects. in particular, we caution against using all available text in causal adjustment methods without any human validation or supervision, since one cannot diagnose any potential errors. solving these open problems, along with the others presented in this paper, would be a major advance for nlp as a social science methodology.