Text-Transport: Toward Learning Causal Effects of Natural Language (2023)

Abstract: As language technologies gain prominence in real-world settings, it is important to understand *how* changes to language affect reader perceptions. This can be formalized as the *causal effect* of varying a linguistic attribute (e.g., sentiment) on a reader’s response to the text. In this paper, we introduce Text-Transport, a method for estimation of causal effects from natural language under any text distribution. Current approaches for valid causal effect estimation require strong assumptions about the data, meaning the data from which one *can* estimate valid causal effects often is not representative of the actual target domain of interest. To address this issue, we leverage the notion of distribution shift to describe an estimator that *transports* causal effects between domains, bypassing the need for strong assumptions in the target domain. We derive statistical guarantees on the uncertainty of this estimator, and we report empirical results and analyses that support the validity of Text-Transport across data settings. Finally, we use Text-Transport to study a realistic setting—hate speech on social media—in which causal effects do shift significantly between text domains, demonstrating the necessity of transport when conducting causal inference on natural language.

## introduction what makes a comment on a social media site seem toxic or hateful @xcite ? could it be the use of profanity, or a lack of insight? what makes a product review more or less helpful to readers @xcite ? is it the certainty of the review, or perhaps the presence of justifications? as language technologies are increasingly deployed in real-world settings, interpretability and explainability in natural language processing have become paramount @xcite . particularly desirable is the ability to understand how changes to language affect reader perceptions-formalized in statistical terms as the causal effect of varying a linguistic attribute on a reader's response to the text (figure 1 ). a core technical challenge for causal inference is that valid causal effects can only be estimated on data in which certain assumptions are upheld: namely, data where confounding is either fully measured or absent entirely @xcite . since confounding-the presence of factors that affect both the reader's choice of texts to read and the reader's response (e.g., political affiliation, age, mood)-is extremely difficult to measure fully in text settings, estimation of causal effects from natural language remains an open problem. one resource-intensive approach is to run a randomized experiment, which eliminates confounding by ensuring that respondents are randomly assigned texts to read @xcite . however, effects estimated from randomized experiments may not generalize outside of the specific data on which they were conducted @xcite . therefore, learning causal effects for a new target domain can require a new randomized experiment each time. in this paper, we propose to bypass the need for strong data assumptions in the target domain by framing causal effect estimation as a distribution shift problem. we introduce text-transport, a method for learning text causal effects in any target domain, including those that do not necessarily fulfill the assumptions required for valid causal inference. leveraging the notion of distribution shift, we define a causal estimator that transports a causal effect from a causally valid source domain (e.g., a randomized experiment) to the target domain, and we derive statistical guarantees for our proposed estimator that show that it has desirable properties that can be used to quantify its uncertainty. we evaluate text-transport empirically using a benchmarking strategy that includes knowledge about causal effects in both the source domain and the target domain. we find that across three data settings of increasing complexity, text-transport estimates are consistent with effects directly estimated on the target domain, suggesting successful transport of effects. we further study a realistic setting-user responses to hateful content on social media sites-in which causal effects do change significantly between text domains, demonstrating the utility of transport when estimating causal effects from natural language. finally, we conduct analyses that examine the intuition behind why text-transport is an effective estimator of text causal effects. ## problem setting consider a collection of texts (e.g., documents, sentences, utterances) x , where person i (@xmath0, . . . , n ) is shown a text x i from x . using the potential outcomes framework @xcite ; @xcite , let y i (x) denote the potential response of respondent i if they were to read text x, where y i : @xmath1. without loss of generality, assume that each respondent in reality reads only one text x i , so their observed response is y i (x i ). then the average response µ(p ) across the n respondents when texts x are drawn from a distri-bution p is given by: now let x be parameterized as @xmath2, a c (x)}, where a(x) is the text attribute of interest and a c (x) denotes all other attributes of the text x. note that for a text causal effect to be meaningful, a(x) must be interpretable. this may be achieved by having a human code a(x) or using a lexicon or other automatic coding method. again for simplicity, we assume a(x) ∈ {0, 1}. definition 1 (natural causal effect of an attribute). let p 1 (x) be a distribution such that a(x) = 1 and a c (x) ∼ p (a c (x)|a(x) = 1), and let p 0 (x) be a distribution such that a(x) = 0 and a c (x) ∼ p (a c (x)|a(x) = 0). then the causal effect of a(x) on y is given by: here, τ * is the natural effect of a(x). linguistic attributes are subject to aliasing @xcite , in which some other linguistic attributes (e.g., the k-th linguistic attribute a c (x) k ) may be correlated with both the linguistic attribute of interest a(x) and the response y , such that for example, optimism may naturally co-occur with positive emotion, meaning that the natural effect of optimism also contains in part the effect of positive emotion. in contrast, the isolated effect would contain only the effect of optimism. in this paper, we choose to focus on natural effects due to the way linguistic attributes manifest in real-world language. that is, since optimism nearly always co-occurs with positive emotion, it may be difficult to interpret the effect of optimism when positive emotion is removed (the isolated effect), so we instead focus on their collective natural effect. our goal is then to learn τ t , the natural causal effect of the attribute a(x) on response y in the text domain of interest p t . we consider use cases where it is not possible to directly estimate the effect from target data x ∼ p t , either because p t does not fulfill the assumptions required for valid causal inference or simply because the response y is not measured in the domain of interest. ## text-transport to estimate causal effects under a target text distribution p t -without computing effects on p t directly-we propose text-transport, a method for transporting a causal effect from a source text distribution p r that does fulfill the assumptions required for valid causal inference and with respect to which p t is absolutely continuous. our approach can help to generalize the causal findings of p r , which are specific to the data domain of p r , to any text distribution of interest p t . for mathematical convenience, we consider the source distribution p r to be a randomized experiment. we note that any crowdsourced dataset in which samples are randomly assigned to crowdworkers can be considered a randomized experiment. ## experimental setup we conduct empirical evaluations to assess the validity of causal effects obtained through text-transport in three different data settings of increasing complexity ( ## related work text-transport draws on a diverse body of prior work from machine learning and causal inference, including the literature on domain adaptation, distribution shift, and generalizability and transportability. we build on methods from these fields to define a novel, flexible estimator of causal effects in natural language. text causal effects. a number of approaches have been proposed for estimating causal effects from natural language. @xcite construct a conceptual framework for causal inference with text that relies on specific data splitting strategies, while @xcite describe a procedure for learning words that are most predictive of a desired response. however, the interpretability of the learned effects from these works is limited. in a subsequent paper, @xcite introduce a method for estimating effects of linguistic proper-ties from observational data. while this approach targets isolated effects of linguistic properties, it requires responses to be measured on the target domain, and it accounts only for the portion of confounding that is contained with the text. finally, in a recent paper, fong and grimmer (2021) conduct randomized experiments over text attributes to determine their effects. while allowing for valid causal inference, the resulting constructed texts are artificial in nature and constitute a clear use case for text-transport, which can transport effects from the less-realistic experimental text domain to more naturalistic target domains. domain adaptation, distribution shift, and transportability. importance weighting has been widely used in the domain adaptation literature to help models learn under distribution shift @xcite . models are trained with importanceweighted loss functions to account for covariate and label shift @xcite @xcite , correct for selection bias @xcite @xcite , and facilitate offpolicy reinforcement learning @xcite . in parallel, a line of work studying the external validity of estimated causal effects has emerged within statistical causal inference @xcite . these works aim to understand the conditions under which causal effects estimated on specific data can generalize or be transported to broader settings @xcite . prior work has also used density ratio-style importance weights to estimate average causal effects with high-dimensional interventions @xcite . we emphasize that text-transport is conceptually novel and methodologically distinct from these prior works. in our work, we explore an open problem-causal effect estimation from text-and define a new framework for learning causal effects from natural language. we propose a novel solution that uses tools from the domain adaptation and transportability literature, and we introduce novel methods for estimating natural effects in practice in language settings. ## conclusion in this paper, we study the problem of causal effect estimation from text, which is challenging due to the highly confounded nature of natural language. to address this challenge, we propose text-transport, a novel method for estimating text causal effects from distributions that may not fulfill the assumptions required for valid causal inference. we conduct empirical evaluations that support the validity of causal effects estimated with text-transport, and we examine a realistic data setting-hate speech and toxicity on social media sites-in which text-transport identifies significant shifts in causal effects between text domains. our results reinforce the need to account for distribution shift when estimating text-based causal effects and suggest that text-transport is a compelling approach for doing so. these promising initial findings open the door to future exploration of causal effects from complex unstructured data like language, images, and multimodal data.