Causal Effects of Linguistic Properties (2021)

Abstract: We consider the problem of using observational data to estimate the causal effects of linguistic properties. For example, does writing a complaint politely lead to a faster response time? How much will a positive product review increase sales? This paper addresses two technical challenges related to the problem before developing a practical method. First, we formalize the causal quantity of interest as the effect of a writer’s intent, and establish the assumptions necessary to identify this from observational data. Second, in practice, we only have access to noisy proxies for the linguistic properties of interest—e.g., predictions from classifiers and lexicons. We propose an estimator for this setting and prove that its bias is bounded when we perform an adjustment for the text. Based on these results, we introduce TextCause, an algorithm for estimating causal effects of linguistic properties. The method leverages (1) distant supervision to improve the quality of noisy proxies, and (2) a pre-trained language model (BERT) to adjust for the text. We show that the proposed method outperforms related approaches when estimating the effect of Amazon review sentiment on semi-simulated sales figures. Finally, we present an applied case study investigating the effects of complaint politeness on bureaucratic response times.

## introduction social scientists have long been interested in the causal effects of language, studying questions like: to study the causal effects of linguistic properties, we must reason about interventions: what would the response time for a complaint be if we could make that complaint polite while keeping all other properties (topic, pragmatics, etc.) fixed? although it is sometimes feasible to run such experiments where text is manipulated and outcomes are recorded @xcite , analysts typically have observational data consisting of texts and outcomes obtained without intervention. this paper formalizes the estimation of causal effects of linguistic properties in observational settings. estimating causal effects from observational data requires addressing two challenges. first, we need to formalize the causal effect of interest by specifying the hypothetical intervention to which it corresponds. the first contribution of this paper is articulating the causal effects of linguistic properties; we imagine intervening on the writer of a text document and telling them to use different linguistic properties. the second challenge of causal inference is identification: we need to express causal quantities in terms of variables we can observe. often, instead of the true linguistic property of interest we have access to a noisy measurement called the proxy label. analysts typically infer these values from text with classifiers, lexicons, or topic models @xcite @xcite @xcite . the second contribution of this paper is establishing the assumptions we need to recover the true effects of a latent linguistic property from these noisy proxy labels. in particular, we propose an adjustment for the confounding information in a text document and prove that this bounds the bias of the resulting estimates. the third contribution of this paper is practical: an algorithm for estimating the causal effects of linguistic properties. the algorithm uses distantly supervised label propagation to improve the proxy label @xcite @xcite , then bert to adjust for the bias due to text @xcite . we demonstrate the method's accuracy with partially-simulated amazon reviews and sales data, perform a sensitivity analysis in situations where assumptions are violated, and show an application to consumer finance complaints. data and a package for performing textbased causal inferences is available at https:// github.com/rpryzant/causal-text . ## causal inference background causal inference from observational data is wellstudied @xcite @xcite . in this setting, analysts are interested in the effect of a treatment t (e.g., a drug) on an outcome y (e.g., disease progression). for ease, we consider binary treatments. the average treatment effect (ate) on the outcome y is, this is because confounding variables c are associated with both the treatment and outcome, inducing non-causal associations between them, referred to as open backdoor paths @xcite . when all the confounding variables are observed, we can write the ate in terms of observed variables using the backdoor-adjustment formula @xcite , for example, if the confounding variable c is discrete, we group the data into values of c, calculate the average difference in outcomes between the treated and untreated samples of each group, and take the average over groups. ## causal effects of linguistic properties we are interested in the causal effects of linguistic properties. to formalize this as a treatment, we imagine intervening on the writer of a text, e.g., telling people to write with a property (or not). we show that to estimate the effect of using a linguistic property, we must consider how a reader of the text perceives the property. these dual perspectives of the reader and writer are well studied in linguistics and nlp; 1 we adapt the idea for causal inference. figure 1 illustrates a causal model of the setting. let w be a text document and let t (binary) be whether or not a writer uses a particular linguistic property of interest. 2 for example, in consumer complaints, the variable t can indicate whether the writer intends to be polite or not. the outcome is a variable y , e.g., how long it took for this complaint to be serviced. let z be other linguistic properties that the writer communicated (consciously or unconsciously) via the text w , e.g. topic, brevity or eloquence. the linguistic properties t and z are typically correlated, and both variables affect the outcome y . 1 literary theory argues that language is subject to two perspectives: the "artistic" pole -the text as intended by the author -and the "aesthetic" pole -the text as interpreted by the reader @xcite . the noisy channel model @xcite connects these poles by supposing that the reader perceives a noisy version of the author's intent. this duality has also been modeled in linguistic pragmatics as the difference between speaker meaning and literal or utterance meaning @xcite @xcite . gricean pragmatic models like rsa @xcite similarly formalize this as the reader using the literal meaning to help make inferences about the speaker's intent. the reason is that we would need to adjust for the unobserved linguistic properties z, which create open backdoor paths because they are correlated with both the treatment t and outcome y (figure 1 ). to solve this problem, we observe that the reader is the one who produces outcomes. readers use the text w to perceive a value for the property of interest (captured by the variable t ) as well as other properties (captured by z) then produce the outcome y based on these perceived values. for example, a customer service representative reads a consumer complaint, judges whether (among other things) the complaint is polite or not, and chooses how quickly to respond based on this. suppose that the following assumptions hold: ## substituting proxy labels if we observed t , the reader's perception of the linguistic property of interest, then we could proceed by estimating the effect ψ rea. (equivalently, ψ wri. ). however, in most settings, one does not observe the linguistic properties that a writer intends to use (t and z) or that a reader perceives ( t and the information in z). instead, one uses a classifier or lexicon to predict values for this property from the text, producing a proxy label t (e.g. predicted politeness). for this setting, where we only have access to proxy labels, we introduce the estimand ψ proxy which substitutes the proxy t for the unobserved treatment t in the effect ψ rea. : this estimand only requires an adjustment for the confounding information z. we show how to extract this information using pretrained language models in section 5. prior work on causal inference with proxy treatments @xcite requires an adjustment using the measurement model p ( t | t ), i.e. the true relationship between the proxy label t and its target t , which is typically unobserved. in contrast, the estimand ψ proxy does not require the measurement model. the following result shows that the estimand ψ proxy only attenuates the ate that we want, ψ rea. . that is, the bias due to proxy treatments is benign; it can only decrease the magnitude of the effect but it does not change the sign. the proof is in appendix e. this result shows that the proposed estimand ψ proxy , which we can estimate, is equal to the ate ψ rea. that we want, minus a bias term related to measurement error. in particular, if the classifier is better than chance and the treatment effect sign is homogeneous across possible texts -i.e., it always helps or always hurts, an assumption the analyst must carefully assess -then the bias term is positive with the degree of attenuation dependent on the error rate of the proxy label t . the result tells us to construct the most accurate proxy treatment t possible, so long as we adjust for the confounding part of the text. 5 this is a novel result for causal inference with proxy treatments and sidesteps the need for the measurement model. ## textcause , a causal estimation procedure we introduce a practical algorithm for estimating the causal effects of linguistic properties. motivated by theorem 2, we first describe an approach for improving the accuracy of proxy labels. we then use the improved proxy labels, text and outcomes to fit a model that extracts and adjusts for the confounding information in the text ( z). in practice, one may observe additional covariates c that capture confounding properties, e.g., the product that a review is about or complaint type. we will include these covariates in the estimation algorithm. ## experiments we evaluate the proposed algorithm's ability to recover causal effects of linguistic properties. since ground-truth causal effects are unavailable without randomized controlled trials, we produce a semisynthetic dataset based on amazon reviews where only the outcomes are simulated. we also conduct a real-world study using real-world complaints and bureaucratic response times. our key findings are ## related work our focus fits into a body of work on text-based causal inference that includes text as treatments @xcite @xcite ), text as outcomes @xcite , and text as confounders @xcite ; @xcite ; see @xcite for a review of that space). we build on @xcite , which proposed a bert-based text adjustment method similar to our w-adjust algorithm. this paper is related to work by @xcite , which discusses assumptions needed to estimate causal effects of text-based treatments in randomized controlled trials. there is also work on discovering causal structure in text, as topics with latent variable models @xcite and as words and n-grams with adversarial learning @xcite and residualization @xcite . there is also a growing body of applications in the social sciences @xcite @xcite @xcite @xcite . this paper also fits into a long-standing body of work on measurement error and causal inference @xcite @xcite @xcite @xcite . most of this work deals with proxies for confounding variables. the present paper is most closely related to @xcite , which also deals with proxy treatments, but instead proposes an adjustment using the measurement model. ## conclusion this paper addressed a setting of interest to nlp and social science researchers: estimating the causal effects of latent linguistic properties from observational data. we clarified critical ambiguities in the problem, showed how causal effects can be interpreted, presented a method, and demonstrated how it offers practical and theoretical advantages over the existing practice. we also release a package for performing text-based causal inferences. 10 this work opens new avenues for further conceptual, methodological, and theoretical refinement. this includes improving non-lexicon based treatments, heterogeneous effects, overlap violations, counterfactual inference, ethical considerations, extensions to higher-dimensional outcomes and covariates, and benchmark datasets based on paired randomized controlled trials and observational studies.