Sensitivity Analysis for Causal Mediation through Text: an Application to Political Polarization (2021)

Abstract: We introduce a procedure to examine a text-as-mediator problem from a novel randomized experiment that studied the effect of conversations on political polarization. In this randomized experiment, Americans from the Democratic and Republican parties were either randomly paired with one-another to have an anonymous conversation about politics or alternatively not assigned to a conversation — change in political polarization over time was measured for all participants. This paper analyzes the text of the conversations to identify potential mediators of depolarization and is faced with a unique challenge, necessitated by the primary research hypothesis, that individuals in the control condition do not have conversations and so lack observed text data. We highlight the importance of using domain knowledge to perform dimension reduction on the text data, and describe a procedure to characterize indirect effects via text when the text is only observed in one arm of the experiment.

## introduction increasing large and varied text corpora are becoming available to researchers. especially in the field of computational social science, text data are yielding new insights and opening new areas of study @xcite @xcite . text are being used to study who sets the political agenda @xcite , measure partisanship in congressional speeches @xcite , and legislator attitudes expressed on twitter @xcite ). an understudied area of this rapidly expanding field is how to perform causal inference with text data, especially when text is the treatment or outcome @xcite ). text data pose unique challenges due to the complex and highdimensional structure that they impose on the already complex task of causal inference. for instance, concerns that social media might be caus-ing political polarization by increasing ideological segregation @xcite or spreading disinformation @xcite must disentangle the effect of exposure to social media text on polarization and the propensity of highly polarized individuals to be active on social media. the contributions of this paper are two-fold: (1) we introduce a sensitivity analysis for mediation when the information on the mediator is observed for only one group in an experiment. (2) we demonstrate the difficulty of using unsupervised text models for performing a causal analysis. to illustrate these contributions, we analyze data from a novel randomized experiment studying the dynamics of political polarization @xcite . american voters were randomly paired across party lines to have meaningful, political conversations. of particular interest is the text of the conversations. we want to characterize the kind of conversations that cause depolarization. this is inherently a question of causal mediation: we are interested in both the direct effect of simply having a conversation and the indirect effect of the conversation content. the study we analyze follows a common design where information must be observed asymetrically for the different treatment arms. specifically, text data are only available for the units who had conversations and are not available for control individuals, making explicit causal mediation impossible. this asymmetry is required by the research question. the original study sought to estimate the effect of having a conversation with an out-partisan about politics, which necessitates the control group having no conversation. contrasting treatment with a control that had a non-political conversation, for example, would only estimate the effect of having a conversation with an out-partisan about politics. this asymmetry is also a common feature of peer encouragement or counseling studies where treated units have a conversation with a peer or professional and control units have no interactions @xcite @xcite . the natural followup to such studies is to identify what kind of conversations produce the best results. however, the experimental data cannot answer such questions because those mediators are missing for the control group. to accommodate such designs, we develop a procedure for a mediation sensitivity analysis designed to benchmark the observed correlation with the causal mediation effects that could be observed if control units had produced text data. our work also highlights the difficulty of using unsupervised models of text in the context of causal inference. in our application, we have expert knowledge of likely causal pathways, but, because these models do not leverage this expert knowledge, they are unable to identify semantically meaningful conversation features. moreover, the features they do identify appear to not play a significant role in the causal story of the experimental data. the outline of the paper is as follows: first, we introduce our illustration data and describe the causal and text complications. section 3 describes the causal inference framework, concentrating on effect mediation and on how to incorporate text data into causal inference pipelines, and prior work on these issues. section 4 demonstrates our finding that partner politeness is correlated with depolarization. section 5 develops the mediation sensitivity procedure for this result. section 6 demonstrates the failure of unsupervised text models to capture causal mechanisms. section 7 concludes. ## polarization and insights from text a growing body of research raises concerns that social media are increasing ideological segregation and incivility between political parties in the united states @xcite . a particular concern, is that increasing exposure to elites from the other side may produce a backlash effect that increases polarization @xcite . in this paper we revisit data from a study that experimentally tested whether prolonged out-group contact could decrease polarization in the u.s. by having democrats and republicans engage on an anonymous chat platform @xcite . in february 2020, approximately 1,500 republicans and democrats were recruited by a prominent survey firm and given a survey to measure their political views. the questions covered both issue-based polarization (how close one is to each party on policy views) and affect-based polarization (one's sentiment toward the other party). individuals randomized to treatment were sent a seemingly-unrelated invitation to download a mobile chat application within 48 hours of completing the survey. each person who logged into the app was randomly assigned a partner from the other party and prompted to discuss either gun control or immigration. politics were not mentioned in the recruitment dialog or prior to logging into the app. after the conversations were finished, another seemingly unrelated survey was sent to all study participants that contained the same questions as the first survey. within-person depolarization was measured by averaging the difference in post-and pre-survey responses to all polarization questions. the original study reveals that individuals assigned to treatment became significantly more depolarized @xcite . this paper focuses on analyzing the text of the conversations. we want to identify what conversation features caused the depolarization. however, we cannot rely on the randomization to identify causal effects for two reasons. the first and primary reason is that the control group did not have conversations, so we cannot compare treatment and control text. the original experiment was designed to estimate the effect of having a conversation about politics with an out-partisan, and as such, individuals in the control group had no conversations. the second reason we cannot rely on the randomization to identify causal effects is that the conversation text is properly thought of as a mediator variable, something causally affected by treatment that also affects the outcome. randomized treatment does not guarantee that there is no unobserved confounding of the mediator-outcome relationship. the conversation itself is both an outcome and exposure of interest. use of text in both of those contexts is challenging, and the subject of much of our literature review. we address these challenges by first, relying on prior work that identifies politeness or civility as a key feature in producing persuasive text, measuring politeness of messages received by individuals in treatment, then linking that measure to the outcome of interest, depolarization. next, we impute politeness measures for control individuals in a way that preserves the observed relationship between politeness and depolarization among the treated and test how large the difference in politeness between treatment and control must be to observe significant mediation. ## causal inference and prior work the general goal behind causal inference is to understand "what if?" questions -what happens to an outcome of interest y if one intervenes to set variable t . information is frequently collected alongside the intervention and outcome; modern causal inference is often concerned with how this additional information interacts with the causal effects of interest. in this paper we are concerned with one such mechanism -that of causal mediation -and how text acts as part of that mechanism. ## preliminary analysis of text data the work on politeness and incivility lead us to focus on politeness as a potential mediator of depolarization. individuals exposed to more polite partners may change their prior beliefs about the other side more than those exposed to less polite partners. we use the r package politeness to identify linguistic features of politeness with sentence parsing and dictionary methods @xcite . this package extends the work of @xcite and @xcite by combing the indicators of politeness in both works. 3 to measure exposure to politeness, we extract all of the politeness features for each message sent by participants. then, we standardize each feature to be mean zero variance one, and sum the results across all messages received by each user. we focus on messages received rather than sent because we want to measure how one's conversation partner affects one's own depolarization. we sum rather than average across politeness features because we want to measure total experienced politeness rather than a per-message measure. we wish to characterize a short and polite conversation as less exposure than a long and polite conversation. we refer to this constructed measure as the politeness index. note that the random assignment of partners makes the no unmeasured confounding assumption plausible. first, we examine the relationship between the politeness index and depolarization among the treated units who finished their conversations (completed at least 10 exchanges). we regress depolarization on politeness both in the full sample and split by party, controlling for demographic features. the results are shown in we also look at how politeness and the relationship between depolarization and politeness differ by conversation topic. while participants assigned to talk about immigration were significantly more polite, there was no significant difference in the relationship between depolarization and politeness by topic. figure 2 shows scatter-plots of politeness and the depolarization index by topic. adding conversation topic to the regressions in crucially, we cannot conclude that the above are causal relationships based on these regressions. while partners were randomly assigned, partner politeness was not. if partner politeness is affected by one's own messages, as is likely the case, then there could be unmeasured confoundimmigration. as such, we only use the context-independent politeness features, e.g. whether a participant expresses gratitude, not the specific projection of those features learned in a different, non-political context. ing whereby certain participant features cause both polite replies by the conversation partner and depolarization. we believe our demographic controls cover many of the potential confounders but must rely on the untestable assumption that there are no other confounders. in the next section we develop a framework for understanding the causal nature of the relationship between conversations, the text and depolarization. ## mediation sensitivity we would like to use politeness as a mediator and estimate the direct and indirect effects (introduced in section 3) using the structural equation model: the total effect, identified by regressing y on t without m , is τ + αβ. while control individuals clearly have a perceived level of politeness for the opposite party, because they did not participate in a conversation, there is no politeness to measure for them. this means β cannot be estimated. what we can do is benchmark the strength of the relationship identified in ## conclusion this paper contributes to the methodological literature on text in causal inference by considering text as a mediator, i.e. considering text simultaneously as both a treatment and outcome, and the substantive literature on political polarization and exposure to members of opposing parties. we develop a sensitivity procedure for mediation analysis when the mediator is only observed for one arm of the experiment. our procedure allows researchers to assess the strength of correlations between the outcome and mediators by determining how much treatment would have to affect the mediator to observe a significant indirect effect. when applied to text data, domain knowledge is especially important to both guide selection of potential mediators and assess practical significance of the required treatment effect on the mediator. beyond the experiment analyzed here, this procedure is useful in marketing applications when one wants to assess the alignment of digital advertisements with surrounding web-page context in reference to control units shown no ads @xcite . observational studies sometimes collect data on treated and control units from different sources with different sets of covariates, as in observational analysis of the lallonde data @xcite . this procedure can also be used in a power analysis to guide sample size selection for future studies with similar treatments that will collect mediator data for all participants. zach wood-doughty, ilya shpitser, and mark dredze. 2018. challenges of using text classifiers for causal inference. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4586-4598, brussels, belgium. association for computational linguistics. michael yeomans, alejandro kantor, and dustin tingley. 2018. the politeness package: detecting politeness in natural language. the r journal, 10(2):489-502. shabnam h. a. zanjani, william d. diamond, and kwong chan. 2011. does ad-context congruity help surfers and information seekers remember ads in cluttered e-magazines? journal of advertising, 40(4):67-84.