Text as Causal Mediators: Research Design for Causal Estimates of Differential Treatment of Social Groups via Language Aspects (2021)

Abstract: Using observed language to understand interpersonal interactions is important in high-stakes decision making. We propose a causal research design for observational (non-experimental) data to estimate the natural direct and indirect effects of social group signals (e.g. race or gender) on speakers’ responses with separate aspects of language as causal mediators. We illustrate the promises and challenges of this framework via a theoretical case study of the effect of an advocate’s gender on interruptions from justices during U.S. Supreme Court oral arguments. We also discuss challenges conceptualizing and operationalizing causal variables such as gender and language that comprise of many components, and we articulate technical open challenges such as temporal dependence between language mediators in conversational settings.

## introduction interactions between individuals are key components of social structure @xcite . while we rarely have access to individuals' internal thoughts during these interactions, we often can observe the language they use. using observed language to better understand interpersonal interactions is important in high-stakes decision making-for instance, judges' decisions within the united states legal system @xcite or police interaction with citizens during traffic stops @xcite . in these settings, analysts may be interested in understanding the behavior of decision makers as individuals or at the subgroup or aggregate level. important decision makers sometimes treat some social groups (e.g. women, racial minorities, or ideological communities) differently than others @xcite ). yet, quantitative analyses of this problem often do not account for all possible mechanisms that could induce this differential treatment. for instance, one might ask, during u.s. supreme t: speaker 1 social group y: speaker 2 response m 1 : speaker 1 text aspect 1 t: advocate gender y: justice interrupts advocate m 1 : (delivery) advocate speech disfluencies m 2 : (content) topics discussed a. general framework b. theoretical case study: u.s. supreme court oral arguments m 2 : speaker 1 text aspect 2 proposed general framework for differential treatment of social groups via language aspects and b. instantiation of the framework for a theoretical case study of u.s. supreme court oral arguments. in both diagrams, t is the treatment variable, y is the outcome variable, and m are mediator variables. this is a simplified schema; see fig. 2 for an expanded diagram. court oral arguments, is a justice interrupting female advocates more because of their gender, because of the content of the advocates' legal arguments, or because of the advocates' language delivery (fig. 1b )? accounting for these language mechanisms could help separate and estimate the remaining "gender bias" of justices. we reformulate the previous question as a general counterfactual query @xcite about two speakers: how would speaker 2 respond if the signal they received of speaker 1's social group flipped from a to b but speaker 1 still used language typical of social group a? here, our question is about the direct causal effect of treatment-speaker 1's signaled social group-on outcome-speaker 2's responsethat is not through the causal pathway of the mediator-an aspect of language (fig. 1a ). 1 the fundamental problem with this and any counterfactual question is that we cannot go back in time and observe an individual counterfactual while holding all other conditions the same @xcite . furthermore, in many high-stakes, real-world settings (e.g. the u.s. supreme court), we cannot run experiments to randomly assign treatment and approximate these counterfactuals. instead, in these settings, causal estimation must rely on observational (non-experimental) data. in this work, we focus on this observational setting and build from causal mediation methods @xcite @xcite to specify a research design of causal estimates of differential treatment of social groups via language aspects. other work has used causal mediation analysis to better understand components of natural language processing (nlp) models @xcite . however, this work is more closely aligned with studies that focus on causal estimation in which text is one or more causal variables (e.g., @xcite @xcite @xcite . our focus is on the research design, and we therefore intentionally do not present empirical results. instead, we discuss the potential promises and challenges of this causal research design with both general examples and concrete examples from a theoretical case study of u.s. supreme court arguments. this aligns with @xcite who argues "design trumps analysis" in observational studies and emphasizes the importance of conceptualizing a study before any outcome data is analyzed. overall, we make the following contributions: ## theoretical case study: gender bias in u.s. supreme court interruptions to motivate our causal research design and illustrate challenges that arise with it, we focus on a specific theoretical case study-the effect of advocate gender on justice interruptions via advocates' language during united states supreme court oral arguments (fig. 1b ). the substantive motivation for this theoretical case study is built from previous work examining the role of interruption and gender on the court. patton and smith (2017) found female lawyers are interrupted earlier in oral arguments, allowed to speak for less time, and subjected to longer speeches by justices; @xcite found female justices are interrupted at disproportionate rates by their male colleagues; and @xcite found justices are more likely to vote for the female advocate's side when the female advocate uses emotional language. counterfactual questions. we present a novel causal approach to understanding gender bias in supreme court oral arguments that corresponds to the following counterfactual questions: however, advocate utterances matched on this hedging indicator could have a number of latent mediators and confounders. in to get frustrated and interrupt. the cases are from different areas of the law, foot_0 and scalia may interrupt more during cases that are in areas he has more personal interest. the advocate utterance in ex. b is longer (more tokens) and longer utterances may be more likely to be interrupted. in ex. b, scalia interrupts adams just prior to the target utterance which possibly indicates a more "heated" portion of the oral arguments during which interruptions occur more on average. with these confounding and additional mediator challenges, a simple causal matching approach (e.g. @xcite ; @xcite ) is unlikely to work and we advocate for the causal estimation strategy presented in §3.4. we move from this case study to a formalization of our causal research design in §3. ## causal mediation formalization, identification, and estimation many causal questions involve mediatorsvariables on a causal path between treatment and outcome. for example, what is the effect of gender 4 (treatment) on salary (outcome) with and without considering merit (a mediator)? if one intervenes on treatment, then one would activate both the "direct path" from gender to salary and the "indirect path" from gender through merit to salary. thus, a major focus of causal mediation is specifying conditions under which one can separate estimates of the direct effect from the indirect effect-the former being the effect of treatment on outcome not through mediators and the later the effect through mediators. we use this causal mediation approach to formally define our framework. for each unit of analysis (see §4.1), i, let t i represent the treatment variable-the social group, e.g. gender of an advocate-and y i represent the outcome variablethe second speaker's response, e.g. a judge's interruption or non-interruption of an advocate. for each defined mediator j, let m j i represent the mediating variable-an aspect of language, e.g. an advocate's speech disfluencies or the topics of an utterance. let x i represent any other confounders between any combination of the other variables. we use the potential outcomes framework @xcite to define the natual direct and indirect effects. 5 let m i (t) represent the (counterfactual) potential value the mediator would take if t @xmath0. these correspond to the two counterfactual questions from §2 if t @xmath1. ## conceptualization and operationalization of causal variables for any causal research design-and particularly those in the social sciences-there are often challenges conceptualizing the theoretical causal variables of interest. even after these theoretical concepts are made concrete, there are often multiple ways to operationalize these concepts. we discuss conceptual and operational issues for our both our general research design and our theoretical case study. in particular, we recommend researchers formalize variables such as gender and language as constitutive variables made of multiple components (fig. 2 ) as per hu and kohler-hausmann (2020), or sen and wasow (2016)'s "bundle of sticks." ## challenges and threats to validity we discuss additional challenges and threats to validity for our research design that should be addressed before implementing the design and claiming the estimates from the design are causal. temporal dependence of utterances. so far, we have assumed the "units of analysis" of text are independent ( §4.1). however, previous utterances in a conversation often influence the target utterances. for our case study, if judge a interrupted advocate b often in @xmath2, interruption at t is more likely (the two speakers are possibly in a "heated" part of the conversation) and advocate b's speech disfluencies at t are also more likely (the advocate could be mentally fatigued). potential avenues forward include changing the unit of analysis to the entire conversational thread between the two target speakers or building extensions to the multiple mediator literature, i.e. @xcite ; @xcite @xcite . dependence between multiple language mediators. our framework assumes one can computationally separate aspects of language. foot_6 however, some sociolinguists argue aspects of language such as "style" cannot be separated from "content" because style originates in the content of people's lives and different ways of speaking signal socially meaningful differences in content @xcite . if our mediator independence assumption (eqn. 5) is violated, then we would have to turn to alternate estimation strategies to deal with this dependence. dependence between social group perception and linguistic perception. separating the direct and indirect causal paths in our framework relies on there being a decision-maker's latent perception of social group variable on the direct path between treatment and outcome and that this variable is independent from a decision-maker's latent perception of language variable on the indirect path from treatment through mediators to outcome. however, "indexical inversion" considers "how language ideologies associated with social categories produce the perception of linguistic signs" @xcite . suppose judge a perceives advocate b as female, then judge a might perceive advocate b's language as more feminine even if it is linguistically identical to language used by male advocates. furthermore, latent gender perception and latent language perception might interact in affecting the outcome through mechanisms such as rewarding "conforming to gender norms"-an advocate who is perceived as a man might get penalized for using feminine language whereas an advocate perceived as a woman might get rewarded, e.g. @xcite . ## conclusion in this work, we specify a causal research design for differential treatment of social groups with language as a mediator. we believe this research design is important for studying the direct and indirect causal effects in high-stakes decision making such as gender bias in the united states supreme court. separating the indirect effect of treatment on outcome through interpretable language aspects allows us to estimate counterfactual queries about differential treatment when speakers use and do not use the same language. despite open theoretical and technical challenges, we remain optimistic that researchers can build upon this framework and continue to improve our understanding of decision makers' differential treatment of social groups.