Causal Matching with Text Embeddings: A Case Study in Estimating the Causal Effects of Peer Review Policies (2023)

Abstract: A promising approach to estimate the causal effects of peer review policies is to analyze data from publication venues that shift policies from single-blind to double-blind from one year to the next. However, in these settings the content of the manuscript is a confounding variable—each year has a different distribution of scientific content which may naturally affect the distribution of reviewer scores. To address this textual confounding, we extend variable ratio nearest neighbor matching to incorporate text embeddings. We compare this matching method to a widely-used causal method of stratified propensity score matching and a baseline of randomly selected matches. For our case study of the ICLR conference shifting from single- to double-blind review from 2017 to 2018, we find human judges prefer manuscript matches from our method in 70% of cases. While the unadjusted estimate of the average causal effect of reviewers’ scores is -0.25, our method shifts the estimate to -0.17, a slightly smaller difference between the outcomes of single- and double-blind policies. We hope this case study enables exploration of additional text-based causal estimation methods and domains in the future.

## introduction for over two hundred years, peer review has been the key means of evaluating scholarly work and establishing scientific legitimacy @xcite . although many claim double-blind peer review reduces evaluation biases due to known author identities @xcite @xcite , others claim there is little statistical evidence for a preference over single-blind @xcite . in this work, we argue that studying the impact of peer review anonymization policies is inherently a causal question. if we intervene and assign a manuscript to double-blind review, what is resulting effect on the manuscript's review score compared to what the score would have been under single-blind review? the gold standard for these types of causal estimation questions is randomized controlled trials (rcts) which produce unbiased effect estimates @xcite . however, in the case of peer review, an rct is unethical because applying different review policies to different manuscripts could potentially harm the dissemination of scientific findings and researchers' careers. in the absence of an rct, one can use observational (non-randomized) data from publication venues before and after a policy change to estimate causal effects. however, a major obstacle to unbiased effect estimation for observational data is accounting for confounding variables that affect both treatment and outcome. in the case of peer review, we represent our domain assumptions via the causal diagram in figure 1 . the content of a manuscript affects peer review scores-popular scientific content might naturally have higher reviewer scoresand is correlated with review policy-the distribution of content in submitted manuscripts might be different in years with single-versus doubleblind review. 1 many methods have been proposed to statistically adjust for confounding variables in general @xcite @xcite . we follow the framework presented by @xcite who review settings for which text data is a proxy for confounding variables. from this prior work, we distill three important criteria for choosing text-based confounding adjustment method for case studies like ours: the method should (1) allow for empirical checks of causal overlap, (2) incorporate modern text representations, and (3) enable human validation of intermediate steps. causal overlap-a necessary condition for an estimate to be causal-requires any unit to have a non-zero probability of assignment to each treatment condition for all possible values of confounders @xcite . if causal overlap is not satisfied, one has to either abandon the project or shift the target causal estimand 2 . second, text data contains many layers of linguistic granularity and there are challenges to operationalizing a variable like "manuscript content". general-purpose language representations have greatly improved performance of predictive natural language processing (nlp) tasks, e.g. @xcite ; @xcite ; @xcite , and we hypothesize we could use them to help find semantically similar treated and untreated documents. finally, human validation is important because unlike prediction settings, causal settings have no ground-truth @xcite . in our case study, we do not have access to counterfactual outcomes for the same manuscript under both treatment settings. thus, we are essentially combining the "black box" of causal estimation with another "black box" of nlp techniques, so it is crucial that we are able to evaluate intermediate steps of the causal estimation pipeline in order to lend validity to our results. in summary, we contribute the following: ## conclusion and future work in this work, we implement vrm-e, a method for text-based causal confounding adjustment that satisfies our three criteria of empirically checking causal overlap, incorporating modern nlp embeddings, and human validation of intermediate steps. for our case study, we find domain-experts prefer vrm-e matches 70% of the time compared to random matches and stratified propensity score matches. while the sign of the causal effectnegative-of switching from single blind to double blind reviewing on average reviewer scores is consistent across all methods, vrm-e estimates a slightly less negative effect. future work could investigate the causal mechanisms behind the negative causal effect and explore heterogeneous treatment effects due to author identity. additional directions could examine different causal outcomes on the text of peer reviews such as discourse-level sentence labels @xcite or politeness @xcite . we hope this case study enables exploration of additional text-based causal estimation methods and domains in the future.