B ayesian Topic Regression for Causal Inference (2021)

Abstract: Causal inference using observational text data is becoming increasingly popular in many research areas. This paper presents the Bayesian Topic Regression (BTR) model that uses both text and numerical information to model an outcome variable. It allows estimation of both discrete and continuous treatment effects. Furthermore, it allows for the inclusion of additional numerical confounding factors next to text data. To this end, we combine a supervised Bayesian topic model with a Bayesian regression framework and perform supervised representation learning for the text features jointly with the regression parameter training, respecting the Frisch-Waugh-Lovell theorem. Our paper makes two main contributions. First, we provide a regression framework that allows causal inference in settings when both text and numerical confounders are of relevance. We show with synthetic and semi-synthetic datasets that our joint approach recovers ground truth with lower bias than any benchmark model, when text and numerical features are correlated. Second, experiments on two real-world datasets demonstrate that a joint and supervised learning strategy also yields superior prediction results compared to strategies that estimate regression weights for text and non-text features separately, being even competitive with more complex deep neural networks.

## introduction causal inference using observational text data is increasingly popular across many research areas @xcite . it expands the range of research questions that can be explored when using text data across various fields, such as in the social and data sciences; adding to an extensive literature of text analysis methods and applications @xcite ; @xcite . where randomized controlled trials are not possible, observational data might often be the only source of information and statistical methods need to be deployed to adjust for confounding biases. text data can either serve as a proxy for otherwise unobserved confounding variables, be a confounding factor in itself, or even represent the treatment or outcome variable of interest. the framework: we consider the causal inference settings where we allow for the treatment variable to be binary, categorical or continuous. in our setting, text might be either a confounding factor or a proxy for a latent confounding variable. we also allow for additional non-text confounders (covariates). to the best of our knowledge, we are the first to provide such statistical inference framework. considering both text and numerical data jointly can not only improve prediction performance, but can be crucial for conducting unbiased statistical inference. when treatment and confounders are correlated with each other and with the outcome, the frisch-waugh-lovell theorem @xcite , described in section 2.2, implies that all regression weights must be estimated jointly, otherwise estimates will be biased. text features themselves are 'estimated data'. if they stem from supervised learning, which estimated the text features with respect to the outcome variable separately from the numerical features, then the resulting estimated (causal) effects will be biased. our contributions: with this paper, we introduce a bayesian topic regression (btr) framework that combines a bayesian topic model with a bayesian regression approach. this allows us to perform supervised representation learning for text features jointly with the estimation of regression parameters that include both treatment and additional numerical covariates. in particular, information about dependencies between outcome, treatment and controls does not only inform the regression part, but directly feeds into the topic modelling process. our approach aims towards estimating 'causally sufficient' text representations in the spirit of @xcite . we show on both synthetic and semi-synthetic datasets that our btr model recovers the ground truth more accurately than a wide range of benchmark models. finally, we demonstrate on two real-world customer review datasets -yelp and booking.com -that a joint supervised learning strategy, using both text and non-text features, also improves prediction accuracy of the target variable compared to a 'two-step' estimation approach with the same models. this does not come at a cost of higher perplexity scores on the document modelling task. we also show that relatively simple supervised topic models with a linear regression layer that follow such joint approach can even compete with much more complex, non-linear deep neural networks that do not follow the joint estimation approach. in the spirit of @xcite , we assume that our models can learn a supervised text representation z @xmath0, y i , t i , c i ), which in our case, together with c i blocks all 'backdoor path' between y i and t i , so that we can measure the causal effect intuitively, to obtain such z i and consequently an unbiased treatment effect, one should estimate the text features in a supervised fashion taking into account dependencies between w i , y i , t i , and c i . ## conclusions in this paper, we introduced btr, a bayesian topic regression framework that incorporates both numerical and text data for modelling a response variable, jointly estimating all model parameters. motivated by the fwl theorem, this approach is designed to avoid potential bias in the regression weights, and can provide a sound regression framework for statistical and causal inference when one needs to control for both numerical and text based confounders in observational data. we demonstrate that our model recovers the ground truth with lower bias than any other benchmark model on synthetic and semi-synthetic datasets. experiments on real-world data show that a joint and supervised learning strategy also yields superior prediction performance compared to 'two-stage' strategies, even competing with deep neural networks. margaret e roberts, brandon m stewart, and edoardo m airoldi. 2016. a model of text for experimentation in the social sciences. journal of the american statistical association, 111(515):988-1003. margaret e roberts, brandon m stewart, and richard a nielsen. 2020. adjusting for confounding with text matching. american journal of political science, 64(4):887-903. takahiro shinozaki and mari ostendorf. 2007. crossvalidation em training for robust parameter estimation. in ieee international conference on acoustics, speech and signal processing-icassp, volume 4, pages iv-437. alan stuart, steven arnold, j keith ord, anthony o'hagan, and jonathan forster. 1994. kendall's advanced theory of statistics. wiley. chenhao tan, lillian lee, and bo pang. 2014. the effect of wording on message propagation: topic-and author-controlled natural experiments on twitter. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 175-185, baltimore, maryland. association for computational linguistics. victor veitch, dhanya sridhar, and david blei. 2020. adapting text embeddings for causal inference. in conference on uncertainty in artificial intelligence, pages 919-928. pmlr. xinyi wang and yi yang. 2020. neural topic model with attention for supervised learning. in international conference on artificial intelligence and statistics, pages 1147-1156. pmlr. yining wang and jun zhu. 2014. spectral methods for supervised topic models. in advances in neural information processing systems, pages 1511-1519. zach wood-doughty, ilya shpitser, and mark dredze. 2018. challenges of using text classifiers for causal inference. in proceedings of the conference on empirical methods in natural language processing. conference on empirical methods in natural language processing, volume 2018, page 4586. nih public access. jun zhu, amr ahmed, and eric p xing. 2012. medlda: maximum margin supervised topic models. journal of machine learning research, 13(aug):2237-2278.