C ausal NLP Tutorial: An Introduction to Causality for Natural Language Processing (2022)

Abstract: Causal inference is becoming an increasingly important topic in deep learning, with the potential to help with critical deep learning problems such as model robustness, interpretability, and fairness. In addition, causality is naturally widely used in various disciplines of science, to discover causal relationships among variables and estimate causal effects of interest. In this tutorial, we introduce the fundamentals of causal discovery and causal effect estimation to the natural language processing (NLP) audience, provide an overview of causal perspectives to NLP problems, and aim to inspire novel approaches to NLP further. This tutorial is inclusive to a variety of audiences and is expected to facilitate the community’s developments in formulating and addressing new, important NLP problems in light of emerging causal principles and methodologies.

## introduction establishing causal relationships is a fundamental goal of scientific research @xcite . it naturally boils down to questions of causality when we need to quantify the effectiveness of a vaccine, the persuasive power of a public health ad, or the impact of a lockdown policy: how would the treatment (e.g., vaccine) affect the outcome (e.g., infection rates) compared to a counterfactual world with no treatment? once formally identified, the direction and strength of causal relationships play a key role in the formulation of clinical treatments, public policy, and other long-standing prescriptive strategies. with such broad applications, a growing body of literature focuses on the interplay between nlp and causal inference @xcite @xcite @xcite . despite the interdisciplinary interest in causal inference with text, research in this space seems to remain scattered across domains without clear definitions, notations, benchmark datasets, and an understanding of the state of the art and challenges that remain. for example, it is unclear how deficiencies in nlp methods (such as their inaccuracy with low-resource languages and their tendency to propagate biases in the data) affect downstream causal estimates. in addition, hyperparameter selection and modeling assumptions in nlp are motivated by accuracy and tractability considerations; how these choices affect downstream causal estimates is underexplored. this tutorial aims to address three questions: (1) what is causality? (2) how can the causal formulation help improve nlp models? (3) how can causality help nlp and computational social science to discover causal phenomena in our society? specifically, we will first introduce the fundamentals of causality for the nlp audience, then review research using the causal formulation to help nlp models (in terms of robustness, fairness, and interpretability), and finally introduce how causality can help nlp and computational social science to discover causal relations behind social phenomena. ## tutorial overview this introductory tutorial aims to introduce causality to the nlp research community. while causality plays a major role in scientific research, it has only now started to disseminate into the nlp community. this is why this tutorial will first focus on providing a generalized introduction to causality and its importance and relevance to the nlp community. we will then dive into the intersection of causality and nlp, and divide it into two distinct areas: using causal formalisms to make nlp methods more interpretable, robust and fair, and discovering causal relations in social phenomena that involve text variables. accordingly, we divide the content of the tutorial into the following three parts: specifically, we will answer the following questions: how can we define causality? is causality an indispensable notion in science and machine learning? why do we care about causality? how can we infer the causal effect of one variable on another? how can one learn causality from purely observational data? how can we recover latent causal variables and their relations? with these motivations, we will then extend the causal formulation to nlp. here, we will identify and highlight existing limitations in nlp methods, and will propose three application areas where causal ideas might help: interpretability @xcite , robustness (e.g., @xcite and fairness (e.g., @xcite . for each potential application area, we will highlight the relevance of causal thinking in solving important open problems in nlp @xcite @xcite . ## tutorial outline for the three-hour tutorial, we will use 2.5 hours to cover three main topics: introduction of causality, how causality can help improve nlp models, and how causality can be applied to nlp and computational social science. and finally, we will use the remaining 30 minutes for an interactive exercise and q&a. an outline of the tutorial content is as follows: introduction to causality (60-min lecture + 5min break) • motivations: what is causality? why is causality helpful for nlp? • main content: basics of causal effect estimation, causal discovery, and causal representation learning. • example work: pearl (2000); feder et al. (2021b); xie et al. (2020); yao et al. (2022). 2. causality to help improve nlp models (60min lecture + 5-min break) • motivations: if the goal is to help improve nlp models, how can causality help? what are some use case examples? • main content: inspirations from causality to help a variety of nlp topics: model robustness, domain adaptation, debiasing models, interpretability, and fairness. • example work: schölkopf et al. (2021); feder et al. (2021c); veitch et al. (2021); jin et al. (2021c); stolfo et al. (2022); hupkes et al. (2022). 3. applications of causality for nlp and computational social science (20-min lecture) • motivations: if the goal is to identify causal phenomena in our society, how can we learn causality on variables that involve text? • main content: use of scms and potential outcomes for nlp social applications such as explaining social media behavior, political phenomena, effective education, and gender bias in the research community. we will also cover cases where causal discovery and inference can help verify linguistic theories. • example work: veitch et al. (2020); jin et al. (2021b); ding et al. (2022). 4. interactive exercise (20 min) ## tutorial breadth as for the contents of this tutorial, we will mainly cover beginner-friendly introductory materials of nlp, from the studies of established causality researchers out of the nlp domain, such as judea pearl, donald rubin, bernhard schölkopf, clark glymour, and peter spirtes. apart from the work from these causality researchers, when it comes to the more specific connection of nlp and causality, we will cover the research work of various researchers: dyanya sridhar (mila), victor veitch (university of chicago), zach wood-doughty (northwestern university), justin grimmer (stanford), brandon m. stewart (princeton), margaret e. roberts (ucsd), reid pryzant (microsoft), and many others. amir feder (he/him) is a postdoc at columbia university, working with prof david blei. amir develops methods that integrate causality into natural language processing to generate more robust and interpretable models. he is also interested in investigating and developing linguistically informed algorithms for predicting and understanding human behavior. amir is currently also a visiting researcher (part time) at google research's medical brain team, where he works on methods that leverage causal methodology for medical language models. he is a co-organizer of the first workshop on causal inference and nlp (ci+nlp) at emnlp 2021 @xcite . kun zhang (he/him) is an associate professor at carnegie mellon university and mbzuai. his research interests lie in causal discovery and causality-based learning. he develops methods for automated causal discovery from various kinds of data, investigates learning problems including transfer learning and deep learning from a causal view, and studies philosophical foundations of causation and machine learning. he co-authored a best student paper for the conference on uncertainty in artificial intelligence (uai) and a best finalist paper for the conference on computer vision and pattern recognition (cvpr), and received the best benchmark award of the 2nd causality challenge. he has taken essential roles at many events of causal inference, including the general and program co-chair of the 1st conference on causal learning and reasoning (clear 2022), program co-chair of the uai 2022, co-organizer of the 9th causal inference workshop at uai 2021, co-organizer of neurips 2020 workshop on causal discovery and causality-inspired machine learn-ing, 2020, co-editor of a number of journal special issues on causality, and many others. ## diversity efforts our organizing committee includes both junior and senior instructors, as well as diverse genders, racial/ethnic backgrounds, and affiliations across america, europe and asia, which will help make people from various backgrounds feel more welcome to our workshop. the topic of our workshop is causal inference, which can serve as a helpful tool for many nlp tasks, and the methods can scale up to various languages and domains. in addition, we advertise the tutorial to diversity-oriented venues (e.g., widening nlp, queerinai, blackinai, wiml). ## target audience & prerequisites there is no required audience background. preferred knowledge includes the basics of statistics (e.g., understanding of probability distribution of single variables, joint probability distributions, and conditional probability distributions), and the basics of nlp (e.g., understanding of sentence embeddings, and the setup of simple nlp tasks such as classification). ## recommended reading list we compiled a recommended reading list of causality and nlp papers at @xcite . 1 among the papers, the top three recommended readings are @xcite , @xcite and @xcite . ## other information tutorial type: introductory. tutorial materials: we will make available on our github @xcite all tutorial presentation materials, including slides, captioned video recordings, codes, and the recommended paper list. ## ethical considerations the theme of the tutorial focuses on introducing the method of causal inference to nlp. the introduction materials will stay on the technical side. there will not be direct links to applications that will raise ethical concerns. additionally, since one of the instructor's research background is nlp for social