A Survey of the State of Explainable AI for Natural Language Processing (2020)

Abstract: Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an overview of the current state of Explainable AI (XAI), considered within the domain of Natural Language Processing (NLP). We discuss the main categorization of explanations, as well as the various ways explanations can be arrived at and visualized. We detail the operations and explainability techniques currently available for generating explanations for NLP model predictions, to serve as a resource for model developers in the community. Finally, we point out the current gaps and encourage directions for future work in this important research area.

## introduction traditionally, natural language processing (nlp) systems have been mostly based on techniques that are inherently explainable. examples of such approaches, often referred to as white box techniques, include rules, decision trees, hidden markov models, logistic regressions, and others. recent years, though, have brought the advent and popularity of black box techniques, such as deep learning models and the use of language embeddings as features. while these methods in many cases substantially advance model quality, they come at the expense of models becoming less interpretable. this obfuscation of the process by which a model arrives at its results can be problematic, as it may erode trust in the many ai systems humans interact with daily (e.g., chatbots, recommendation systems, information retrieval algorithms, and many others). in the broader ai community, this growing understanding of the importance of explainability has created an emerging field called explainable ai (xai). however, just as tasks in different fields are more amenable to particular approaches, explainability must also be considered within the context of each discipline. we therefore focus this survey on xai works in the domain of nlp, as represented in the main nlp conferences in the last seven years. this is, to the best of our knowledge, the first xai survey focusing on the nlp domain. as will become clear in this survey, explainability is in itself a term that requires an explanation. while explainability may generally serve many purposes (see, e.g., @xcite , our focus is on explainability from the perspective of an end user whose goal is to understand how a model arrives at its result, also referred to as the outcome explanation problem @xcite . in this regard, explanations can help users of nlp-based ai systems build trust in these systems' predictions. additionally, understanding the model's operation may also allow users to provide useful feedback, which in turn can help developers improve model quality @xcite . explanations of model predictions have previously been categorized in a fairly simple way that differentiates between (1) whether the explanation is for each prediction individually or the model's prediction process as a whole, and (2) determining whether generating the explanation requires post-processing or not (see section 3). however, although rarely studied, there are many additional characterizations of explanations, the most important being the techniques used to either generate or visualize explanations. in this survey, we analyze the nlp literature with respect to both these dimensions and identify the most commonly used explainability and visualization techniques, in addition to operations used to generate explanations (sections 4.1-section 4.3). we briefly describe each technique and point to representative papers adopting it. finally, we discuss the common evaluation techniques used to measure the quality of explanations (section 5), and conclude with a discussion of gaps and challenges in developing success-ful explainability approaches in the nlp domain (section 6). related surveys: earlier surveys on xai include @xcite and @xcite . while adadi and berrada provide a comprehensive review of basic terminology and fundamental concepts relevant to xai in general, our goal is to survey more recent works in nlp in an effort to understand how these achieve xai and how well they achieve it. guidotti et al. adopt a four dimensional classification scheme to rate various approaches. crucially, they differentiate between the "explanator" and the black-box model it explains. this makes most sense when a surrogate model is used to explain a black-box model. as we shall subsequently see, such a distinction applies less well to the majority of nlp works published in the past few years where the same neural network (nn) can be used not only to make predictions but also to derive explanations. in a series of tutorials, @xcite discuss fairness and trust in machine learning (ml) that are clearly related to xai but not the focus of this survey. finally, we adapt some nomenclature from @xcite which presents a software toolkit that can help users lend explainability to their models and ml pipelines. our goal for this survey is to: (1) provide the reader with a better understanding of the state of xai in nlp, (2) point developers interested in building explainable nlp models to currently available techniques, and (3) bring to the attention of the research community the gaps that exist; mainly a lack of formal definitions and evaluation for explainability. we have also built an interactive website providing interested readers with all relevant aspects for every paper covered in this survey. 1 ## methodology we identified relevant papers (see appendix a) and classified them based on the aspects defined in sections 3 and 4. to ensure a consistent classification, each paper was individually analyzed by at least two reviewers, consulting additional reviewers in the case of disagreement. for simplicity of presentation, we label each paper with its main applicable category for each aspect, though some papers may span multiple categories (usually with varying degrees of emphasis.) all relevant aspects for every paper covered in this survey can be found at the aforementioned website; to enable readers of this survey to discover interesting explainability techniques and ideas, even if they have not been fully developed in the respective publications. ## categorization of explanations explanations are often categorized along two main aspects @xcite . the first distinguishes whether the explanation is for an individual prediction (local) or the model's prediction process as a whole (global). the second differentiates between the explanation emerging directly from the prediction process (selfexplaining) versus requiring post-processing (posthoc). we next describe both of these aspects in detail, and provide a summary of the four categories they induce in ## aspects of explanations while the previous categorization serves as a convenient high-level classification of explanations, it does not cover other important characteristics. we now introduce two additional aspects of explanations: (1) techniques for deriving the explanation and (2) presentation to the end user. we discuss the most commonly used explainability techniques, along with basic operations that enable explainability, as well as the visualization techniques commonly used to present the output of associated explainability techniques. we identify the most common combinations of explainability techniques, operations, and visualization techniques for each of the four high-level categories of explanations presented above, and summarize them, together with representative papers, in although explainability techniques and visualizations are often intermixed, there are fundamental differences between them that motivated us to treat them separately. concretely, explanation derivation -typically done by ai scientists and engineers -focuses on mathematically motivated justifications of models' output, leveraging various explainability techniques to produce "raw explanations" (such as attention scores). on the other hand, explanation presentation -ideally done by ux engineersfocuses on how these "raw explanations" are best presented to the end users using suitable visualization techniques (such as saliency heatmaps). ## explanation quality following the goals of xai, a model's quality should be evaluated not only by its accuracy and performance, but also by how well it provides explanations for its predictions. in this section we discuss the state of the field in terms of defining and measuring explanation quality. ## insights and future directions this survey showcases recent advances of xai research in nlp, as evidenced by publications in major nlp conferences in the last 7 years. we have discussed the main categorization of explanations (local vs global, self-explaining vs post-hoc) as well as the various ways explanations can be arrived at and visualized, together with the common techniques used. we have also detailed operations and explainability techniques currently available for generating explanations of model predictions, in the hopes of serving as a resource for developers interested in building explainable nlp models. we hope this survey encourages the research community to work in bridging the current gaps in the field of xai in nlp. the first research direction is a need for clearer terminology and understanding of what constitutes explainability and how it connects to the target audience. for example, is a model that displays an induced program that, when executed, yields a prediction, and yet conceals the process of inducing the program, explainable in general? or is it explainable for some target users but not for others? the second is an expansion of the evaluation processes and metrics, especially for human evaluation. the field of xai is aimed at adding explainability as a desired feature of models, in addition to the model's predictive quality, and other features such as runtime performance, complexity or memory usage. in general, trade-offs exist between desired characteristics of models, such as more complex models achieving better predictive power at the expense of slower runtime. in xai, some works have claimed that explainability may come at the price of losing predictive quality @xcite , while other have claimed the opposite @xcite . studying such possible trade-offs is an important research area for xai, but one that cannot advance until standardized metrics are developed for evaluating the quality of explanations. the third research direction is a call to more critically address the issue of fidelity (or causality), and to ask hard questions about whether a claimed explanation is faithfully explaining the model's prediction. finally, it is interesting to note that we found only four papers that fall into the global explanations category. this might seem surprising given that white box models, which have been fundamental in nlp, are explainable in the global sense. we believe this stems from the fact that because white box models are clearly explainable, the focus of the explicit xai field is in explaining black box models, which comprise mostly local explanations. white box models, like rule based models and decision trees, while still in use, are less frequently framed as explainable or interpretable, and are hence not the main thrust of where the field is going. we think that this may be an oversight of the field since white box models can be a great test bed for studying techniques for evaluating explanations.