Explanation-Based Human Debugging of NLP Models: A Survey (2021)

Abstract: Debugging a machine learning model is hard since the bug usually involves the training data and the learning process. This becomes even harder for an opaque deep learning model if we have no clue about how the model actually works. In this survey, we review papers that exploit explanations to enable humans to give feedback and debug NLP models. We call this problem explanation-based human debugging (EBHD). In particular, we categorize and discuss existing work along three dimensions of EBHD (the bug context, the workflow, and the experimental setting), compile findings on how EBHD components affect the feedback providers, and highlight open problems that could be future research directions.

## introduction explainable ai focuses on generating explanations for ai models as well as for their predictions. it is gaining more and more attention these days since explanations are necessary in many applications, especially in high-stake domains such as healthcare, law, transportation, and finance @xcite . some researchers have explored various merits of explanations to humans, such as supporting human decision making @xcite , increasing human trust in ai @xcite , and even teaching humans to perform challenging tasks @xcite . on the other hand, explanations can benefit the ai systems as well, for example, when explanations are used to promote system acceptance @xcite , to verify the model reasoning @xcite , and to find potential causes of errors @xcite . in this paper, we review progress to date specifically on how explanations have been used in the literature to enable humans to fix bugs in nlp models. we refer to this research area as explanation-based human debugging (ebhd), as a general umbrella term encompassing explanatory debugging @xcite and humanin-the-loop debugging @xcite . we define ebhd as the process of fixing or mitigating bugs in a trained model using human feedback given in response to explanations for the model. ebhd is helpful when the training data at hand leads to suboptimal models (due, for instance, to biases or artifacts in the data), and hence human knowledge is needed to verify and improve the trained models. in fact, ebhd is related to three challenging and intertwined issues in nlp: explainability @xcite , interactive and human-in-the-loop learning @xcite , and knowledge integration @xcite . although there are overviews for each of these topics (as cited above), our paper is the first to draw connections among the three towards the final application of model debugging in nlp. whereas most people agree on the meaning of the term bug in software engineering, various meanings have been ascribed to this term in machine learning (ml) research. for example, @xcite considered bugs as implementation errors, similar to software bugs, while @xcite defined a bug as a particularly damaging or inexplicable test error. in this paper, we follow the definition of (model) bugs from @xcite as contamination in the learning and/or prediction pipeline that makes the model produce incorrect predictions or learn error-causing associations. examples of bugs include spurious correlation, labeling errors, and undesirable behavior in out-of-distribution testing. the term debugging is also interpreted differently by different researchers. some consider debugging as a process of identifying or uncovering causes of model errors @xcite , while others stress that debugging must not only reveal the causes of problems but also fix or mitigate them @xcite 2015; yousefzadeh and o'leary, 2019). in this paper, we adopt the latter interpretation. ## categorization of existing work ## research on human factors though the major goal of ebhd is to improve models, we cannot disregard the effect on feedback providers of the debugging workflow. in this section, we compile findings concerning how explanations and feedback could affect the humans, discussed along five dimensions: model understanding, willingness, trust, frustration, and expectation. although some of the findings were not derived in nlp settings, we believe that they are generalizable and worth discussing in the context of ebhd. ## open problems this section lists potential research directions and open problems for ebhd of nlp models. ## conclusion we presented a general framework of explanationbased human debugging (ebhd) of nlp models and analyzed existing work in relation to the components of this framework to illustrate the state-of-the-art in the field. furthermore, we summarized findings on human factors with respect to ebhd, suggested design practices accordingly, and identified open problems for future studies. as ebhd is still an ongoing research topic, we hope that our survey will be helpful for guiding interested researchers and for examining future ebhd papers.