A Fine-grained Interpretability Evaluation Benchmark for Neural NLP (2022)

Abstract: While there is increasing concern about the interpretability of neural models, the evaluation of interpretability remains an open problem, due to the lack of proper evaluation datasets and metrics. In this paper, we present a novel benchmark to evaluate the interpretability of both neural models and saliency methods. This benchmark covers three representative NLP tasks: sentiment analysis, textual similarity and reading comprehension, each provided with both English and Chinese annotated data. In order to precisely evaluate the interpretability, we provide token-level rationales that are carefully annotated to be sufficient, compact and comprehensive. We also design a new metric, i.e., the consistency between the rationales before and after perturbations, to uniformly evaluate the interpretability on different types of tasks. Based on this benchmark, we conduct experiments on three typical models with three saliency methods, and unveil their strengths and weakness in terms of interpretability. We will release this benchmark ( https://www.luge.ai/#/luge/task/taskDetail?taskId=15 ) and hope it can facilitate the research in building trustworthy systems.

## introduction in the last decade, deep learning (dl) has been rapidly developed and has greatly improved various artificial intelligence tasks in terms of accuracy @xcite @xcite . however, as dl models are black-box systems, their inner decision processes are opaque to users. this lack of transparency makes them untrustworthy and hard to be applied in decisionmaking applications in fields such as health, commerce and law @xcite . consequently, there is a growing interest in explaining the predictions of dl models @xcite @xcite @xcite . accordingly, many in order to accurately evaluate model interpretability 2 with human-annotated rationales 3 (i.e., evidence that supports the model prediction), many researchers successively propose the properties that a rationale should satisfy, e.g., sufficiency, compact-ness and comprehensiveness (see section 3.3 for their specific definitions) @xcite @xcite . however, the existing datasets are designed for different research aims with different metrics, and their rationales do not satisfy all properties needed, as shown in meanwhile, many studies focus on designing guidelines and metrics for interpretability evaluation, where plausibility and faithfulness are proposed to measure interpretability from different perspectives @xcite @xcite @xcite . plausibility measures how well the rationales provided by models align with human-annotated rationales. with different annotation granularities, token-level and span-level f1-scores are proposed to measure plausibility @xcite . faithfulness measures to what extent the provided rationales influence the corresponding predictions. some studies @xcite propose to compare the model's prediction on the full input to its prediction on input masked according to the rationale and its complement (i.e., non-rationale). however, it is difficult to apply this evaluation method to nonclassification tasks, such as machine reading comprehension. furthermore, the model prediction on the non-rationale has gone beyond the standard output scope, e.g., the prediction label on the nonrationale should be neither positive nor negative in the sentiment classification task. thus the metric provided by this method can not generally and may not precisely evaluate the interpretability. in order to address the above problems, we release a new interpretability evaluation benchmark which provides fine-grained rationales for three tasks and a new evaluation metric for interpretability. our contributions include: ## related work as our work provides a new interpretability evaluation benchmark with human-annotated rationales, in this section, we mainly introduce saliency methods for the rationale extraction, interpretability evaluation datasets and metrics. ## evaluation data construction as illustrated in figure 1 , the construction of our datasets mainly consists of three steps: 1) data collection for each task; 2) perturbed data construction; 3) iterative rationale annotation and checking. we first introduce the annotation process, including the annotation criteria for perturbations and rationales. then we describe our data statistics. in addition, we show other annotation details in appendix a. ## metrics following existing studies @xcite @xcite , we evaluate interpretability from the perspectives of plausibility and faithfulness. plausibility measures how well the rationales provided by the model agree with human-annotated ones. and faithfulness measures the degree to which the provided rationales influence the corresponding predictions. different from existing work, we adopt token-f1 score for plausibility and propose a new metric map for faithfulness. token f1-score is defined in equation 1, which is computed by overlapped rationale tokens. since an instance may contain multiple golden rationale sets, for the sake of fairness, we take the set that has the largest f1-score with the predicted rationale as the ground truth for the current prediction. map (mean average precision) measures the consistency of rationales under perturbations and is used to evaluate faithfulness. according to the original/perturbed input pair, map aims to calculate the consistency of two token lists sorted by token importance score, as defined in equation 2. the high map indicates the high consistency. meanwhile, we also report results of metrics proposed in deyoung et al. (2020), i.e., iou f1score for plausibility, and the joint of sufficiency and comprehensiveness for faithfulness. iou f1-score is proposed on span-level rationales, which is the size of token overlap in two sets divided by the size of their union, as shown by s i in equation 3. a rationale is considered as a match if its s i is equal to or greater than 0.5, as illustrated by the greater function. greater(si, 0.5) the joint of sufficiency (score-suf) and comprehensiveness (score-com) is shown in equation 4 . a lower sufficiency score implies the rationale is more sufficient and a higher comprehensiveness score means the rationale is more influential in the prediction. a faithful rationale should have a low sufficiency score and a high comprehensiveness ## limitation discussion we provide a new interpretability evaluation benchmark which contains three tasks with both english and chinese annotated data. there are three limitations in our work. finally, we hope more evaluation metrics and analyses are proposed based on our benchmark. and we hope our benchmark can facilitate the research progress of interpertability. ## conclusion we propose a new fine-grained interpretability evaluation benchmark, containing token-level rationales, a new evaluation metric and corresponding perturbed examples for three typical nlp tasks, i.e., sentiment analysis, textual similarity and machine reading comprehension. the rationales in this benchmark meet primary properties that a rationale should satisfy, i.e., sufficiency, compactness and comprehensiveness. the experimental results on three models and three saliency methods prove that our benchmark can be used to evaluate interpretability of both models and saliency methods. we will release this benchmark and hope it can facilitate progress on several directions, such as better interpretability evaluation metrics and causal analysis of nlp models.