Locally Aggregated Feature Attribution on Natural Language Model Understanding (2022)

Abstract: With the growing popularity of deep-learning models, model understanding becomes more important. Much effort has been devoted to demystify deep neural networks for better explainability. Some feature attribution methods have shown promising results in computer vision, especially the gradient-based methods where effectively smoothing the gradients with reference data is the key to a robust and faithful result. However, direct application of these gradient-based methods to NLP tasks is not trivial due to the fact that the input consists of discrete tokens and the “reference” tokens are not explicitly defined. In this work, we propose Locally Aggregated Feature Attribution (LAFA), a novel gradient-based feature attribution method for NLP models. Instead of relying on obscure reference tokens, it smooths gradients by aggregating similar reference texts derived from language model embeddings. For evaluation purpose, we also design experiments on different NLP tasks including Entity Recognition and Sentiment Analysis on public datasets and key words detection on constructed Amazon catalogue dataset. The superior performance of the proposed method is demonstrated through experiments.

## introduction with the growing popularity of deep-learning models, model understanding becomes more and more critical in many folds. in one aspect, model understanding helps us understand what the model is doing by identifying crucial features among unstructured raw data. for example, @xcite the model explainability technique to discover motifs in regulatory dna elements from distinct molecular signatures in the field of genomics. in another aspect, model understanding helps people audit or debug the deep models. an interesting example is that ribeiro et al. @xcite found that their image classification model sometimes misclassifies a husky as a wolf. the model explainability tool reveals that their model relies on the snow in the background rather than the appearance when distinguishing the two animals. more importantly, model understanding helps gain trust when making important decisions based on the model. in the nlp domain, deep language models are quickly evolving and show superior performance in various benchmark tasks. however, even experts struggle to understand the mechanism of complex language models. much effort has been devoted to demystifying the "black box" of deep models. a natural idea is through feature attribution, explaining the model by attributing the prediction to each input feature according to how much it affects the model output, of which two main directions emerge. one is model agnostic approaches including shapley regression values @xcite and @xcite . we can apply these methods regardless of the model structure, however, they could suffer from computational inefficiency in the scenario of high dimensional input space and complex deep models when making inferences across all possible permutations or with small perturbations in the local neighborhood. another direction is model-specific approaches which look into the internal model mechanism to understand specific models. gradient-based feature attribution models are often adopted to explain neural networks since gradients can be easily accessed through back-propagation, which gives a great computational advantage over model-agnostic methods. since the gradient map itself is often noisy and challenging to interpret, most gradient-based methods aim to stabilize the feature attribution score by smoothing the gradients or learning from the reference data @xcite @xcite . however, direct application of these gradient-based methods to nlp problems is not trivial, due to the fact that the input consists of discrete tokens and the "reference" tokens are not explicitly defined. in this paper, we propose locally aggregated feature attribution (lafa), a novel gradientbased approach that leverages sentence-level embedding as a smoothing space for the gradients, motivated by the observation that the feature attribution is often shared by similar text inputs. for example, key features in product descriptions on an online marketplace are often shared by similar products. we implement a neighbor-searching method to ensure the quality of neighboring sentences. furthermore, to evaluate feature attribution methods in nlp, we consider two situations. for datasets with golden labels of feature score, we use the area under curve (auc) or pearson correlation as the performance metric. as for datasets without golden labels, we conduct a similar evaluation task following prior works @xcite by masking tokens with high importance scores and find the change in the predicted log-odds. in summary, our contributions are threefold: first, we build a novel context-level smooth gradient approach for feature attribution in nlp. the key ingredients of our method are constructing an appropriate aggregation function over the smoothing space. second, to the best of our knowledge, this is the first proposal to conduct numerical studies on multiple nlp tasks, including entity recognition and sentiment analysis, for feature attribution. third, our method achieves superior performance compared with the state-of-the-art feature attribution methods. the paper is organized as follows. section 2 elaborates the current challenges of feature attribution in nlp and recaps the preliminaries about gradient-based feature attribution approaches. the proposed feature attribution method is described in section 3, followed by a review of other existing approaches in section 4. the evaluation tasks and the application results on nlp are presented in section 5. ## feature attribution in nlp challenge direct application of gradient-based methods to nlp problems is not trivia. there are three main challenges. first, nlp models consist of non-differentiable discrete input tokens, hence the gradient hook can only reach out to the embedding space and gradient-based feature attribution methods are not directly applicable to word tokens. second, the reference data in nlp are difficult to define. it is studied by @xcite that using the gradient as the feature attribution may suffer from the problems of model saturation or thresholding. model saturation means the perturbation of some elements in the input cannot change the output, and the thresholding problem indicates discontinuous gradients can produce misleading importance scores. such problems can be addressed by comparing the difference between the gradient of input and reference data. the guiding principle to select reference data is to ask ourselves that "what am i interested in measuring differences against?" for example, in the tasks of binary classification on dna sequence inputs, the reference data are chosen as the expected frequencies of dna sequence or randomly shuffling the original sequence. however, in nlp tasks, randomly shuffling texts as reference may not be grammatically sensible. lastly, we note that the evaluation of the language model is much more challenging than the explanations of the images. in the image application, the important features of an image obtained from feature attribution methods can be visually validated by checking the composition of objects. however, the detected important features in language may require more domain knowledge to validate. problem definition feature attribution task can be formally formulated as follows. a deep model f is provided to be explained, which is fine-tuned on dataset x . the input sentence is denoted as x 0 = (w 1 , w 2 , .., w t ) t where w i represents i-th word. the goal for feature attribution is to determine function m(•) by quantifying the importance score of each word m(x) = (m 1 , m 2 , .., m t ) t , where m i denotes the importance score for w i . simple gradient as feature attribution as illustrated in the first challenge above, in nlp models, directly taking derivative on each word is infeasible due to the non-differentiable embedding layer. we can resolve the challenge as follows. the fist layer of the nlp model usually maps input discrete tokens to embedding from a predefined dictionary. however, simply using the gradients of one token as feature attribution would lead to noisy results @xcite . the next section describes a novel feature attribution approach that smoothes the gradients by leveraging similar input texts. ## the proposed framework: lafa the proposed method contains three steps: (1) find the appropriate neighbors of the input text for gradient smoothing; (2) calculate the gradients of texts as well as neighbors; (3) aggregation of the gradients. the proposed framework is summarized in the upper panel of figure 1 . one motivating example is shown in the lower panel of figure 1 . in this motivating example, the input text is a description of computer. the key features of the computer should include "brand", "cpu type" and "ram size". the simple gradient method may miss certain feature, such as "ram size" in the example, while the gradients on similar texts can provide more contexts. the proposed method is constructed to aggregate the information from similar texts. ## related work in nlp, transformer-based models yield great successfulness and some works focus on explaining the attention mechanism. for example, serrano and smith, 2019 and jain and wallace, 2019 inspected a single attention layer and found out that attention weights only weakly and inconsistently correspond to feature importance; wiegreffe and pinter, 2019 argued that we cannot separate the attention layer and should view the entire model as a whole. in this section, we mainly review the gradient-based methods for feature attribution. feature attribution on single input simonyan et al @xcite computed the "saliency map" denoted as simple gradient from the derivative of the output with respect to the input in an image classification task. in the nlp application, "saliency map" is obtained as the derivative of the output with respect to the word embedding as in equation ( 2 ). however, "saliency map" can be visually noisy. several methods are proposed to improve the gradient method from different perspectives. gradient*input method (shrikumar et al., 2017) improves the visual sharpness of the "saliency map" by multiplying gradient with the input itself. in nlp, we can write it as: layerwise relevance propagation method @xcite is shown to be equivalent to the gradient*input method up to a scaling factor. smooth gradient method @xcite smoothes the feature attribution score by adding random noises to the input and taking average of the gradients from noisy inputs, formally: guided backpropagation method (springenberg et al., 2014) modifies the back-propagation to preserve negative gradients in the relu activation layer which also sharpens the "saliency map" visually. other methods, such as grad-cam or guided-cam @xcite , are applicable to specific architecture of neural networks in the field of computer vision. since language models like bert do not contain specific architecture utilized in guided backpropagation or grad-cam method, we ignore the mathematical formulation here. feature attribution on input with reference data integrated gradient method computes the feature score by integrating the gradients from single pre-determined reference input to the target input @xcite . in computer vision problems, black image is usually considered as the reference data, and integrating gradients from the black image to the input image represents the feature attribution of the input image. in nlp problems, we can define the i-th element of feature attribution as: deeplift @xcite assigns the feature score by comparing the difference of contribution between input and some reference inputs via gradient. as discussed in @xcite , deeplift can be considered as an approximation of shapley value estimation. specifically, as in the application of shap 2 , the feature attribution of shap-deep as a variant of deeplift is defined as: ## experiments in this section, we compare the proposed method to the state-of-the-art feature attribution methods under different use cases. ## conclusion this paper presents a novel locally aggregated feature attribution method in nlp, which efficiently captures the important features by leveraging similar input texts in the embedding space. we focused on feature attribution of single input based on a fine-tuned model instead of training a language model, henceforth the computation time is of less concern. one limitation of the lafa model is that it requires informative neighbor sentences that carry similar information. otherwise, aggregating information from other sentences could be misleading. experiments in our datasets show that our method is effective, but the improvements gained from the lafa varies among different datasets based on the information that neighbor carries. there are several future directions worthy of study. firstly, labeling feature attribution result in the nlp requires massive human labor, and few datasets are available with golden feature attribution label. developing new evaluation techniques to further measure model performance is interesting to investigate. also, readable feature attribution results could help human beings to develop more business applications. for example, developing a key-value pair like processor-i5 as important feature can provide a more plausible feature attribution result to customers.