Mitigating the Inconsistency Between Word Saliency and Model Confidence with Pathological Contrastive Training (2022)

Abstract: Neural networks are widely used in various NLP tasks for their remarkable performance. However, the complexity makes them difficult to interpret, i.e., they are not guaranteed right for the right reason. Besides the complexity, we reveal that the model pathology - the inconsistency between word saliency and model confidence, further hurts the interpretability. We show that the pathological inconsistency is caused by the representation collapse issue, which means that the representation of the sentences with tokens in different saliency reduced is somehow collapsed, and thus the important words cannot be distinguished from unimportant words in terms of model confidence changing. In this paper, to mitigate the pathology and obtain more interpretable models, we propose Pathological Contrastive Training (PCT) framework, which adopts contrastive learning and saliency-based samples augmentation to calibrate the sentences representation. Combined with qualitative analysis, we also conduct extensive quantitative experiments and measure the interpretability with eight reasonable metrics. Experiments show that our method can mitigate the model pathology and generate more interpretable models while keeping the model performance. Ablation study also shows the effectiveness.

## introduction neural networks have achieved remarkable success in various nlp tasks, while the extremely high complexity of such models makes them difficult to interpret. complex models may learn significantly different attributions with similar accuracy during training as datasets are often full of ambiguities @xcite . if a model is deployed without ensuring that it is right for the right reason, it may completely fail to make reliable predictions on new data, which is very dangerous. for example, some figure 1 : word saliency and model confidence on case sentence. normal model can not distinguish well between the influence of important and unimportant words, and the confidence on positive class always focuses on a high region. our method mitigates the pathology. models will counter-intuitively consider prepositions to have extremely high saliency in rumor detection tasks. interpretable models can ensure that the attribution of model prediction is consistent with human intuition, allowing the model to be trusted in critical applications. in addition to the complexity, the pathology also makes models more difficult to interpret @xcite . neural networks are more linear than expected, leading models to overfit the negative loglikelihood loss to output low-entropy distributions over classes, and thus models will be overconfident on examples outside the training data distribution @xcite . this consequently leads to the models giving counter-intuitive high confidence predictions on meaningless rubbish examples, and the word saliency will drastically change with even unimportant words reduced. the model pathology indicates that words with low saliency actually have a more significant impact on prediction than expected. we further extend the pathology to a more general definition: the saliency and confidence is inconsistent. specifically, we show that the important words (with high saliency) are actually not so important to the model prediction and the unimportant words (with low saliency) are actually not so unimportant in normal models, as the representation of the text with tokens in different saliency reduced are somehow collapsed. the model prediction confidence will only slightly change when words are reduced, and the important words cannot be distinguished from unimportant words in terms of model confidence changing. traditional methods usually train models with additional supervision, i.e., annotation on rationales, to force models better distinguish the influence between important words and unimportant words. however, human annotation is costly and often unavailable. in this paper, to mitigate the pathology, i.e., the inconsistency between saliency and confidence, and train a more interpretable model while avoiding the dependence on extra labeled data, we propose a model-agnostic training method called pathological contrastive training (pct). inspired by contrastive learning, we encourage the original text to be closer to the text with unimportant words reduced while keeping away from the text with important words reduced. our method can generate more interpretable models while keeping model performance. an example of model pathology and the effectiveness of our method is shown in figure 1 . the major contributions of this paper are summarized as follows: ## related work training interpretable model. a common method to obtain interpretable model is to let the model learn from the human-labeled rationales @xcite @xcite . however, the labeled data is costly. other works try to assign interpretable properties to model through unsupervised regularization. feng et al. ( 2018 ) train model with an objective containing an entropy regularization term to mitigate the model pathology that the confidence remained almost constant and sometimes increased when unimportant words are reduced. evaluating rationales. lack of unified metrics for the interpretability of nlp models, many previous works measure the quality of the prediction rationales directly by human study, e.g., by visualizing the attribution through a saliency heatmap @xcite and asking humans to give the quality of rationales provided by the model @xcite . to reduce the human work in the rationales evaluating, deyoung et al. (2020) propose automatic metrics including comprehensiveness and sufficiency. feng et al. (2018) utilized reduced length to measure the pathology of the model. contrastive learning. contrastive learning is first applied to unsupervised computer vision tasks @xcite @xcite , while the discrete nature of the text makes methods designed for continuous images fail to construct textual contrastive pairs. previous works propose various textual data augmentation methods for construing textual contrasts, e.g., by generating overlapping or contained spans @xcite , by randomly performing word deletion, span deletion, reordering, and synonym substitution @xcite , by using back-translation @xcite , and by performing adversarial attacks, shuffling, cutoff and dropout on the embedding @xcite . ## pathological contrastive training according to the above analysis, the representation collapse characteristic of neural networks causes the influence of high saliency words and low saliency words on prediction to be indistinguishable. to mitigate this issue, we propose pct that utilizes saliency-based samples augmentation for contrasting learning. the key idea of our method can be summarized as: the original normal text are encouraged to be closer to the derived text with unimportant words reduced while keeping away from the derived text with important words reduced. as shown in figure 4 , the framework comprises the following three major components: data augmentation module. we limit the contrast scope to within a mini-batch rather than the entire training set, as the latter is extremely computationally expensive. the data augmentation module will generate positive and negative samples in a self-supervised manner before the new mini-batch is sent to the model. suppose there are k normal examples in a mini-batch, for each sample in the batch, we first use gradient-based attribution method to obtain the saliency of the normal input sentence s(x i ), and define m words with the highest saliency and lowest saliency as important words and unimportant words, respectively. we then cumulatively reduce the important words in a descending order of saliency value to generate a text set containing text with multiply important words are reduced { ximp i } m @xmath0. parallelly, we generate the text set { xump i } m @xmath1. adding the original text x i to { xump i } m @xmath2, we have the positive set for each text set, there are at most m sentences if not considering x i . thus, a sentence has at most m positive pairs and (2k -1)m negative pairs. target model f. model is utilized as an encoder that extracts representations for both the original text and the augmented text. our method does not impose restrictions on the type of model. specifically, for bert, we use the representation of @xcite token at the last hidden layer as sentence representations. for other models (e.g., cnn, lstm), we use the average pooling of the token embedding at the layer before the last dense layer as sentence representation. model-agnostic contrastive loss objective. this loss objective controls the representation distances of the samples in a mini-batch. to mitigate the representation collapse issue, we maximize the agreement of representation from the same set and keep distance of representation from different sets, the loss function for a sample x p involving in set x i (same for both x - i and x + i ) is defined as: where r(•) is the sentence representation, 1 [@xmath3, 0} is an indicator function for excluding the sample itself, sim(r i , r j ) = r i r j / r i r j , i.e., the cosine similarity, τ is a temperature parameter. the final loss l con for contrastive learning is computed by averaging the loss on every sample in each text set in a mini-batch. this loss is a generalization of the nt-xent (the normalized temperature-scaled cross-entropy loss) @xcite , as more than one positive pairs for each sample are considered . besides the contrastive part, we also incorporate supervised information in the final loss objective l pct for optimizing on both model performance and interpretability: where l sup is the supervised loss objective (e.g., cross-entropy loss), α is a parameter balancing the two objectives. the joint training objective ensures that the accuracy of model is not hurt while addressing the representation collapse issue. ## evaluation to verify the effectiveness of our method, we evaluate pct with two other baselines on three popular datasets involving four different models. ## conclusion in this paper, we propose pct, a contrastive learning framework for addressing the representation collapse issue and mitigating the inconsistency between word saliency and model confidence for natural language models. we construct the contrastive pairs with saliency-based word reduction. our model-agnostic method can generate more interpretable models without extra data and changes to the model. extensive quantitative and qualitative evaluations demonstrate that our method can mitigate the model pathology while keeping model performance. we hope the analysis and the method proposed in our paper will provide a new perspective on model interpretability.