Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods (2023)

Abstract: Feature attribution scores are used for explaining the prediction of a text classifier to users by highlighting a k number of tokens. In this work, we propose a way to determine the number of optimal k tokens that should be displayed from sequential properties of the attribution scores. Our approach is dynamic across sentences, method-agnostic, and deals with sentence length bias. We compare agreement between multiple methods and humans on an NLI task, using fixed k and dynamic k. We find that perturbation-based methods and Vanilla Gradient exhibit highest agreement on most method–method and method–human agreement metrics with a static k. Their advantage over other methods disappears with dynamic ks which mainly improve Integrated Gradient and GradientXInput. To our knowledge, this is the first evidence that sequential properties of attribution scores are informative for consolidating attribution signals for human interpretation.

## introduction feature attribution scores are a glimpse behind the scenes of neural models, or at least that is the promise. various interpretability methods have been developed that can generate attribution scores to interpret the degree to which a language model's features (tokens) contributed to the predicted label. however, attribution values from different methods can vary considerably even on the same instance @xcite . contradictory interpretations cast doubts on their usefulness and reliability in a practical setting. when comparing attribution methods, the focus commonly lies on assessing agreement with human explanations (often referred to as plausibility @xcite ) and on agreement between methods @xcite . however, the evaluation procedures have not been standardised yet and varying influential factors such as the exact task, data, and model have led to contradictory conclusions. for example, @xcite find that perturbationbased methods better agree with human preferences than gradient-based methods, while @xcite report the reverse tendency. a factor that may also influence results but has been largely understudied is the impact of the chosen number of k most salient tokens that are taken into consideration. previous analyses either use a fixed value of k or determine the most salient tokens based on a fixed threshold for the attribution values. a clear drawback of using a fixed number of k is that it can result in excluding tokens with scores close to the top-k and including tokens with low scores, disregarding significant score gaps with truly important tokens. @xcite set k to a fixed value of six and examine the effect of visualizing the most salient features on decisionmaking accuracy, time, and agreement for a fraud analysis task. @xcite set k to the low fixed values of 1 and 2 as their analysis focuses on the identification of specific shortcut cues in a closed experimental setup. @xcite determine k using a fixed threshold and identify tokens as salient if their attribution @xmath0.1. when visualizing attributions to users, they use constant values of k (5 and 10). absolute attribution scores tend to decrease for sentences with a larger number of tokens which is not captured by their thresholdbased approach. @xcite determine k as a 1/4 fraction of the average sentence length in the data and set it to 11. as token-level agreement generally increases the closer k gets to the number of tokens in a sentence (see appendix c for a visualisation of the sentence length bias), the evaluation of attribution methods is clearly top-k dependent. @xcite indicate that intrinsic model evaluation is generally sensitive to different values of k but do not measure its effect on agreement. both @xcite and @xcite find consistent disagreement between attribution methods, but do not account for the influence of k. 1 contributions in this paper, we systematically explore the role of k on the observed methodmethod agreement and method-human agreement for extracting explanations for a natural language inference task. for this purpose, we develop the new metric agreement@k. we propose to determine k dynamically for each method and each instance based on the sequential properties of the attribution profile. our approach is inspired by methods for event detection and detects attribution peaks in a method-agnostic fashion that circumvents the sentence-length bias by allowing different values of k for each instance. we find that: we take a novel perspective on human attribution evaluation by interpreting it as a ranking task. ## experimental setup we fine-tune a model on a natural language inference task and analyze the agreement between feature attribution methods. ## agreement at fixed k we compare attribution methods with each other and with human labels and analyze the role of k. method-method agreement we identify three groups of mean agreement@k across pairs of attribution methods in figure 2a . the agreement between partition shap-lime clearly stands out (yellow line), in particular for small k. all comparisons involving integrated gradient or gradient-xinput end up in the group with the least agreement (purple lines) and the remaining pairs obtain medium-level agreement (green lines). agreement increases for bigger k for pairs of methods with low to medium agreement. our results contrast previous work which identified higher pairwise agreement between gradient-based methods compared to perturbation-based methods @xcite . ## dynamic top-k estimation we have seen that the value of k has a strong influence on the agreement between methods. perturbation-based methods are more in line with human annotations for smaller settings of k while the attribution profiles of gradient-based methods require relatively larger settings of k to obtain a similar reflection of human preferences. we propose to determine the number of k salient tokens dynamically based on the attribution profiles of the methods. inspired by event detection in time series (taylor (2000), @xcite , e.g.), we consider attribution profiles as sequences of token-level scores that indicate the local presence or absence of a peak. each peak is a point in the sequence (i.e., the sentence) to which the model attributed a higher salience compared to neighboring points. we apply peak detection based on local maxima in the attribution profile to estimate a k that is dynamic across method-instance combinations. the attribution profile of each individual method thus serves as the indicator of its peaks. a local maximum is defined as a point x i in a sequence such that it is greater than both its immediate left neighbor, x i-1 , and its immediate right neighbor, x i+1 . we additionally enforce the constraint that x i needs to be higher than the mean attribution of the sequence, corresponding to above-average model behavior. with our dynamic k approach, we favor relative differences in attribution values over absolute thresholds for identifying the top-k tokens. figure 3 shows that when determining k dynamically, the agreement of integrated gradient and gradientxinput with the other methods increases. we find that dynamic k indeed varies across instances and across methods. for example, attributions by partition shap leads to fewer local maxima and therefore to lower values for dynamic k (4.5±1.7 on average) compared to integratedgradient (7.3±2.6). we note that these ranges are close to human preferences for k on the nli task (4±3). ## further insights we briefly discuss complementary insights to the main results, as well as two examples of dynamic k on instances from the dataset. average human preference human annotations of token-level explanations were available for this task and we compared the effect of dynamic k to the average number of annotations. however, these types of annotations are expensive, which often leads to selecting a fixed k. normally, a valid approach is choosing a fixed k that is close to the average number of annotated tokens per sentence, as the resulting ranges of dynamic k for our methods reflected. in absence of annotations, this average number is unknown. dynamic k highlights the implications of choosing a fixed k that deviates from the average. we observe a clear distinction between lower and higher than average for gradientxinput and for integrated gradient. this further suggests that, in the absence of human annotations, dynamic k may provide an estimate of the average human preference for k. peaks as signals we analyse two examples to better understand the dynamic k approach. figures 1 and 4 show the application of dynamic k versus fixed k set to 3, 5, and 7 for two different methods. in figure 1 , dynamic k highlights the same top three tokens that have been selected by fixed k, namely reading, paper and argue. tokens that are only selected by @xmath2. figure 4 illustrates differences in the attribution patterns captured by fixed k and dynamic k. while fixed k selects any tokens in descending order of attribution score, dynamic k captures signals. it highlights two peaks that were not captured by fixed k (due to their lower absolute values) because they stand out relative to the surrounding tokens. signals can be described as tokens that represent a salient part or phrase in the sentence beyond the token level (e.g. a man in a suit; a lady). by focusing on signals, dynamic k skips tokens with a relatively high score but that can be attributed to the same signal of a neighboring salient token. ## discussion and conclusion attribution methods disagree on the salience of tokens. our analyses show that the observed level of agreement is sensitive to the number of k tokens taken into consideration. we propose a dynamic k that can be directly applied to any attribution profile. in contrast to fixed k, it takes local relative differences of the attribution values into account. our analyses with dynamic k indicate that different attribution methods capture varying degrees of attribution scope. determining dynamic k purely on attribution profiles yields a level of plausibility that is comparable to determining the average human preference for k and is therefore a viable alternative in the absence of task-specific human data. furthermore, as dynamic k is estimated for each instance separately, it can account for sentence length bias. our peak detection method is an intuitive approach for determining salient tokens based solely on attribution profiles. it focuses on isolated key tokens which might not adequately capture human tendencies of chunking words into phrases. in future work, we plan to analyze how our findings generalize to other task and model conditions and want to explore alternative methods to dynamically determine k by combining the attribution with linguistic information towards better span-level visualisations (see figure 5 ). this line of research needs to be closely coupled with cognitive analyses of human preferences.