The Impact of Integration Step on Integrated Gradients (2024)

Abstract: Integrated Gradients (IG) serve as a potent tool for explaining the internal structure of a language model. The calculation of IG requires numerical integration, wherein the number of steps serves as a critical hyperparameter. The step count can drastically alter the results, inducing considerable errors in interpretability. To scrutinize the effect of step variation on IG, we measured the difference between theoretical and observed IG totals for each step amount.Our findings indicate that the ideal number of steps to maintain minimal error varies from instance to instance. Consequently, we advocate for customizing the step count for each instance. Our study is the first to quantitatively analyze the variation of IG values with the number of steps.

## introduction researchers have focused on explainable ai (xai), which aims to provide insights into model behavior and predictions. one popular xai method is feature attribution @xcite , generally referring to techniques that clarify why each feature was influential in determining the model's prediction. integrated gradients (ig) @xcite is one of the well-known feature attribution approaches and has been widely used in image @xcite and language processing @xcite due to the many desirable explanation axioms and ease of gradient computation @xcite . in recent years, ig has been applied to analyze language models @xcite , and efforts have been made to enhance its performance specifically for language processing tasks @xcite @xcite . in ig, a property known as completeness @xcite posits that the sum of the contributions of each feature equals the difference between the output and the sum. this fundamental property offers a way of interpreting the value of each contribution as its influence on the output as follows eq. 2. it also quantifies each contribution value relative to the output, enabling comparisons between contributions. however, completeness is often violated because the numerical integration required to compute ig introduces errors. completenes violation compromises the interpretability of the contributions and the results obtained from comparisons among the contributions. we have also identified instances where errors adversely affect the interpretation of the contributions (see figure 2 ). hence, to ensure the reliability and accuracy of ig, it is vital to determine the number of steps to minimize such errors properly. given these factors, it is necessary to ensure an adequate number of steps to reduce errors to guarantee ig's reliability. however, as shown in therefore, in this study, we measured the error between the theoretical and measured values of the total ig sum at each number of steps to quantitatively analyze the change in the contribution value depending on the number of ig steps in the lm. the results show that the ideal number of steps that minimize the error varies from instance to instance, bert, lstm @xcite even for the same dataset model. this result argues that the number of steps should be set on an instance-by-instance basis. our study is the first to quantitatively analyze the variation of ig values with the number of steps. ## integrated gradients the method of generating post-hoc explanations for each model output is known as feature attribution @xcite . this method allows for the assessment of the contribution of input features to the prediction results of machine learning models. it provides insights into how much a model's predictions rely on specific features. integrated gradients (ig) @xcite is a type of feature attribution method. ig is popular over other feature attribution methods due to its simplicity, relatively low computational cost, and adherence to mathematically rigorous axioms @xcite . in the field of nlp, ig has proven valuable, with researchers developing enhanced methods tailored to language-specific tasks @xcite @xcite and utilizing it for lm analysis @xcite . the ig formula for an input x along the i-th dimension is as follows: the ig calculation involves sampling along a linear path from the baseline vector to the input vector and computing and integrating the gradient for each sample. here, the sampling points are determined by the numerical integration method and the number of steps. the number of steps is a vital hyperparameter that determines the integration accuracy. however, many steps require much backpropagation, resulting in high computational costs. ## experimental results since the riemann sum results were consistently better than the gauss-legendre integration results, we report the riemann sum results in the following experiments. see appendix a.4 for details. here, we show the case of the bert(large)-agnews model, but results for other models are given in appendix a.7. ## discussion our analysis reveals that the number of integration steps required for each instance is different. therefore, we recommend that the number of steps be increased sequentially until it falls below a specific error, thereby reducing the error. for example, we can ensure that the ig satisfies completeness by initially setting the number of steps to 2 n and running with increasing n until the error converges to a constant. optimizing the number of steps on an instanceby-instance basis would also make ig more efficient since our analysis has shown that the number of steps required is negligible for many instances (figure 1 ). however, we keep this part as a future study since constructing a methodology to find better solutions. ## conclusion the researcher subjectively determines the number of steps in ig for each dataset and model, which raises questions about the reliability of ig. in this study, we quantitatively analyzed the error for each number of steps. as a result, half of the instances in which the appropriate number of steps is around 100 steps, but on the other hand, instances in which the error does not converge even at 1000 steps or more were confirmed. these results indicate that the current mainstream method of fixing the number of steps for each model or data set runs the risk of producing instances with broken contributions and undermining the reliability of ig's analysis results. to solve this, we also proposed to change the integration step for each instance. our study is the first to quantitatively analyze the variation of ig values with the number of steps and to identify problems with existing integration methods.