Can Input Attributions Explain Inductive Reasoning in In-Context Learning? (2025)

Abstract: Interpreting the internal process of neural models has long been a challenge. This challenge remains relevant in the era of large language models (LLMs) and in-context learning (ICL); for example, ICL poses a new issue of interpreting which example in the few-shot examples contributed to identifying/solving the task. To this end, in this paper, we design synthetic diagnostic tasks of inductive reasoning, inspired by the generalization tests in linguistics; here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates the task demonstrated. The question is whether conventional input attribution (IA) methods can track such a reasoning process, i.e., identify the influential example, in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.

## introduction in the natural language processing (nlp) field, input attribution (ia) methods, e.g., relying on gradient norm @xcite , have typically been employed to interpret inputoutput associations exploited by neural nlp models @xcite . recently, as large language models (llms) and mechanistic interpretability (mi) research @xcite ) has gained attention, the research focus is leaning toward understanding the circuits within llms by intervening in their internal representations and information flow. despite the enriched scope of research, such rapid progress has missed some intriguing questions bridging the ia and mi eras: in particular, * currently affiliated with preferred networks, inc. do conventional ia methods still empirically work in the modern nlp setting, specifically with llms and in-context learning (icl)? in this paper, we revisit ia methods in interpreting llm-based icl @xcite . specifically, we assess how well ia methods can track the most influential example in a few-shot examples. this question is worth investigating for several reasons. first, input attribution would still serve as a necessary and sufficient explanation in typical practical cases; some users might simply seek which part of the context is heavily referred to by an llm system rather than llms' internal processes identified by mi methods. second, the modern nlp setting, specifically icl, differs from the conventional settings where ia methods have been tested -identifying the input-output association within a specific instance (x k , y k ). in contrast, applying ia methods to the entire icl input (few-shot examples) entails tracking the model's learning process over these instances as well as the pure input-output association within a specific target instance. this involves the interpretation of which example among the demonstrations [(x 1 , y 1 ), • • • , (x k-1 , y k-1 )] contributed to identifying the targeted task/rule and then answering a target question x k . this is rather a type of instance-based interpretation of neural models @xcite @xcite , and it has been little explored such interpretation is feasible with ia methods. to test ia methods in icl, we introduce a test suite comprising controlled synthetic inductive reasoning tasks. otherwise, formally defining such informativeness and assessing ia methods is challenging, especially in a wild, natural setting; critical examples may not be unique @xcite , a gap might exist between faithfulness and plausibility perspectives @xcite , and a model can rather rely on prior knowledge without using any input examples @xcite . our task design, inspired by the poverty of the stimulus scenario @xcite @xcite @xcite or mixed signals generalization test @xcite , introduces one inherently unique aha example in input demonstrations. this aha example, when paired with any of the other examples, triggers the identification of the underlying reasoning rule. more specifically, most in-context examples are ambiguous in the sense that they are compatible with several rules (e.g., adding two tokens or doubling the number of tokens, in figure 1 ), and only one disambiguating (aha) example resolves the ambiguity and limits the correct rule to be unique (@xmath0. the question is whether such an informative example can be empirically caught by ia methods. in our experiments, ia methods are compared to other common interpretability approaches relying on, such as attention weights and post-hoc explanation generations. the experimental results demonstrate: • gradient norm, the simplest ia method, frequently outperforms other interpretability methods (e.g., integrated gradient, attention weights), suggesting that the advantage of more recently proposed ia methods does not always generalize in interpreting icl×llm. • we revealed that different interpretability methods exhibited different advantages in interpreting the icl process, against scaling in several aspects. for example, ia methods perform better in many-shot scenarios, whereas a post-hoc explanation generations work well on larger models. • our tested interpretability methods, including simple gradient norm, did not work stably across different tasks and models, posing their general limitations in interpreting icl with ia methods. some existing ia methods frequently failed even on a very simple associative recall task, and there is room to sophisticate previously developed interpretability tools to be suitable for llms. ## problem settings we evaluate the performance of each ia method in identifying the crucial in-context example e * necessary for defining the task. in real-world tasks, it is generally unclear which in-context example is the most influential in solving the task, and the task may be solved even without relying on any of the examples (e.g., solved by leveraging prior knowledge). therefore, these are not suitable as a benchmark to evaluate the interpretability method, and we design a synthetic and controlled tasks. our setting is the extension of @xcite ; we employ a set of ambiguous inductive learning scenarios inspired by the cognitively-motivated lm analyses @xcite ; inter alia). in these scenarios, a task f is mostly ambiguous in demonstrations e in the sense that several compatible rules exist to explain the transformations @xmath1. we extend this setting by adding only one disambiguating example e * ("aha example"), which determines the correct rule f * to be unique, and test whether each ia method can identify this special example as long as models correctly employ this clue e * to resolve the problem. for instance, most examples shown in figure 1 are ambiguous (with gray color) w.r.t. the two possible rules of (i) adding the same token twice or (ii) multiplying the number of tokens by two. this ambiguity is resolved by comparing the aha example e * (blue example in figure 1 ) with any one of the other ambiguous examples. as shown in ## experiments figure 2 shows the results in 10-shot settings, with both top-2 and top-1 metrics. additional analyses are presented in appendices a, f, and g. ## discussion this section discusses the potential reasons for the unexpected results presented in § 5, highlighting the challenging issues in interpreting icl. why did ie fail to achieve 100% attribution accuracy? our tasks can not be answered without disambiguating aha examples. thus, it is somewhat unintuitive to see the non-100% attribution accuracy of the ie method (again, the llms understood the task as they achieved 100% accuracy in the tasks) -what happened here? to obtain a hint to clarify ie's potential limitations, we analyze model behaviors when the disambiguating example is excluded. interestingly, llms adopted a specific generalization (rule) in each task when there was no disambiguating example ( figure 3 : ia score (attribution accuracy) for interpreting gemma-2-2b models across all six tasks. gradient-based methods are relatively robust to the number of few-shot examples, while there is a consistent, large drop in attribution accuracy in sa. note that both x-axis and y-axis are in log scale. @xcite . catching such generalization bias with ia methods represents an inherent challenge, highlighting their potential limitations in interpreting llms. why was i×g worse than gn? the advantage of the i×g method, compared to the gn, is the consideration of the norm of the input embedding @xcite . since a large vector tends to have a large dot product with another vector, the norm of the input token embedding (vector) is expected to affect the ia score of i×g. then, no improvement of the i×g over the gn suggests that, at least in our settings, the norm of the embeddings was not informative to estimate the input attribution. the norm of the embedding largely has decontextualized information about the word, such as frequency, and it may make sense that such information is not helpful to interpreting our controlled, synthetic icl tasks consisting of alphabet characters, numbers, or random words. why was ig worse than gn? ig is a path-based approach; the gradient is accumulated from a baseline vector (typically a zero vector) to the targeted input representation (in our case, the sequence of input embeddings representing few-shot examples). this approach is somewhat intuitive when considering an attribution for a particular word or sentence; for example, suppose one computes an attribution to the word "excellent" in an input, ig may trace the path from zero to the "excellent" vector, which will go through the goodness direction, involving the intermediate points corresponding to, e.g., "okay" "decent," "good," "excellent" @xcite . then, one critical question iswhat does this path mean in interpreting the prompt and task representations? different prompt representations will no longer correspond to the same ld am vo ta pt ar 0% 50% 100% ig (zero vectors) ig (pad token) task ia score (top-2) figure 4: ia score (attribution accuracy) obtained with different baselines for ig when interpreting the gemma-2-2b models across all six tasks. zero vectors consistently yield higher accuracy @xmath2. will the baseline choice for ig change the results? existing work on bert @xcite shows that the [mask] token is a better baseline for ig, whereas the [pad] token and zero vectors yield no substantial difference in performance @xcite . although we adopted zero vectors as the baseline in our main experiments, given that causal lms do not have mask tokens, we further discuss the impact of this baseline choice on our results. we conduct a case study on the gemma-2-2b models, using pad tokens (<@xmath3. the results are shown in figure 4 . the ia scores obtained with zero vectors consistently surpass those obtained @xmath4, suggesting that zero vectors are the better choice for interpreting causal llms. positional impact of the aha example llms are known to exhibit positional bias to prioritize information at the beginning or the end of the input @xcite . given this concern, we also examine the positional bias underlying our interpretability results. figure 5 shows attribution accuracy by the position of disambiguating, aha example. ia methods exhibited positional bias, though the patterns varied across tasks and methods. for instance, in the vo task, ie and gn performed better when the aha example appeared at the beginning, while in the ar task, ig performed better when it appeared at the end. see appendix g for other models' results as well as for 50 and 100-shot settings. instance-based explanation instance-based explanation seeks explanation in the training data rather than relying on immediate input during inference, as ia methods do @xcite @xcite . these two paradigms of instance-based and ia-based explanations have been studied somewhat separately since the information source to seek the explanation is clearly different. on the other hand, in icl, the training examples are now in the input during inference that can be analyzed by ia methods. in this sense, our investigation can be seen as a new exploration of instance-based explanation with the help of ia methods. ## conclusions we have pointed out and tackled the problem of interpreting the inductive reasoning process in icl as a missing but reasonable milestone to be explored in llm interpretability research. our revisit to the ia methods in interpreting the icl process has clarified their limitations from a new angle as well as provided fruitful insights and discussions on their practical usage in modern nlp. some methods frequently failed even on a very simple associative recall task, and there is room to sophisticate previously developed interpretability tools to be suitable for llms.