Black-box language model explanation by context length probing (2023)

Abstract: The increasingly widespread adoption of large language models has highlighted the need for improving their explainability. We present *context length probing*, a novel explanation technique for causal language models, based on tracking the predictions of a model as a function of the length of available context, and allowing to assign *differential importance scores* to different contexts. The technique is model-agnostic and does not rely on access to model internals beyond computing token-level probabilities. We apply context length probing to large pre-trained language models and offer some initial analyses and insights, including the potential for studying long-range dependencies. The [source code]( https://github.com/cifkao/context-probing/ ) and an [interactive demo]( https://cifkao.github.io/context-probing/ ) of the method are available.

## introduction large language models (lms), typically based on the transformer architecture @xcite , have recently seen increasingly widespread adoption, yet understanding their behaviour remains a difficult challenge and an active research topic. notably, as the length of the context that can be accessed by lms has grown, a question that has attracted some attention is how this influences their predictions. some recent studies in this line of research suggest that even "long-range" lms focus heavily on local context and largely fail to exploit distant ones @xcite @xcite . a more nuanced understanding of how contexts of different lengths influence lms' predictions may hence be valuable for further improving their performance, especially on tasks like long-form text generation where long-range dependencies are of critical importance. figure 1 : a screenshot of a demo foot_1 of the proposed method. after selecting a target token (here "birds"), the preceding tokens are highlighted according to their (normalized) differential importance scores (@xmath0, @xmath1, obtained using our method. the user can also explore the top predictions for contexts of different lengths (here the context "house, shouting about lunatics. [. . .] mortally afraid of"). in this work, we propose context length probing, a simple explanation technique for causal (autoregressive) language models, based on tracking the predictions of the model as a function of the number of tokens available as context. our proposal has the following advantages: furthemore, we propose a way to use this technique to assign what we call differential importance scores to contexts of different lengths. this can be seen as complementary to other techniques like attention or saliency map visualization. interestingly, contrary to those techniques, ours appears promising as a tool for studying long-range dependencies, since it can be expected to highlight important information not already covered by shorter contexts. ## related work a popular way to dissect transformers is by visualizing their attention weights (e.g. @xcite . however, it has been argued that this does not provide reliable explanations and can be misleading @xcite . a more recent line of work @xcite explores "mechanistic explanations", based on reverse-engineering the computations performed by transformers. these techniques are tied to concrete architectures, which are often "toy" versions of those used in real-world applications, e.g. attention-only transformers in elhage et al. other options include general-purpose methods like neuron/activation interpretation (e.g. @xcite @xcite , saliency maps (e.g. @xcite and influence functions @xcite . these require access to internal representations and/or the ability to backpropagate gradients, and have some caveats of their own @xcite . more closely related to our work are studies that perform ablation (e.g. by shuffling, truncation or masking) on different contexts to understand their influence on predictions (o'connor and andreas, 2021; @xcite @xcite . to our knowledge, all such existing works only test a few select contexts or greedily search for the most informative one; in contrast, we show that it is feasible to consider all context lengths in the range from 1 to a maximum c max , which permits us to obtain fine-grained insights on the example level, e.g. in the form of the proposed differential importance scores. moreover, many existing analyses (e.g. @xcite rely on specific training or finetuning, which is not the case with our proposal. ## results we apply the proposed technique to publicly available pre-trained large transformer language models, namely gpt-j (wang and komatsuzaki, 2021) and two gpt-2 (radford et al., 2019) variantssee ## limitations and future directions experiments. we acknowledge the limited scope of our experiments, including only 8 (closeddomain) documents, 3 models and a single language. this is largely due to the limited availability of suitable large lms and their high computational cost. still, we believe that our experiments are valuable as a case study that already clearly showcases some interesting features of our methodology. computational cost. while we have demonstrated an efficient strategy to obtain predictions for all tokens at all possible context lengths, it still requires running the model n times for a document of length n . for a k-fold reduction in computational cost, the technique may be modified to use a sliding window with stride @xmath3. see appendix a.1 for details. 4 16 64 256 1024 context length -6 -5 -4 -3 -2 -1 log 10 (norm. | |) 600 800 1000 -5 -4 gpt2 gpt2-xl gpt-j-6b figure 5: normalized ∆-score log-magnitude (mean and std. dev.) by context length and by model. only positions @xmath4. 0 0.005 sym part adp cconj punct adv det num adj intj pron aux sconj noun x verb propn gpt-j-6b 0 0.005 gpt2-xl 0 0.005 gpt2 figure 6: mean ∆-score by pos tag of the context token and by model. choice of metrics. the proposed methodology allows investigating how any given metric is impacted by context, yet our study is limited to nll loss and the proposed kl divergence metric (the latter for defining importance scores). these may not be optimal for every purpose, and other choices should be explored depending on the application. for example, to study sequences generated (sampled) from a lm, one might want to define importance scores using a metric that does depend on the generated token, e.g. its nll loss or its ranking among all candidates. (indeed, our web demo also supports ∆-scores defined using nll loss values.) ## conclusion and future directions we have presented context length probing, a novel causal lm explanation technique based on tracking the predictions of the lm as a function of context length, and enabling the assignment of differential importance scores (∆-scores). while it has some advantages over existing techniques, it answers different questions, and should thus be thought of as complementary rather than a substitute. a particularly interesting feature of our ∆-scores is their apparent potential for discovering longrange dependencies (lrds) (as they are expected to highlight information not already covered by shorter contexts, unlike e.g. attention maps). remarkably, our analysis suggests a power-lawlike inverse relationship between context length and importance score, seemingly questioning the importance of lrds in language modeling. while lrds clearly appear crucial for applications such as longform text generation, their importance may not be strongly reflected by lm performance metrics like cross entropy or perplexity. we thus believe that there is an opportunity for more specialized benchmarks of lrd modeling capabilities of different models, such as that of @xcite , for example. these should further elucidate questions like to what extent improvements in lm performance are due to better lrd modeling, how lrds are handled by various transformer variants (e.g. @xcite @xcite , or what their importance is for different tasks. when the lm is run on a segment of the document, the effective context length for each target token is equal to its offset from the beginning of the segment, e.g. the context for predicting " d" is " the" (@xmath5, the context for "urs" is " the d" (@xmath6, etc.