Interpretability of Language Models via Task Spaces (2024)

Abstract: The usual way to interpret language models (LMs) is to test their performance on different benchmarks and subsequently infer their internal processes.In this paper, we present an alternative approach, concentrating on the _quality_ of LM processing, with a focus on their language abilities.To this end, we construct ‘linguistic task spaces’ – representations of an LM’s language conceptualisation – that shed light on the connections LMs draw between language phenomena.Task spaces are based on the interactions of the learning signals from different linguistic phenomena, which we assess via a method we call ‘similarity probing’.To disentangle the learning signals of linguistic phenomena, we further introduce a method called ‘fine-tuning via gradient differentials’ (FTGD).We apply our methods to language models of three different scales and find that larger models generalise better to overarching general concepts for linguistic tasks, making better use of their shared structure. Further, the distributedness of linguistic processing increases with pre-training through increased parameter sharing between related linguistic tasks. The overall generalisation patterns are mostly stable throughout training and not marked by incisive stages, potentially explaining the lack of successful curriculum strategies for LMs.

## introduction recently, language models (lms) have reached a level of sophistication in language production where their output is often indistinguishable from human-generated language @xcite . however, the complexity inherent in language production means that effective models are also inherently complex, making them challenging to interpret. commonly, linguistic interpretability involves assessing an lm's ability through simple evaluation tasks like grammatical acceptability judgments of various language constructions (e.g. @xcite . while these methods inform us about a model's performance, they do not provide insights into the quality of the model's solutions. this is especially the case when error analysis is not possible due to high model performance. however, it is the quality of processing that is interesting from the viewpoint of the interpretability researcher, the cognitive scientist or linguist. here, we introduce a method to interpret the language processing of lms holistically. we show how linguistic knowledge in lms interconnects. we build upon the framework of @xcite that proposes to consider linguistic phenomena as 'tasks' an lm has to optimise and allows us to analyse the interactions of those tasks, similar to ideas from multi-task learning (mtl). for example, consider the following sentences: in both sentences, a downward-entailing environment (negation vs. conditional) allows for the negative polarity item (npi) anything to be used. understanding whether it is acceptable to produce an npi in either sentence can be considered a different task. lms can use different rules to solve these tasks: an lm might learn the co-occurrence statistics of certain trigger words (e.g. 'not' vs. 'if ') with npis. on the other hand, it might generalise to a more abstract linguistical conceptualisation and understand that both -negation and conditionals -create downward-entailing environments permitting npis. with either rule, the model resolves acceptability judgements correctly, while the quality of both solutions is decisively different. hence, assessing the generalisation of linguistic tasks reveals how lms conceptualise language. similar to 'task spaces' in mtl (more details in § 2.1), we can represent an lm's generalisation behaviour in a linguistic task space, a multidimensional space relating linguistic tasks according to their similarity. to construct linguistic task spaces, we introduce similarity probing, a method to estimate linguistic similarity. this method involves selectively fine-tuning lms on specific linguistic tasks and assessing the impact of the finetuning on other tasks (see figure 1 ), as well as the alignment of tasks in gradient space. we extricate single linguistic tasks from their entanglement in natural language via a method we call fine-tuning via gradient differentials (ftgd). ftgd selectively updates a small, relevant subspace of parameters with 'gradient differentials'. the contributions of this paper can be summarised as follows: in this section, we summarise related work and additional background on task spaces in mtl ( § 2.1), the use of lms for linguistic theorizing ( § 2.2) and fine-tuning machine learning models in lowdimensional subspaces ( § 2.3). ## methods in this paper, we connect work on task spaces in mtl @xcite @xcite with the idea of linguistic similarity spaces @xcite in a method we call similarity probing. similarity probing consists of three steps: first, we evaluate our untuned model on a wide range of linguistic phenomena (i.e. a 'linguistic task'). then, we selectively tune a separate lm on each linguistic task. finally, we evaluate each model again on all linguistic tasks and assess the tuning's impact in terms of performance transfers and compare different properties of the gradient updates (more details follow in § 3.2). fine-tuning a linguistic phenomenon is not straightforward. for that reason, we start by introducing a modified training procedure we will call 'finetuning via gradient differentials' ( § 3.1). ## experiments in the empirical analysis of our method, we pretrain three different generative lms up to various stages. then, we test the ftgd on a trained-out checkpoint to ensure that it works as intended. subsequently, we apply it to all intermediate checkpoints to interpret the change in the lms' language conceptualisation throughout the training process. ## discussion in this paper, we construct linguistic task spaces, representing an lm's language conceptualization, which can be used for linguistic hypothesis testing and as a holistic interpretability tool. we introduce ftgd to selectively fine-tune latent, entangled concepts such as linguistic tasks, and 'similarity prob-ing' to estimate similarities between linguistic tasks through their transfer learning and gradient analysis. we analyse the resulting similarity spaces of lms throughout pre-training to interpret their learning process. we find that learning of linguistic tasks begins localized and becomes more distributed with training, with increased parameter sharing among linguistic tasks and gradient alignment especially between linguistically related tasks. learning theory suggests that a trained model requires fewer dimensions due to increasingly efficient compression rules (e.g. @xcite . opposing the assumed reduction in intrinsic dimension, training in our experiments actually increases the number of extrinsic dimensions on which a task is learned. intrinsic and extrinsic dimensions might be inversely related in language models, as previously observed by @xcite . a more distributed processing of tasks allows for more overarching structure sharing and generalisation across different subconcepts, which potentially helps to achieve lower intrinsic dimensionality. furthermore, we find that generalization patterns remain surprisingly stable throughout pre-training, without stark shifts to new patterns-a behaviour more typical of human-like learning. this potentially reflects the weakness of classical neural network models to generalise systematically @xcite ; @xcite ; @xcite ). future generations of lms employing more human-like learning paradigms (see e.g. lake and baroni, 2023) may exhibit stronger shifts in generalization patterns. the observed continuity might explain the lack of successful curriculum learning strategies for language modelling in the past (see e.g. @xcite @xcite : in a learning process without notable shifts in generalisation patterns, changes in the data distribution during training are not beneficial. future reseach beyond language, our approach to interpreting lm conceptualisation can be applied to other domains to better understand the current weaknesses of lms, such as numerical reasoning and cross-lingual concept learning. furthermore, the potential for explicit linguistic hypothesis testing, though underexplored in this paper, can help bridge the gap between formal linguistic and com-putational linguistic research. large, state-of-theart llms may uncover subtle structural similarities that are informative to linguists. ## limitations there are several limitations to the presented methods. first, our fine-tuning and evaluation data are i.i.d. and come from a very narrow distribution: the data are not natural but synthetic, and all data are generated using the same templates. we use this very narrow i.i.d. data to asses the fine-tuning success during probing. however, we cannot be entirely sure whether we succeeded in fine-tuning a specific linguistic task rather than some idiosyncracies of the narrow data distribution. while our ftgd approach might elevate this issue slightly, it does not dispel our doubts completely. the optimal way to guarantee our results would be the evaluation on a set from a separate distribution. second, while our approach applies to all types of knowledge domains, it requires minimal pairs of tasks within that domain to fine-tune them selectively. minimal pairs are primarily used in linguistics and are uncommon in other knowledge domains. third, as discussed in the previous section, a major weakness of our probing approach lies in the necessary top-down definition of 'anchors' that we use to span the space. we utilise human-defined tasks and relate them to each other. however, a more accurate linguistic space can probably be described by 'anchors' that are defined through the model itself and span the conceptual space with maximal expressivity.