Where Does Linguistic Information Emerge in Neural Language Models? Measuring Gains and Contributions across Layers (2022)

Abstract: Probing studies have extensively explored where in neural language models linguistic information is located. The standard approach to interpreting the results of a probing classifier is to focus on the layers whose representations give the highest performance on the probing task. We propose an alternative method that asks where the task-relevant information emerges in the model. Our framework consists of a family of metrics that explicitly model local information gain relative to the previous layer and each layer’s contribution to the model’s overall performance. We apply the new metrics to two pairs of syntactic probing tasks with different degrees of complexity and find that the metrics confirm the expected ordering only for one of the pairs. Our local metrics show a massive dominance of the first layers, indicating that the features that contribute the most to our probing tasks are not as high-level as global metrics suggest.

## introduction probing neural language models aims at finding evidence of learned linguistic structure in the models' parameters by empirically testing hypotheses about the learned representations @xcite . this is often done by training a probing classifier on a diagnostic task with the representations at different layers as the input, and comparing task performance across layers. while probes are conceptually simple and widely used, the methodology and in particular the interpretation of the obtained results is subject to ongoing discussion @xcite . a classical pattern we often see when plotting probing accuracy across layers is that for higherlevel linguistic tasks, the model aggregates information over several layers until it reaches its highest performance. often, the curves start steep, flatten out, and eventually drop again in the final layers. in this paper, we zoom in on models' relative information gains between one layer and the previous one. by this, we aim to turn the focus of probing away from information that is already present in the non-contextualized embedding layer and focus exclusively on information that needs context to be retrieved. we also aim to make the contribution of each layer to the model's overall linguistic capabilities explicit. we argue that under the hypothesis that information in language models is structured as in classical nlp pipelines @xcite , the depth at which information emerges is as important as the impact of that information on model performance. when we think of language processing as a pipeline, we are primarily interested in where linguistic features emerge for the first time, not how long they are passed on to later layers. to formalize information gains, we modify the conditional probing framework proposed by @xcite . their method explicitly quantifies information that is not already present in a baseline representation. we modify this method along two lines: first, we make it local, by conditioning on the respective previous layer instead of a global baseline. second, we report the results as a share of total emergent information, across all layers of the network. this makes layer contributions comparable across tasks, where the overall performance may differ. we demonstrate how to use our framework in practice, by applying it to two pairs of syntactic probing tasks. each pair is formed by tasks that are structurally equal but for which we can reasonably assume a natural order in which the relevant linguistic representations emerge within the models. the first pair compares predicting partof-speech (pos) tags that are the most frequent for a word form (mfts) to predicting tags that are not @xcite . recent work has hypothesized that non-mfts may be best represented in the deeper layers of a model such as bert @xcite : each layer's contribution beyond the information already present in the uncontextualized layer is more significant for deeper layers, and therefore pos information could be found later in the model than previously assumed @xcite . the second pair of tasks compares predicting the position of a word's dependency head (the syntactic parent) to predicting the position of the head's head (the grandparent). information for predicting grandparents has in previous work been found in deeper layers than information for predicting parents @xcite . for each pair of tasks, we test where the relevant information emerges in the model, how the resulting pattern compares to global metrics, and if the metrics reflect the expected order of tasks. our results show that while the expected hierarchy holds for the parent vs. grandparent task, information for non-mfts emerges earlier in the model than for mfts. this contradicts previous expectations. also, results on seven independent monolingual bert models in different languages show that the orderings and patterns we observe are not robust, which raises questions about their generality, and about the validity of probing at large. ## related work how linguistic information is distributed across the layers of a neural model is one of the central questions in the probing literature. the consensus is that there is a hierarchical ordering of tasks. @xcite find a soft hierarchy of tasks when probing different layers of recurrent neural networks, from pos information being low in the hierarchy, to syntactic parents, grandparents, and great-grandparents. for their elmo model, @xcite find that parts-of-speech are better predicted from the first hidden layer and word senses from the second. @xcite probe bert for a range of different nlp tasks and find that the layers that are the most predictive for each task are ordered like a classical pipeline: from parts-ofspeech over syntactic dependencies, named entities and semantic roles to coreference. how probing experiments should be designed and evaluated is subject to ongoing discussion. some authors argue for simple classifiers @xcite to prevent the probes from learning the task and memorizing associations by themselves, while others make the case for more expressive models @xcite . while probes are most commonly evaluated using accuracy, recent work has proposed the use of alternative metrics that measure the effort of learning @xcite or emphasize the performance early in the training @xcite . @xcite propose to probe in an extrapolation setting, evaluating, among other setups, on the non-mfts in diagnostic pos tagging experiments. ## a taxonomy of metrics we start by categorizing methods along three dimensions: the first one (as proposed by @xcite ) concerns the relation of the baseline and the representation: how much more information can we extract from the representation than from the baseline (baselined probing), or how much information is extractable from the representation that does not overlap with information from the baseline (conditional probing). the second dimension, proposed by us, concerns the type of information intended to be measured: information relative to a non-contextualized baseline (a global baseline), or information gain relative to the previous layer (a local baseline). the local setting challenges the view that a linguistic property's place in the model is the layer where most usable information for it can be extracted. instead, we consider the layers where most usable information is gained relative to the previous layer to reflect the linguistic property's place within the model's hierarchy. we formulate and test the local correspondents of baselined and conditional probing in sections 3.4 and 3.5. thirdly, we modify the local metrics so that they, in addition to the absolute reporting of the results, also support the reporting of the relative share that each layer contributes to the final performance. while absolute numbers convey more information, relative numbers improve the comparability of results across tasks. we modify our local conditional metric from an absolute to a relative metric in section 3.6. ## experiments in our experiments, we apply the metrics defined in the previous section to study the performance of language models on two suitable probing tasks. ## results this section presents the results of our experiments. we have structured our presentation around the two ranking methods. ## discussion where is a feature located? the results for the different metrics show how it depends on the perspective which place within the model we assign to a linguistic property. while the most overall information is located in the middle layers, it is the early layers that maximize the local metrics by a huge margin, meaning that this is the place in the model where most information either emerges or becomes accessible. the complementary use of both families of metrics, local and global, gives us a more holistic picture of how information is structured within the model, as we argue that it it is not clear if a hierarchy of tasks within the model should be determined by where most information is added or where most information is accessible overall. an intuition supporting the former is the comparison to a human-made pipeline model, where tasks would naturally be placed where the information required for them is added -for instance, where the pos tagger is located, adding pos tags to the set of features. in these pipelines, there is no notion of how long information will be passed to higher-level tasks. expected hierarchies do not generally hold we see that when probing for mfts versus non-mfts, the perceived natural hierarchy of tasks (hypothesis 1) does not hold for the local metrics, neither in a coarse max layer analysis nor in the more fine-grained early contribution setting, as shown in sections 5.1 and 5.2. in the plots of the global metrics in section 5.1 we see that non-mfts often show both steeper gains in the beginning and more pronounced losses in the later layers, indicating that it is more specialized contextual information that the non-mfts require, but that information does not appear to emerge later than in the model that for mfts. we conclude that observations of a clear hierarchy of tasks depend on the focus on most usable overall information: they are already weaker in the global conditional setup, and are in one of two cases contradicted by local metrics. the massive dominance of layer 1 and 2 in all local metrics especially for the non-mfts but even for the parents and the grandparents tasks raises questions about how high-level the information that contributes the most to the overall performance on the probing task actually is. as the very first possibility of accessing contextual information already presents the heaviest boost, the features that are most crucial to solve the task appear to be surprisingly shallow. probing results are not robust an interesting point we noted across all metrics, but more distinctly for all local metrics as well as global conditional probing, is that the results are not stable across bert models in different languages. we do not consider the possibility to relate the different distribution of information across models to linguistic properties of the languages as we believe that this is impossible with the relatively small set of non-parallel models we analyze. apart from the language, they differ in several variables: most importantly, the data they are trained on, but some also in training details. however, we see it as an exciting path for future research to explore what causes a model to structure information in certain ways, and if this has implication on the model's performance on downstream tasks or robustness. relevance as the differences between measuring emergent information with and without conditioning on the previous layer are relatively small, one could suggest that the information is present in a similar form in ordinary accuracy plots across lay-ers: the slope of the curve can be used to estimate it. but in practice, such interpretations do not appear to be obvious: even though they can, as we showed, shed a different light at probing results when made explicit, previous work exclusively focused on the point of highest overall information. a discussion on the relevance of the parts of the network where information emerges has been absent from the literature. apart from that, we argue that conditional probing has strong theoretical advantages, as it explicitly accounts for information in the baseline representation, and that this makes basing the emergent information metrics on it favorable. ## conclusion we have collected and suggested metrics that model the information distribution in a model's layers from different perspectives: globally and locally, with or without conditioning on the baseline, and looking at absolute and relative gains of information. we used them on two pairs of probing tasks. first, we tested whether information for pos tags that are not the most frequent for a word is found in deeper layers than general pos information and found that while this is the case for overall information measured by global metrics, local metrics highlight that the most significant gains consistently happen in the very first layers in particular for the non-most frequent tags. for second task of predicting the syntactic parents versus grandparents of a token, however, the expected hierarchy in the model holds in the local setup at least in more finegrained relative metrics. these mixed results emphasize the additional insights that zooming in to local information gains can give us into the model, the task, and the probing methodology. probing experiments on seven monolingual bert models in different languages show that the metrics' behavior varies between models. while it is currently not feasible to relate the differences to specific properties of the models such as the language or the domain of the training data, a controlled training of parallel models where the additional variables are controlled for may enable such a comparison and is an insightful direction for future work. ajede, gabrielė aleksandravičiūtė, ika alfina, lene antonsen, katya aplonova, angelina aquino, carolina aragon, maria jesus aranzabe, bilge nas arıcan, hórunn arnardóttir, gashaw arutie, jessica naraiswari arwidarasti, masayuki asahara, deniz baran aslan, luma ateyah, furkan atmaca, mohammed attia, aitziber atutxa, liesbeth augustinus, elena badmaeva, keerthana balasubramani, miguel ballesteros, esha banerjee, sebastian bank, verginica barbu mititelu, starkaður @xcite . universal dependencies 2.9. lindat/clariah-cz digital library at the institute of formal and applied linguistics (úfal), faculty of mathematics and physics, charles university.