Mediators in Determining what Processing BERT Performs First (2021)

Abstract: Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing what tasks. However, little work addressed potential mediating factors in such comparisons. As a test-case mediating factor, we consider the prediction’s context length, namely the length of the span whose processing is minimally required to perform the prediction. We show that not controlling for context length may lead to contradictory conclusions as to the localization patterns of the network, depending on the distribution of the probing dataset. Indeed, when probing BERT with seven tasks, we find that it is possible to get 196 different rankings between them when manipulating the distribution of context lengths in the probing dataset. We conclude by presenting best practices for conducting such comparisons in the future.

## introduction the strong performance of end-to-end models and the difficulty in understanding their inner workings has led to extensive research aimed at interpreting their behavior @xcite @xcite . this notion has led researchers to investigate the behavioral traits of networks in general @xcite and representative architectures in particular @xcite . within nlp, transformer-based pretrained embeddings are the basis for many tasks, which underscores the importance in interpreting their behavior @xcite , and especially the behavior of bert @xcite , perhaps the most widely used of transformer-based models. in this work, we analyze the common approach of probing ( §2), used to localize where "knowledge" of particular tasks is encoded; localization is often carried out in terms of the layers most responsible for the task at hand (c.f. @xcite . various works @xcite @xcite showed that some tasks are processed in lower levels than others. we examine the extent to which potential mediating factors may account for observed trends and show that varying some mediating factors (see §2) may diminish, or even reverse, the conclusions made by @xcite . specifically, despite reaffirming t19's experimental findings, we contest t19's interpretation of the results, namely that the processing carried out by bert parallels the classical nlp pipeline. indeed, t19 concludes that lexical tasks (pos tagging) are performed by the lower layers, followed by syntactic tasks, whereas more semantic tasks are performed later on. this analysis rests on the assumption that the nature of the task (lexical, syntactic, or semantic) is the driving force that determines what layer performs what analysis. we show that other factors should be weighed in as well. specifically, we show that manipulating the distribution of examples in the probing dataset can lead to a variety of different conclusions as to what tasks are performed first. we argue that potential mediators must be considered when comparing tasks, and focus on one such mediator -the context length, which we define as the number of tokens whose processing is minimally required to perform the prediction. we operationalize this notion by defining it as the maximal distance between any two tokens for which a label is predicted. this amounts to the span length in tasks that involve a single span (e.g., ner), and to the dependency length in tasks that address the relation between two spans. see §2. our motivation for considering context length as a mediator is grounded in previous work that presented the difficulty posed by long-distance dependencies in various nlp tasks @xcite , and particularly in previous work that indicated the transformers' difficulty to generalize across different dependency lengths @xcite . we show that in some of the cases where one task seems to be better predicted by a higher layer than another task, controlling for context length may reverse that order. indeed we show that 196 different rankings between the seven tasks explored in t19 may be obtained with a suitable distribution over the probing datasets, namely 196 different ways to rank the tasks according to their expected layer. moreover, our results show that when context length is not taken into account, one task (e.g., dependency parsing) may seem to be processed at a higher layer than another (e.g., ner), when its expected layer (see §2) is, in fact, lower for all ranges of context lengths ( §3.1.1). ## background we begin by laying out the terminology and methodology we will use in the paper. ## experiments we hypothesize that the context length is a mediating factor in the e layer of a task. in order to test this hypothesis, we run the following experiments, aiming at isolating the context length. we use the spr1 dataset @xcite to probe spr, the english web treebank for the dep. task @xcite , the semeval 2010 task 8 for the rc task @xcite , and the ontonotes 5.0 dataset @xcite for the other tasks. configurations follow the defaults in the jiant toolkit implementation @xcite . in addition, we work with the bertbase model. ## conclusion we showed that when performing edge probing to identify what layers are responsible for addressing what tasks, it is imperative to take into account potential mediators, as they may be responsible for much of the observed effect. specifically, we showed that context length has a significant impact on a task's e layer . our analysis shows the wide range of relative orderings of the expected layers for different tasks when assuming different context length distributions; from extreme edge cases, like the one we observed in §3.1.1, to more common, but potentially misleading ones, where the difference between expected layers may dramatically increase or decrease depending on the context length distribution. most importantly, it shows that by manipulating the context length distribution, we may get a wide range of outcomes. our work suggests that mediating factors should be taken into account when basing analysis on the e layer . on a broader note, alternative hypotheses should be considered, before limiting oneself to a single interpretation. future work will consider the effect of other mediating factors. the two methods we used, nde and controlled effect, can be used to examine the impact of other mediating factors and should be adopted as part of the field's basic analysis toolkit (cf. @xcite . nde should be used when several effects are examined simultaneously, as it facilitates the assessment of their effect on the tasks' complexity. it is also advisable to use nde when a more practical examination is required, i.e., when distributions of the mediators are given empirically, as it is easier to derive the mediating factors' impact using this method. in contrast, the controlled effect method should be used when examining the effects of two variables (e.g., tasks and mediating factors) or when comparing several tasks with one mediating effect.