The Architectural Bottleneck Principle (2022)

Abstract: In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a model's representations contain. This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component. Relying on this principle, we estimate how much syntactic information is available to transformers through our attentional probe, a probe that exactly resembles a transformer's self-attention head. Experimentally, we find that, in three models (BERT, ALBERT, and RoBERTa), a sentence's syntax tree is mostly extractable by our probe, suggesting these models have access to syntactic information while composing their contextual representations. Whether this information is actually used by these models, however, remains an open question.

## introduction the surprising performance of pretrained language models on diverse natural language processing tasks has sparked interest in their analysis. probing is one of the most prevalent methods employed to engage in such an analysis. in a typical probing study @xcite @xcite , the weights of the model under consideration are first frozen. a probe is then trained on top of the model's contextual representations in an attempt to predict one of the input sentence's properties, e.g., its syntactic parse. unfortunately, best practices on how to design such probes remain contested. on one side of the debate, some argue for simplicity, suggesting that simple probes are to be * equal contribution. in this paper, we propose the architectural bottleneck principle (abp) as a guideline for constructing useful probes. under the abp, a probe's architecture should mirror a component of the model being probed. previous work has mostly focused on how much information is contained in a set of representations. however, if we care about whether the information is in fact used by the model, we should instead ask how much information the model in question could use. 1 under this perspective, the probed model's architecture acts as a natural bottleneck to how much information the model could use-and should thus also act as a constraint when probing. 2 as a concrete example, we posit that a transformer's attention head serves as a bottleneck to its use of syntactic information, as these are the only components in a transformer with access to multiple tokens at once. following the abp, we thus propose the attentional probe, which looks exactly like an attention head. this probe allows us to answer one specific question: how much syntactic information could a transformer use while computing its attention weights? our results reveal that most-albeit not allsyntactic information is extractable with this simple attention head architecture: while we estimate english sentences to contain on average 31.2 bits of information about their syntactic tree structure, the attentional probe can extract up to 28.0 bits. furthermore, while these results hold for three popular transformer-based language models (bert, albert and roberta), they do not for a similar but untrained model. this suggests that training a model shapes its representations to encode syntactic information. we find this trend holds across four typologically diverse languages (basque, english, tamil, and turkish). in contrast, when we keep bert's pretrained parameters frozen and analyze the weights of its pretrained attention heads, we observe that they do not seem to encode syntax under our operationalisation. ergo, while we know these models could use syntactic information to compute attention weights, whether they actually do remains an open question. ## a taxonomy of probing principles there are many competing approaches for how to design an effective probe @xcite . we taxonomise them into principles here. focusing on how much information a model could use in its final layer, 3 @xcite propose what we term the linearity principle; many subsequent studies then adopted it in designing their probes @xcite @xcite . other researchers, however, argued that a model's nonfinal layers do not necessarily encode information linearly @xcite . they then suggested that a probe should measure the total amount of information present in a model's representations-independent of whether it is actually used by the model. this led to a second principle, which we outline below. @xcite . a probe's goal is to estimate how much information is encoded in a set of representations. thus, probes should be as complex as necessary to extract all relevant information. ## probing with information theory in this paper, we take the position that the goal of probing is to determine how much information one can extract from the representations being probed. following @xcite , we now operationalise this value formally using information theory, which offers us a clean framework to quantify information. specifically, the measure we are interested in is a v-information @xcite ). foot_2 @xcite famously quantified the amount of information that a random variable (r) contains about another (a) as their mutual information ## an attentional probe in our experiments, we will focus on a transformer's attention mechanism. concretely, many researchers (e.g., @xcite @xcite have asserted that syntactic information is used by transformers when computing their attention weights (albeit not uncontroversially; for a review, see @xcite . further, attention heads are the only components in a transformer which have access to multiple words at the same time. thus, exploring the abd in the context of attention heads is a natural starting point. specifically, following the abp, we will investigate how much information a transformer's attention head could extract from the representations fed into it. given an input sentence s, a transformer @xcite will generate a set of representations @xmath0. an attention head then uses these representations to compute the attention weights this variational family includes the set of all distributions computable by an attention head architecture. in practice, however, we cannot compute the infimum over the set v as required in eq. ( 5 ). as an approximation, we train our attentional probe to minimise a cross-entropy loss, which gives us an estimate of the v-entropy. we expand on this point in app. a.1. foot_4 models. empirically, we study multilingual bert in all four languages under consideration @xcite as well as roberta and albert @xcite , which are only available in english. in line with the abd, we keep our probe's hidden size the same as in the probed architectures. finally, we also probe an untrained transformer model with the same architecture as bert as a baseline. training. we train our probes with adamw @xcite using its default hyper-parameters in pytorch (paszke et al., 2019). 7 baselines and skylines. 8 we contrast our attentional probe's v-information against two other values. first, as a baseline, we investigate a special case of our model where @xmath1, inspired by recent work on structural probing @xcite @xcite . notably, this equality leads to symmetric attention weight matrices; by modelling the root explicitly, however, we still get a distribution over directed trees. this baseline evaluates whether previous work, by over-constraining their probes, has underestimated the amount of information available to a transformer's attention mechanism. we report this value as structural v-information. second, as a skyline, we compare our attentional probe to an estimate of the true mutual information i(r; a), model layer i v i c v bert 7 28.0 31.2 90% roberta 9 25.5 31.2 82% albert 7 27.7 31.2 89% we present our main results in fig. 1 . first, our probes estimate that most syntactic information is extractable in the middle layers, as previously reported by @xcite . second, fig. 1 shows that a large amount of syntactic information is encoded in the representations fed to the attention heads. further, while we estimate close to 31 bits of information to be encoded in english, tamil, and basque sentences, we only estimate around 15 bits to be encoded in turkish sentences; we suspect this is due to turkish having the shortest sentences in the corpus (see app. i for these lengths). third, we find that, out of the total syntactic information present in the sentences, nearly all is available to the transformer-based models under consideration. in english, for instance, we find the v-coefficient of the most informative layer to be 90%, 82%, and 89% in bert, roberta and albert, respectively; see tab. 1 . this means they have access to roughly 85% of all syntactic information in a sentence. these trends are consistent across the four languages we have considered. notably, this is not the case for the untrained bert representations, which suggests this structure is a byproduct of the language models' pretraining procedures. additionally, we find that our structural baseline considerably underestimates the models' potential ability to reconstruct a syntax tree; the best english structural baseline recovers only 23 bits of information (versus 28 bits by the attentional probe). one can see this effect in fig. 1 , where all of the structural baseline results fall beneath their corresponding attentional probe counterparts. foot_8 1 2 3 4 5 6 7 8 9 10 11 # layer 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 uas attentional probe head 1 head 2 head 3 head 4 head 5 head 6 head 7 head 8 untrained right branching left branching we also display the uas achieved by the attentional probe, the best head per layer of an untrained bert, and a right and left-branching baseline. in a final experiment, we plug bert's attention weights, as computed with its pretrained attention heads, directly into eq. ( 8 ) and analyse its resulting unlabelled attachment scores. these results are presented in fig. 2 for english (as well as in app. h for the other analysed languages). in short, they reveal that, while attention heads could use a large amount of syntactic information, none of the actual heads compute weights that strongly resemble syntax trees; see htut et al. 2019 for similar results. as bert has 8 attention heads, however, it might be the case that the syntactic information is used in a distributed manner, with each head relying on a subset of this information (see tab. 3 in @xcite for results partly supporting this hypothesis). ## conclusions in this paper, we have approached probing from a new perspective. rather than asking how much information is encoded by the model, we ask how much information its components could extract. we then quantify this amount using v-information. evaluating the attention mechanism of popular transformer language models, we find that the majority of the information about the syntax tree of a sentence is in fact extractable by the model. this, however, is not true for randomly initialised transformer models. our results, thus, lead us to conclude that a transformer's training leads its attention heads to have the potential to decode syntax trees.