The Universe of Utterances According to BERT (2023)

Abstract: It has been argued that BERT “rediscovers the traditional NLP pipeline”, with lower layers extracting morphosyntactic features and higher layers creating holistic sentence-level representations. In this paper, we critically examine this assumption through a principle-component-guided analysis, extracing sets of inputs that correspond to specific activation patterns in BERT sentence representations. We find that even in higher layers, the model mostly picks up on a variegated bunch of low-level features, many related to sentence complexity, that presumably arise from its specific pre-training objectives.

## introduction the transformer architecture of neural networks @xcite shows state-of-the-art performance on a range of nlp tasks @xcite . at the same time, the question of what transformer models learn exactly has motivated a number of studies into the representations that they construct @xcite @xcite , with an increasingly popular answer being that they recreate the classical nlp pipeline of incremental abstraction, from morphosyntax to semantics @xcite . in this paper, we put this finding to the test, asking to what extent representations learned by pretrained bert @xcite capture systematic meaning distinctions, as opposed to more shallow and potentially idiosyncratic properties. the challenge of this question is that it is open-ended: in order not to bias the analysis, we do not want to rely on a set of categories that we a priori expect to be relevant, in contrast to most probing approaches, which correlate model representations with properties of inputs or performance metrics (see section 2 for details). instead, we adapt the approach that @xcite proposed to analyze decoder-only transformer models with causal masking. they regard feed-forward (ff) sublayers in such models as neural memory units and extract inputs that produce maximal activations in a random subset of their neurons. manual analysis of these sets shows what categorization of inputs arises inside the model. 1 we extend the approach by geva et al. in two dimensions. first, we apply it to pre-trained bidirectional encoder-only transformer models like bert @xcite , which, unlike causal lms, do not have a specific token guaranteed to represent all of the input. to do so, we analyze two types of "prominent" tokens: the cls pseudo-token, often used for whole-sentence representation @xcite ; even if its usefulness for downstream tasks is debatable, cf. reimers and gurevych 2019), and the first subword of the root element in sentences annotated with universal dependencies @xcite . we regard these two tokens as good candidates for loci of high-level, abstract representations of inputs learned by bert. second, we replace the analysis of random neurons by guided exploration. we find that embeddings of both cls tokens and root tokens at upper layers are highly intercorrelated. therefore we propose to analyze major principal components of activation matrices, in essence tracking influential groups of highly congruent neurons. we exploit this approach to provide an analysis of the sentence patterns that bert attends to. we find that while lower levels of bert are predictably more attuned to lexical effects, activations in higher levels track a wide range of idiosyncratic phenomena from various linguistic levels, from individual wordforms and bigrams to lexical classes (rare words), syntactic patterns (e.g., clauses with imperatives), and miscellaneous sentence types (recommendations, incomplete sentences, short exclamations). overall, our results indicate that the typology of sentences according to bert is dominated by what may be called natural classes @xcite -clusters of objects that are characterized by a combination of values of several featureswith the embeddings showing little evidence of principled semantic properties. foot_1 2 related work probing transformers two prominent avenues of the study of transformers in nlp are (i) probing analysis of internal representations of linguistic inputs computed by the models (e.g., @xcite @xcite and (ii) the analysis of the attention patterns that transformers converge on to compute these representations @xcite and many others). both strategies rely on predefined arrays of nlp tasks and features, which are either used as benchmarks or are selected to highlight peculiarities of models. in contrast, geva et al. analyze the representation of the last unmasked token of the input sequence in the causal language model by @xcite , which serves as the representation of the whole prefix. by sampling neurons from feedforward sublayers of the model and manually inspecting sentences that give rise to maximum values of these neurons, they show that the latter recognise different patterns in the input -with lower-layer activations tuned to more superficial lexical and syntatctic features and upper layers arguably more tuned to semantics. we extend this approach to bidirectional lms. ## conclusions the good performance of transformers on downstream tasks is often explained by their ability to extract meaningful linguistic, generalizing features from raw text @xcite @xcite . when approaching this problem from the point of view of a particular set of tasks, however, there is always the danger that good model performance is due to accidental covariates in the data that help models solve the task without creating useful generalizations @xcite . our analysis of loci in bert that are highly likely to aggregate linguistic generalizations about the input sentence indicates that this problem might indeed be present in this model as well: we find a conspicuous absence of high-level generalizations and prominent shallow features even in the final layers, arguably because they prove useful in solving the cloze and next-sentence-prediction pre-training tasks. many of these are complexity-related, similar to biases found in word embeddings @xcite . these findings arguably go some way towards explaining the instability of the performance of different instances of bert on the same downstream task @xcite and of the variance in the effects of bert interventions @xcite . the question of whether it is possible to create a pre-training task that would nudge the model towards extracting high-level features remains open.