When classifying grammatical role, BERT doesn’t care about word order... except when it matters (2022)

Abstract: Because meaning can often be inferred from lexical semantics alone, word order is often a redundant cue in natural language. For example, the words chopped, chef, and onion are more likely used to convey “The chef chopped the onion,” not “The onion chopped the chef.” Recent work has shown large language models to be surprisingly word order invariant, but crucially has largely considered natural prototypical inputs, where compositional meaning mostly matches lexical expectations. To overcome this confound, we probe grammatical role representation in English BERT and GPT-2, on instances where lexical expectations are not sufficient, and word order knowledge is necessary for correct classification. Such non-prototypical instances are naturally occurring English sentences with inanimate subjects or animate objects, or sentences where we systematically swap the arguments to make sentences like “The onion chopped the chef”. We find that, while early layer embeddings are largely lexical, word order is in fact crucial in defining the later-layer representations of words in semantically non-prototypical positions. Our experiments isolate the effect of word order on the contextualization process, and highlight how models use context in the uncommon, but critical, instances where it matters.

## introduction and prior work large language models create contextual embeddings of the words in their input, starting with a static embedding of each token and progressively adding more contextual information in each layer @xcite @xcite . while these contextual embedding models are often praised for capturing rich grammatical structure, a spate of recent work has shown that they are surprisingly invariant to scrambling word order @xcite @xcite @xcite and that grammatical knowledge like part of speech, often attributed to contextual embeddings, is actually also captured by fixed embeddings @xcite . these results point to a puzzle: how can syntactic contextual information be important for language understanding when the words themselves, not their order, are what matter? we argue that this apparent paradox arises because of the redundant structure of language itself. lexical distributional information alone inherently captures a great deal of meaning @xcite @xcite , and typically both humans and machines can reconstruct meanings of sentences under local scrambling of words @xcite . in this paper, we study model behaviour in cases where word order is informative and is not redundant with lexical information. we focus on the feature of grammatical role (whether a noun is the subject or the object of a clause). most natural clauses are prototypical: in a sentence like "the chef chopped the onion", the grammatical roles of chef and onion are clear to humans from the words alone, without word order or context (see @xcite , for experiments in english and russian in which human participants successfully guessed which of two nouns was the subject and which was the object of a simple transitive clause, in the absence of word order and contextual information). this means syntactic word order is often redundant with lexical semantics. whether hand-constructed or corpus-based, most studies probing contextual representations have used prototypical sentences as input, where syntactic word order may not have much information to contribute to core meaning beyond the words themselves. yet human language can use syntax to deviate from the expectations generated by lexical items: we can also understand the absurd meaning of a rare non-prototypical sentence like "the onion chopped the chef" @xcite . is this use of syntactic word order available to pretrained models? in this paper, we train grammatical role probes on the embedding spaces of bert and gpt-2 foot_0 , and evaluate them on these rare non-prototypical examples, where the meaning of words in context is different from what we would expect from looking at the words alone. we focus on english because grammatical role is directly dependent on word order in english, and because we had access to sufficiently large english parsed corpora such that we could generate non-prototypical sentences, easily check them, and filter to grammatical ones. we probe for grammatical role because it is key to the basic compositional semantic structure of a sentence @xcite @xcite . while fixed lexical semantics contains information about grammatical role (animate nouns are likely to be subjects, etc), the grammatical role of a word in english is ultimately determined by syntactic word order. probing grammatical role lets us examine the interplay between syntactic word order and lexical semantics in forming compositional meaning through model layers. for all of our experiments, we train grammatical role probes with standard data and test them on either prototypical cases or non-prototypical cases (where word order matters), to understand if grammatical embedding under normal circumstances is sensitive to word order. our experiments reveal three key findings: more generally, we highlight the importance of examining models using non-prototypical examples, both for understanding the strength of lexical influence in contextual embeddings, but also for accurately isolating syntactic processing where it is taking place. foot_1 as opposed to more general syntactic probing tasks (e.g., dependency parsing), grammatical role is a linguistically significant yet specific task that is both syntactic and semantic. as such, we can choose these linguistically-informed sets of nonprototypical examples where the lexical semantics does not match the compositional meaning implied by the syntax. non-prototypical examples give us a unique perspective on how syntactic machinery like word order influences compositional meaning representation independently from lexical semantics. stud-ies in probing have controlled for lexical semantics by substituting content words for nonce words ("jabberwocky" sentences, as in hall @xcite or random real words ("colorless green idea" sentences, as in @xcite . a tradeoff is that these methods lead to out-of-distribution sentences whose words are unlikely to ever co-occur naturally. rather than bleaching the effect of lexical semantics, our setup lets us examine the interplay between lexical semantics and syntactic representation in a controlled environment, isolating the effects of syntactic word order while using in-distribution examples. recent work on representation probing has focused on improving probing methodologies to make sure that extracted information is not spurious or not simply lexical @xcite @xcite @xcite . our experiments are a complementary approach, where we use standard probing methods, but use linguistically-informed data selection to address the ambiguity of what classifiers are extracting. ## experiment 2: controlling for distributional information by swapping subjects and objects in experiment 1 we show that the contextualization process consists of gradual grammatical information gain for non-prototypical examples, even though this is largely obscured in the majority prototypical examples where the lexical semantics also contains accurate syntactic information. in this experiment, we ask: does this contextualized information about grammatical role stem from word order and syntax, or from distributional (bag-of-words) effects when seeing all words in the sentence? we answer this question by creating example pairs where we control for distributional information by keeping all the words the same, but swapping the positions of the subject and the object. such pairs of the type "the chef chopped the onion" → "the onion chopped the chef" (real sample in appendix b) have identical distributional information. to accurately classify grammatical role in both sentences, the model we're probing would have to be attuned to the ways in which small changes in word order globally affect meaning. ## discussion while recent work has shown that large language models come to rely largely on distributional semantic information, we consider the model's ability to overcome these distributional cues. research showing that models rely on lexical and distributional information is not at odds with our findings that this can be overridden. in fact, even though humans can accurately understand non-prototypical sentences, human syntactic processing is often influenced by the lexical semantics of words, as evidenced by studies on human subjects @xcite @xcite as well as by lexically-influenced syntactic processes in human languages, like differential object marking @xcite -a phenomenon whereby non-prototypical grammatical objects are marked. more generally, while we have shown that it is tempting for a straightforward probing approach to conclude that grammatical role information is available to the lowest layers of bert, separately analyzing prototypical and non-prototypical arguments makes it clear that the picture is more compli-cated. at lower layers, bert representations can typically classify subjects and objects, but when a non-prototypical meaning is expressed, accurate classification is not available until the higher layers. we argue that considering probing performance on these non-prototypical instances is crucial. a key design feature of human language is the ability to talk about things that aren't there or don't exist @xcite , and it has been argued that the combinatoric power of syntax exists to allow humans to say things that are subtle, surprising, or impossible @xcite . thus, considering probing accuracy on the average task may be misleading. insofar as being able to understand non-prototypical meanings is a hallmark of human language and insofar as these meanings may differ in systematic ways from prototypical meanings, considering such cases is crucial for understanding how language models represent language.