Word Order Does Matter and Shuffled Language Models Know It (2022)

Abstract: Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the importance of word order information. Somewhat counter-intuitively, some of these studies also report that position embeddings appear to be crucial for models’ good performance with shuffled text. We probe these language models for word order information and investigate what position embeddings learned from shuffled text encode, showing that these models retain a notion of word order information. We show this is in part due to a subtlety in how shuffling is implemented in previous work – before rather than after subword segmentation. Surprisingly, we find even Language models trained on text shuffled after subword segmentation retain some semblance of information about word order because of the statistical dependencies between sentence length and unigram probabilities. Finally, we show that beyond GLUE, a variety of language understanding tasks do require word order information, often to an extent that cannot be learned through fine-tuning.

## introduction transformers @xcite , when used in the context of masked language modelling @xcite , consume their inputs concurrently. there is no notion of inherent order, unlike in autoregressive setups, where the input is consumed token by token. to compensate for this absence of linear order, the transformer architecture originally proposed in @xcite includes a fixed, sinusoidal position embedding added to each token embedding; each token carries a different position embedding, corresponding to its position in the sentence. the transformer-based bert @xcite replaces these fixed sinusoidal * equal contribution. order was decided by a coin toss. figure 1 : pearson correlations between position embeddings for full-scale models; the patterns are similar to fully learnable absolute embeddings @xcite and can be said to have learned something about position. we later demonstrate that this is not the case with post-bpe scrambling. position embeddings are the only source of order information in these models; in their absence, contextual representations generated for tokens are independent of the actual position of the tokens in a sentence, and the models thus resemble heavily overparameterised bags-of-words. @xcite pre-trained roberta models on shuffled corpora to demonstrate that the performance gap between these 'shuffled' language models and models trained on unshuffled corpora is minor (when fine-tuned and evaluated downstream on the glue @xcite benchmark). they further show that this gap is considerably wider when a model is pre-trained without position embeddings. in this paper, we attempt to shed some light on why these models behave the way they do, and in doing so, seek to answer a set of pertinent questions: contributions we first demonstrate, in section 3, that shuffled language models do contain word order information, and are quite responsive to simple tests for word order information, particularly when compared to models trained without position representations. in section 4, we demonstrate that pre-training is sufficient to learn this: position embeddings provide the appropriate inductive bias, and performing bpe segmentation after shuffling results in sensible n-grams appearing in the pre-training corpus; this gives models the capacity to learn word order within smaller local windows. other minor cues -like correlations between sentence lengths and token distributionsalso play a role. we further corroborate our analysis by examining attention patterns across models in sec. 5. in section 6, we show that, while shuffled models might be almost as good as their unshuffled counterparts on glue tasks, there exist nlu benchmarks that do require word order information to an extent that cannot be learned through fine-tuning alone. finally, in section 7, we describe miscellaneous experiments addressing the utility of positional embeddings when added just prior to fine-tuning. we experiment with their models, as well as with smaller models that we can train with a smaller carbon footprint. to this end, we downscale the roberta architecture used in @xcite . concretely, we train single-headed roberta models, dividing the embedding and feed-forward dimensionality by 12, for 24 hours on a single gpu, on 100k sentences sampled from the toronto book corpus. to this end, we train a custom vocabulary of size 5,000, which we use for indexing in all our subsequent experiments. while these smaller models are in no way meant to be fine-tuned and used downstream, they are useful proofs-of-concept that we later analyse. ## probing for word order we begin by attempting to ascertain the extent to which shuffled language models are actually capable of encoding information pertaining to the naturalistic word order of sentences. we perform two simple tests on the full-scale models, in line with wang and chen (2020): the first of these is a classification task where a logistic regressor is trained to predict whether a randomly sampled token precedes another in an unshuffled sentence, and the second involves predicting the position of a word in an unshuffled sentence. the fact that we do not fine-tune any of the model parameters is noteworthy: the linear models can only learn word order information if it reflects in the representations the models generate somehow. pairwise classification for this experiment, we train a logistic regression classification model on word representations extracted from the final layer of the transformer encoder, mean pooling over sub-tokens when required. for each word pair x and y, the classifier is given a concatenation of our model m's induced representations m(x) ⊕ m(y) and trained to predict a label indicating whether x precedes y or not. holding out two randomly sampled positions, we use a training sets sized 2k, 5k, and 10k, from the universal dependencies english-gum corpus @xcite (excluding sentences with more than 30 tokens to increase learnability) and a test set of size 2, 000. we report the mean accuracy from three runs. regression using the same data, we also train a ridge-regularised linear regression model to predict the position of a word p(x) in an unshuffled sentence, given that word's model-induced representation m(x). r 2 score is reported per model. to prevent the regressors from memorising word to position mappings, we perform 6-fold cross-validation, where the heldout part of the data contains no vocabulary overlap with the corresponding train set. ## hidden word-order signals in section 3, we observed that @xcite 's shuffled language models surprisingly exhibit information about naturalistic word order. that these models contain positional information can also be seen by visualizing position embedding similarity. figure 1 displays pearson correlations foot_2 for position embeddings with themselves, across positions. here, we see that the shuffled models satisfy the idealised criteria for position embeddings described by @xcite : namely, they appear to be a) monotonous within smaller context windows, and b) invariant to translation. if position embedding correlations are consistent across offsets over the entire space of embeddings, the model can be said to have 'learned' distances between tokens. since transformers process all positions in parallel, and since language models without position embeddings do not exhibit such information, position embeddings have to be the source of this information. in what follows, we discuss this apparent paradox. subword vs. word shuffling an important detail when running experiments on shuffled text, is when the shuffling operation takes place. when tokens are shuffled before bpe segmentation, this leads to word-level shuffling, in which sequences of subwords that form words remain contiguous. such sequences become a consistent, meaningful signal for language modelling, allowing models to efficiently utilise the inductive bias provided by position embeddings. thus, even though our pretrained models have, in theory, not seen consecutive tokens in their pre-training data, they have learned to utilise positional embeddings to pay attention to adjacent tokens. the influence of this is somewhat visible in figure 2 : while models trained on text shuffled before and after segmentation both exhibit shifts in the polarity of their position correlations, only the former show bands of varying magnitude, similar to the full-scale models. ravishankar and søgaard (2021) discuss the implications of these patterns in a multilingual context; we hypothesise that in our context, the periodicity in magnitude is a visible artefact of the model's ability to leverage position embeddings to enable offset attention. in section 5, we analyse the effect of shuffling the pretraining data on the models' attention mechanisms. accidental overlap in addition to the n-gram information which results from shuffling before segmentation, we also note that short sentences tend to include original bigrams with high probability, leading to stronger associations for words that are adjacent in the original texts. this effect is obviously much stronger when shuffling before segmentation than after segmentation. figure 3 shows how frequent overlapping bigrams (of any sort) are, comparing word and subword shuffling over 50k sentences. sentence length finally, we observe some preserved information about the original word order even when shuffling is performed after segmentation. we hypothesize that this is a side-effect of the non-random relationship between sentence length and unigram probabilities. that unigram probabilities correlate with sentence length follows from the fact that different genres exhibit different sentence length distributions @xcite . also, some words occur very frequently in formulaic contexts, e.g., thank in thank you. this potentially means that there is an approximately learnable relationship between the distribution of words and sentence boundary symbols. to test for this, we train two smaller language models on unigram-sampled corpora: for the first, we use the first 100k bookcorpus sentences as our corpus, shuffling tokens at a corpus level (yet keeping the original sentence lengths). the stark difference in position embedding correlations between that and shuffling is seen in figure 2 . for the second, we sample from two different unigram distributions: one for short sentences and one for longer sentences (details in appendix b). while the first model induces no correlations at all, the second does, as shown in figure 4 , implying that sentence length and unigram occurrences is enough to learn some order information. ## attention analysis transformer-based language models commonly have attention heads that attend to neighboring positions @xcite . such attention heads are positional and can only be learned in the presence of order information. we attempt to visualise the attention mechanism for pre-trained models by calculating, for each head and layer, the offset between a token and the token that it pays maximum attention to foot_3 . we then plot how frequent each offset is, as a percentage, over 100 book corpus sentences, in figure 5 , where we present results for two full-scale models, and two smaller models (see §2). when compared to nopos, shuf.n1 has a less uniform pattern to its attention mechanism: it is likely, even at layer 0, to prefer to pay attention to adjacent tokens, somewhat mimicking a convolutional window (cordonnier et al., 2020). we see very similar differences in distribution between our smaller models: shuffling after segmentation, i.e., at the subword level, influences early attention patterns. evaluating on the more rigorous set of super-glue tasks foot_4 @xcite and on the adversarially-filtered winograd schema examples @xcite of the winogrande dataset @xcite produces results which paint a more nuanced picture compared to those of @xcite . the results, presented in this pattern alters for the boolq yes/no question answering dataset @xcite , the commitmentbank (de marneffe et al., 2019), the record reading comprehension dataset @xcite , both the winograd schema tasks, and to some extent the words in context dataset @xcite . for these tasks we observe a larger gap between orig and shuf.n1 (mean: 8.1 points), and an even larger one between orig and nopos (mean: 19.78 points). we note that this latter set of tasks requires inferences which are more context-sensitive, in comparison to the two other tasks or to the glue tasks. consider the winograd schema tasks, for example. each instance takes the form of a binary test with a statement comprising of two possible referents (blue) and a pronoun (red) such as: sid explained his theory to mark but he couldn't convince him. the correct referent of the pronoun must be inferred based on a special discriminatory segment (underlined). in the above example, this depends on a) the identification of "sid" as the subject of "explained" and b) inferring that the pronoun serving as the subject of "convinced" should refer to the same entity. since the winograd schema examples are designed so that the referents are equally associated with their context foot_6 , word order is crucial foot_7 for establishing the roles of "sid" and "mark" as subject and object of "explained" and "he" and "him" as those of "convinced". if these roles cannot be established, making the correct inference becomes impossible. a similar reasoning can be applied to the words in context dataset and the commitmentbank. the former task tests the ability of a model to distinguish the senses of a polysemous word based on context. while this might often be feasible via a notion of contextual association that higher-order distributional statistics are sufficient for, some instances will require awareness of the word's role as an argument in the sentence. the latter task investigates the projectivity of finite clausal complements under entailment cancelling operators. this is dependent on both the scope of the entailment operator and the identity of the subject of the matrix predicate @xcite , both of which are sensitive to word order information. a final consideration to take into account is dataset filtering. two of the tasks where we observe the largest difference between orig, shuf.n1, and nopos -winogrande and record -apply filtering algorithms to remove cues or biases which would enable models to heuristically solve the tasks. this indicates that by filtering out examples containing cues that make them solvable via higher order statistics, such filtering strategies do succeed at compelling models to (at least partially) rely on word order information. dependency tree probing besides glue and paws, @xcite 's analysis also includes several probing experiments, wherein they attempt to decode dependency tree structure from model representations. they show, interestingly, that the shuf.n4, shuf.n3 and shuf.n2 models perform only marginally worse than orig, with shuf.n1 producing the lowest scores (lower, in fact, than shuf.corpus). given the findings of section 3, we are interested in taking a closer look at this phenomenon. here, we surmise that dependency length plays a crucial role in the probing setup, where permuted models may succeed on par with orig in capturing local, adjacent dependencies, but increasingly struggle to decode longer ones. to evaluate the extent to which this is true, we train a bilinear probe (used in @xcite ) on top of all model representations and evaluate its accuracy across dependencies binned by length, where length between words w i and w j is defined as |i -j|. we opt for using the bilinear probe over the pareto probing framework @xcite , as the former learns a transformation directly over model representations, while the latter adds the parent and child mlp units from @xcite -acting more like a parser. we train probes on the english web treebank @xcite and evaluate using uas, the standard parsing metric. figure 6 shows ∆ probing accuracy across various dependency lengths for nopos and shuf.n1, with respect to orig foot_8 ; we include detailed ∆s for all models in appendix c. for nopos, parsing difficulty increases almost linearly with distance, often mimicking the actual frequency distribution of dependencies at these distances in the original treebank (appendix c); for shuf.n1, the picture is a lot more nuanced, with dependencies at a distance of 1 consistently being closer in terms of parseability to orig, which, we hypothesise, is due to its adjacency bias. ## other findings random position embeddings are difficult to add post-training we tried to quantify the degree to which the inductive bias imparted by positional embeddings can be utilised, solely via finetuning. to do so, for a subset of glue tasks (mnli, qnli, rte, sst-2, cola), we evaluate nopos, and a variant where we randomly initialised learnable position embeddings and add them to the model, with the rest of the model equivalent to nopos. we see no improvement in results, except for mnli, that we hypothesise stems from position embeddings acting as some sort of regularisation parameter. to test this, we repeat the above set of experiments, this time injecting gaussian noise instead; this has been empirically shown to have a regularising effect on the network @xcite . adding gaussian noise led to a slight increase in score for just mnli, backing up our regularisation hypothesis. ## on word order in humans it is generally accepted that a majority of languages have "canonical" or "base' word orderings @xcite ) (e.g. subject-verb-object in english, and subject-object-verb in hindi). linguists consider word order to be a coding property -mechanisms by which abstract, syntactic structure is encoded in the surface form of utterances. beyond word order, other coding properties include, e.g. subject-verb agreement, morphological case marking, or function words such as adpositions. in english, word order is among the most prominent coding properties, playing a crucial role in the expression of the main verb's core arguments: subject and object. for more morphologically complex languages, on the other hand, (e.g. finnish and turkish), word order is primarily used to convey pragmatic information such as topicalisation or focus. in such cases, argument structure is often signalled via case-marking, where numerous orderings become possible (shift in topic or focus nonwithstanding). we refer the reader to @xcite for a broader discussion of these topics and their implications when studying syntax through language models. more generally, evidence for the saliency of word order in linguistic processing and comprehension comes from a variety of studies using acceptability judgements, eye-tracking data, and neu-ral response measurements @xcite @xcite @xcite @xcite @xcite . psycholinguistic research has, however, also highlighted the robustness of sentence processing mechanisms to a variety of perturbations, including those which violate word order restrictions @xcite @xcite . in recent work, @xcite tested the hypothesis that composition is the core function of the brain's languageselective network and that it can take place even when grammatical word order constrains are violated. their findings confirmed this, showing that stimuli with shuffled word order where local dependencies were preserved -as is, roughly speaking, the case for many dependencies in the sentences shuf.n4 is trained on -elicited a neural response in the language network that is comparable to that elicited by normal sentences. when interword dependencies were disrupted so combinable words were so far apart that composition among nearby words was highly unlikely -as in shuf.n1, neural response fell to a level compared to unconnected word lists. in machines recently, many nlp researchers have attempted to investigate the role of word order information in language models. for example, @xcite employ diagnostic classifiers and attention analyses to demonstrate that lower (but not higher) layers of bert encode word order information. @xcite find that multilingual bert is sensitive to morphosyntactic alignment, where numerous languages (out of 24 total) rely on word order to mark subjecthood (english among them). @xcite implement an input perturbation framework (n-gram shuffling, phrase swaps, etc.), and employ it towards testing the sensitivity of bert's representations to various types of structure in sentences. they report a sensitivity to larger constituent units of sentences in higher layers, which they deduce to be influenced by hierarchical phrase structure. o'connor and andreas (2021) examine the contribution of various contextual features to the ability of gpt-2 @xcite to predict upcoming tokens. their findings show that several destructive manipulations, including in-sentence word shuffling, applied to mid-and long range contexts lead only to a modest increase in usable information as defined according to the v-information framework of @xcite . similarly, word order information has been found not to be essential for various nlu tasks and datasets. early work showed that natural language inference tasks are largely insensitive to permutations of word order @xcite . @xcite and @xcite discuss this in greater detail, demonstrating that test-time word order perturbations applied to glue benchmark tasks have little impact on lm performance. following up on this, @xcite , which our work builds on, found that pretraining on scrambled text appears to only marginally affect model performance. most related to this study, @xcite introduce two metrics for gauging the local and global ordering of tokens in scrambled texts, observing that only the latter is altered by the perturbation functions found in prior literature. in experiments with glue, they find that local (sub-word) perturbations show a substantially stronger performance decay compared to global ones. in this work, we present an in-depth analysis of these results, showing that lms trained on scrambled text can actually retain word information and that -as for humans -their sensitivity to word order is dependent on a variety of factors such as the nature of the task and the locality of perturbation. while performance on some "understanding" evaluation tasks is not strongly affected by word order scrambling, the effect on others such as the winograd schema is far more evident. ## conclusion much discussion has resulted from recent work showing that scrambling text at different stages of testing or training does not drastically alter the performance of language models on nlu tasks. in this work, we presented analyses painting a more nuanced picture of such findings. primarily, we demonstrate that, as far as altered pre-training is concerned, models still do retain a semblance of word order knowledge -largely at the local level. we show that this knowledge stems from cues in the altered data, such as adjacent bpe symbols and correlations between sentence length and content. the order in which shuffling is performed -before or after bpe tokenization -is influential in models' acquisition of word order, which calls for caution in interpreting previous results. finally, we show that there exist nlu tasks that are far more sensitive to sentence structure as expressed by word order. figure 9: ∆ uas, all models and layers across dependency lengths 1-5+, w.r.t. orig. layer 13 represents a linear mix of all model layers.