How effective is BERT without word ordering? Implications for language understanding and data privacy (2021)

Abstract: Ordered word sequences contain the rich structures that define language. However, it’s often not clear if or how modern pretrained language models utilize these structures. We show that the token representations and self-attention activations within BERT are surprisingly resilient to shuffling the order of input tokens, and that for several GLUE language understanding tasks, shuffling only minimally degrades performance, e.g., by 4% for QNLI. While bleak from the perspective of language understanding, our results have positive implications for cases where copyright or ethics necessitates the consideration of bag-of-words data (vs. full documents). We simulate such a scenario for three sensitive classification tasks, demonstrating minimal performance degradation vs. releasing full language sequences.

## introduction masked language models (mlms) like bert @xcite use an ordered sequence of tokens as input. and rightfully so! any model capable of "language understanding" undoubtedly should need access to the hierarchical, syntactic structures implicitly encoded in language. but are mlms really doing better because they have access to full word sequences? to assess this question, we first compare the internal representations of bert and roberta @xcite when the sequence of unigrams is not available. 1 we do this by using the bagof-words counts of an input to generate a random ordering of the unigrams, i.e., "shuffling" the input. for example, in a sentiment classification corpus, if an intact input was "the movie was great!", a possible shuffled ordering might be "movie the great was" (tokenization details are in §4). we find that, though bert appears to become more sensitive to ordering in later layers, shuffled token representations and self-attention activations still closely resemble their unshuffled counterparts. following cues from prior work @xcite @xcite , we next report the performance of pre-trained mlms fine-tuned on glue, a suite of english-language understanding benchmarks, when given access only to unigram count information by handing models randomly ordered sequences of words (an approach we call bow-bert, for short). for most glue tasks, performance degradation when shuffling is minimal, e.g., mnli, qqp, and qnli accuracy degrade by less than 5 accuracy points. the bad news: despite bert being trained on intact word sequences, bow-bert demonstrates that mlms can readily ignore syntax (while maintaining strong performance) when fine-tuned for even carefully designed downstream language understanding tasks. 2 we thus advocate for reporting bow-bert's performance as a strong baseline. the good news: bow-bert offers a practical modeling choice for researchers who must operate with only bag-of-words representations for legal or ethical reasons. 3 bag-of-words data releases are sometimes the only legal format in which copyright-sensitive corpora may be distributed, e.g., hathitrust foot_3 (16m historical volumes) @xcite , google n-grams @xcite , etc. and while ethical considerations sometimes preclude the full release of privacy-sensitive docu-ments (e.g., medical transcriptions), bag-of-words data release offers the potential for compromise. while releasing unigram counts is one way of anonymizing documents @xcite , recent work in differential privacy @xcite @xcite has resulted in randomized algorithms capable of privatizing bow count data (under varying definitions of privacy). 5 we explore classification tasks on three sensitive corpora, simulating different input fidelity availability: full sequences, bow counts, and differentially private (dp) bow counts. we find that bow-bert often significantly outperforms prior bow models, especially for shorter documents. and, for longer documents, bow-bert can even outperform fullsequence bert. finally, for the (naive) dp configuration we consider, bow-bert is a viable option for classifying shorter privatized documents, though linear bow models remain competitive for longer documents. word order shuffling has been tested as part of the full training process for non-pretrained models. @xcite shuffle words in a dialog corpus, and find that lstms are more sensitive than transformers to word order. @xcite show that shuffling distant context words (e.g., beyond 50 tokens) has little effect in outcome for lm-lstms. @xcite show that lstm autoencoders encode significant ordering information when fit to a corpus of wikipedia sentences. @xcite report minimal performance decreases from word shuffling while training a number of model architectures, e.g., esim @xcite , for snli/mnli tasks. in a multimodal setting, @xcite show that shuffling doesn't affect performance for an lstm in a referring expression task. shuffling inputs to pretrained mlms. while at the time of submission of this work, shuffling results had not been fully reported on the popular glue taskset, prior results have used wordshuffling as a baseline with varying results. @xcite operationalize ablations of reading comprehension skills from @xcite , and report that shuffling n-grams in 10 qa corpora results in 10-20% performance decreases for bert. @xcite report similar results when shuffling questions+answers in mcrc corpora, reporting absolute accuracy drops of between 5-20% when shuffling both passage/question words (e.g., bert on dream drops from 63 → 41 accuracy relative to a 33% constant baseline). k et al. (2020) report that swapping tokens during pretraining of a multilingual bert model results in moderate performance degradation for xnli (e.g., 71 → 63 for en-es) but more significant performance degradation for ner (63 → 40 in the same setting). they find that a purely frequency-based corpus "is not enough for a reasonable cross-lingual performance." several works have examined shuffling inputs in multi-language scenarios (e.g., translation) when languages have variable syntax @xcite . @xcite use a random token permutation to provide a baseline. @xcite find that self-attention networks are surprisingly bad at identifying two tokens that are swapped in the input. @xcite show that shuffling bert inputs decreases word cloze prediction performance on a corpus of 102 sentences without fine-tuning. @xcite incorporate a deshuffling objective into pre-training. in some cases, shuffled inputs provide a stronger baseline than might be assumed, while in others, shuffling significantly degrades performance. at present, determining whether or not order is "needed" for a particular task is largely an experimental, empirical endeavor. ## representation analysis we might expect that shuffling the order of tokens in an input sentence would significantly corrupt the internal representations of bert, but is that actually the case? we investigate with two new metrics. consider applying a pre-trained, fixed bert model to @xmath0. token identifiability measures the similarity of bert's vector representations of a word token (e.g., "movie") in x and x . identifiability is high if the model has similar representations for tokens after their order is shuffled. self-attention distance measures if bert attends to similar tokens for each token in x and x regardless of their order (e.g., is "the movie was great" ≈ "movie the great was" to bert?). self-attention distance is low if the model attends to the same tokens after input shuffling. token identifiability. let mlm l (x) be a r t×d matrix, where t is the number of tokens in sentence x, d is the mlm's dimension, and l is the layer index. in this setting, row i of mlm l (x) is the mlm's representation of the ith token in sentence x. we compare mlm l (x) to e[mlm l (x )], where x is drawn uniformly from the permutations of x: perm(x). for a specific sample x ∼ perm(x), we first take the row-wise cosine similarity of mlm l (x) and mlm l (x ), and treat the resulting t × t matrix as an instance of a bipartite linear assignment problem. the assignment accuracy (aa) score for (x, x ) is the proportion of assigned token pairs that have the same underlying word type. to avoid biasing towards shorter sentences, we take the ratio of the accuracy relative to chance, i.e., self-attention distance. let amlm l,h (x) be the row-l 1 -normalized r t×t matrix representing the self-attention matrix at layer l for attention head h. we can compute the same matrix for a shuffled input amlm l,h (x ), and then perform a transformation to re-order the rows and columns of this matrix to match the original order of tokens in x, yielding amlm x l,h (x ). we then define the row-wise jensen-shannon divergence ds-jsd(amlm l,h (x), amlm x l,h (x )) as the mean row-wise jsd between amlm l,h (x) and the deshuffled reordered attention matrix amlm x l,h (x ). as before, to reduce the effect of sentence length, we normalize using rnd-jsd(amlm l,h (x), amlm x l,h (x )), which chooses a random row/column permutation. 7 the final attention distance metric is defined as results. we randomly sample 100 sentences from each training set of 8 glue tasks, for a total of 800 sentences. to approximate expectations from equations 1 and 2, we sample 32 random permutations per sentence. figure 1 gives the per-layer token identifiability/attention similarity scores for both mlms. for both metrics, later layers are more order sensitive to order, i.e., id-mlm ↓ and ad-mlm ↑. attention heads vary significantly in their order sensitivity: each attention head is a single point in the scatterplot of figure 1b . but, even at late layers, both metrics suggest significantly more than random correspondence: internal representations of bow-(ro)bert(a)clearly resemble their unshuffled counterparts. ## bow-bert for classification we compare bert and roberta to their bow counterparts on nine tasks from glue @xcite . 8 we run single-task training for six epochs, use early stopping, and optimize batch size ({16, 32}) and learning rate ({5, 2, 1, .5} × 10 -5 ) via grid search on the validation set. to shuffle documents: we lowercase, tokenize, remove all tokens that consist only of punctuation, shuffle, then concatenate with whitespaces. we re-shuffle the training tokens each epoch, but fix validation and test tokens to one shuffled permutation. 8 these tasks span nli (mnli @xcite , qnli @xcite , and rte @xcite @xcite ); semantic similarity estimation (qqp, 9 mrpc @xcite , sts-b @xcite ); sentiment analysis (sst-2 @xcite ), and grammaticality judgement (cola @xcite ). we omit wnli @xcite as is common (all models achieve chance performance on that corpus). results. according to the glue diagnostic set (which tests 33 categories of linguistic phenomena) bow-bert has the most trouble with dealing with double negations (e.g., "i have never seen a hummingbird not flying.": mcc degrades 31.7 → -4.3 when switching @xmath1, quantifiers ("our sympathy to all [vs. some] of the victims": 61.8 → 46.1); and temporal logic ("mary left before john entered": 8.0 → -8.6). results for glue diagnostic meta-categories are: knowledge (24.4 → 24.3); pred-arg structure (39.2 → 39.1); logic (24.7 → 22.1); lexical semantics (39.7 → 31.5). ## conclusion and future work we advocate for bow-(ro)bert(a)as a surprisingly strong baseline for language understanding tasks, as well as a performant practical option for 11 our original submission used dp pca instead. but it was brought to our attention that the paper proposing that algorithm was retracted for being non-private (+ discontinued in the library we used after we submitted). we have adjusted our code and recompiled our experiments using a comparable mechanism. our intent isn't to advocate for this particular dp method, but rather, to fairly compare nlp algorithms on the same dp corpora.