Word Order Matters When You Increase Masking (2022)

Abstract: Word order, an essential property of natural languages, is injected in Transformer-based neural language models using position encoding. However, recent experiments have shown that explicit position encoding is not always useful, since some models without such feature managed to achieve state-of-the art performance on some tasks. To understand better this phenomenon, we examine the effect of removing position encodings on the pre-training objective itself (i.e., masked language modelling), to test whether models can reconstruct position information from co-occurrences alone. We do so by controlling the amount of masked tokens in the input sentence, as a proxy to affect the importance of position information for the task. We find that the necessity of position information increases with the amount of masking, and that masked language models without position encodings are not able to reconstruct this information on the task. These findings point towards a direct relationship between the amount of masking and the ability of Transformers to capture order-sensitive aspects of language using position encoding.

## introduction transformer-based language models have become ubiquitous since they demonstrated improvements in most nlp downstream tasks @xcite . a lot of ink has been spilled regarding the amount of linguistic structure that such models captured @xcite , pointing towards the acquisition of diverse linguistic abilities. as neural language models need to process information about the position of their input tokens to capture structural generalizations, a plethora of proposals to encode such information have been made @xcite @xcite @xcite . recent work, however, questioned whether word order information is really useful for pre-trained models to solve downstream tasks @xcite , showing that models could perform well when using only higher-order co-occurrence statistics. other authors @xcite have shown that some transformers could reconstruct partly position information without it being explicitly injected. examining performance on downstream tasks can show that the task simply does not require order information, or that the dataset used to test the model is too easy @xcite , leading to indirect observations regarding a model's ability to reconstruct position information. in turn, we choose to test the importance of position encodings for the pre-training task itself, masked language modeling, to get more direct evidence about whether and when position matters to language models. we do so under different amounts of masking, as intuitively, position information should be increasingly important when more tokens are missing from the context. our experiments show that when masking only one token, the absence of position encoding has little effect on the model's performance. however, its importance increases with the number of masked tokens, forcing the model to leverage position information to perform better on its training objective. this finding should draw our attention towards choosing more carefully the amount of masking to train masked language models -a choice as important as the position encoding scheme itself. ## related work a recent line of research investigated the importance of word order information during pre-training for models to solve downstream tasks, showing little variations when their input sentences are shuffled @xcite @xcite . in a similar line of research, @xcite found that even in absence of position encoding, models were still able to reconstruct the latter when probed for tokens' absolute position information in their intermediate layers. this finding in turn questioned the need for injecting explicitly position information in language models. @xcite also showed that shuffled models were still able to capture position information even when information about word order was removed after subword segmentation, likely because of the dependency between unigram occurrence probability and sentence length. given all this work, it is surprising that the importance of explicit word order information in a neural language model still eludes us. in this study, we choose to investigate more carefully this phenomenon, and propose a methodology carefully designed to evaluate the importance of position encoding for the pre-training objective. ## results we first display the kl-divergence between true probability distributions assuming ordered and unordered contexts in fig. 2 . in accordance with our expectations, foot_7 when increasing the amount of masking, the true distribution of completions given ordered contexts diverges from that of unordered contexts. interestingly though, when only one token is masked, the divergence is low. this suggests that in this setting, models should have little difference regardless of whether they have access to explicit position information. by increasing the amount of masked tokens, we can further observe that the two considered true probabilities p o and p u diverge. we thus expect that models should increasingly rely on position information to approximate the true ordered distribution. we further display how well each model approximates each probability estimate in fig. 3 to verify whether the presence of position encoding is useful to the masked language modeling task under different amounts of masking. expectedly, the model with no position encoding scheme performs similarly to the bert model when only one token is masked. in this setting, the context contains enough information for the model regardless of whether it sees its input tokens as ordered or as a bag of words. when masking more tokens however, this difference becomes increasingly marked. 9 further, we observe that the bert model has a low divergence to the true probability assuming ordered contexts regardless of the amount of masking, while it diverges increasingly from the distribution that assumes no ordering of the context. the opposite pattern holds for the np model. taken together, these results show that position encoding is necessary to approximate the true distribution of the task when it requires position information, that is when the number of masked tokens is increased. in fig. 4 , we compare our models' crossentropies to the task's true entropies. the figure aggregates the two main observations made in this article, that when the number of masked tokens increases : (i) the true entropy of the data with and without position diverge from each other, and (ii) that position encoding is required to approximate the task's true probability distribution assuming ordered contexts. accordingly to our previous ob-servations, the np model, which does not have access to the ordering of tokens, has a cross-entropy that fits the true probability distribution's entropy assuming no ordering of the context (red lines). looking at bert's cross-entropy, we see that this model, which has access to position information, rather fits the true probability distribution assuming the context is ordered. 5 discussion 5.1 position encoding and language modeling previous work claimed that transformer autoregressive language models without position encodings could reconstruct position information by inferring the number of preceding tokens, but not bidirectional transformer models @xcite . testing a roberta model @xcite led to great difference in perplexity when removing position information at the input level. however, we show this difference to strongly depend on the amount of masking : as autoregressive language models predict only one token at a time, the task could be equally easy for models deprived from position information. our results call for increased scrutiny when comparing autoregressive and masked language models, making sure that they are asked to predict comparable numbers of tokens. ## conclusion in this work, we evaluated the importance of position encoding for a masked language model. we showed that without explicit access to position information, a model can obtain performance similar to a model that learns position embeddings, when only one token is masked. we find that when increasing the number of masked tokens, the output probability distribution assuming unordered inputs diverges from that which assumes ordered sentences, reflecting that the task increasingly requires making use of position information. we further show that under this condition, models with position encoding outperform their counterpart deprived from position information. this in turn could draw more attention to the amount of masked tokens, which might be a crucial parameter for models to abstract away from their input sentences' position information, in addition to the chosen position encoding scheme.