On the utility of enhancing BERT syntactic bias with Token Reordering Pretraining (2023)

Abstract: Self-supervised Language Modelling (LM) objectives —like BERT masked LM— have become the default choice for pretraining language models. TOken Reordering (TOR) pretraining objectives, beyond token prediction, have not been extensively studied yet. In this work, we explore challenges that underlie the development and usefulness of such objectives on downstream language tasks. In particular, we design a novel TOR pretraining objective which predicts whether two tokens are adjacent or not given a partial bag-of-tokens input. In addition, we investigate the usefulness of Graph Isomorphism Network (GIN), when placed on top of the BERT encoder, in order to enhance the overall model ability to leverage topological signal from the encoded representations. We compare language understanding abilities of TOR to the one of MLM on word-order sensitive (e.g. Dependency Parsing) and insensitive (e.g. text classification) tasks in both full training and few-shot settings. Our results indicate that TOR is competitive to MLM on the GLUE language understanding benchmark, and slightly superior on syntax-dependent datasets, especially in the few-shot setting.

## introduction pretraining with self-supervised language modelling objectives @xcite @xcite @xcite has become indispensable for stateof-the-art performances on natural language understanding (nlu) benchmarks @xcite @xcite . identifying the mechanisms those models use for * equal contribution. listing order is random † work done while at huawei noah's ark lab. ♠ corresponding author. 1 the term point to objectives that project the last layer representation to vocabulary space in order to output tokens (e.g. mlm, casual lm, or the one of t5). task solving gained prominence @xcite @xcite . such works attempted to shed light on whether pretrained language models (plms) @xcite @xcite learn to encode language through appropriate inductive biases that align with the human understanding of syntax in languages. models not demonstrating this behavior suggest that existing pretraining objectives (like mlm @xcite and its variants) may not be sufficient at encoding the essential aspects of syntax that potentially guide language understanding @xcite . and p @xcite indicate a randomly masked token and position respectively, while transparent targets are ignored during loss calculation. the target of tor is a matrix that point to neighbor token at distance k (+1 in this example). order of tokens being an essential artifact to capture syntactic cues, we propose token reordering (tor), a novel self-supervised task that boosts the awareness to word-order in models. figure 1 shows the difference between mlm @xcite and tor objectives, where in pretraining with mlm some input tokens are masked and the model is tasked with predicting the masked tokens. in tor, token-order information is removed 2 from the input sequence, and a model is tasked to predict the neighbor token-to-token positional relations. we further investigate the utility of a novel structure-aware architecture that consists in end-toend pretraining of a graph isomorphism network (gin) model @xcite placed on top of the bert encoder @xcite . as some nlu tasks may not always require strong syntactic understanding @xcite @xcite , we conduct a thorough empirical analysis on both word-order insensitive tasks from the glue @xcite benchmark, as well as syntax-sensitive ones, namely dependency parsing (dp) @xcite . our study shows that learning representations with an order reconstruction objective is highly effective only when the input sequence is partially (compared to fully) shuffled. second, pretraining with tor leads to competitive performances on order insensitive tasks compared with mlm, and superior performance on order sensitive ones especially in the few-shot setting. third, bert trained with tor shows better sensitivity to absence of word-order information than bert-mlm, thereby being a potential method to alleviate some of the concerns raised on plm's syntax understanding. yet, we find that with enough labelled data, tor have hardly any additional value, which is consistent with other task-specific objectives @xcite . ## related work language modelling objectives such as bert's masked language modelling @xcite , xl-net's permutation language modelling @xcite , gpt next word prediction @xcite , as well as auto-regressive sequence denoising ones of bart @xcite and mass @xcite are popular self-supervised representation learning routines used in nlu tasks. learning contextual word representations is grounded in linguistics @xcite and psycholinguistics @xcite literature that supports that the natural order of words helps humans better capturing semantic information. @xcite in their studies with humans found that local ordering of words 2 through the removal of spatial (positional) information. despite large data and sophisticated inductive biases, plms seem to not quite understand the language like humans do (o' @xcite . recent studies @xcite @xcite show that large language models are insensitive to word-order. these works measure the sensitivity of plms to task performance when a language model is pretrained @xcite or fine-tuned @xcite ) with text sequences with deleted or shuffled tokens. notably, @xcite demonstrate that plms are insensitive to word-order information suggesting further that language modeling objectives alone may not be sufficient to encode the essential aspects of syntactic abstraction of language understanding. exploring alternative pretraining objectives, such as linguistically (e.g. character, part of speech) informed @xcite , task-specific (e.g. question answering) @xcite , and word-order aware ones @xcite has been gaining attention lately. with that, exploring inductive biases that better capture such objectives too has been gaining attention. among such inductive biases, graph neural network (gnn) @xcite has become popular due to their conventional use of structure prediction tasks that involve entities and relations, which also aligns with syntactic tasks such as parsing @xcite , ordering or tagging @xcite . also, @xcite use gnns in pretraining language models for the question answering task. the proposed tor objective is different along two major aspects when compared with its relevant counterparts. first, it uses a partial bag-of-words representation of input sequence compared to full (t5 @xcite deshuffling objective) or trigram window (structbert @xcite word structural objective) tokens shuffling. second, tor uses a pairwise token-to-token relation to represent the output target, compared to projecting hidden representations to the token vocabulary space unlike deshuffling and word structural. further, using the tokens in the input to re-order instead of predicting over the entire vocabulary provides significant computational gains over the other ob-jectives; with tor, we could fit a batch size which is 33% larger than token prediction objectives like mlm. ## tor we formulate a new pre-training task for selfsupervised representation learning for nlu by proposing tor, a token reordering objective. we describe the input representations and target design in §3.1 and §3.2 respectively, and the main details of our proposed bert+gin model and the motivations behind it in §3.3. ## conclusion we revisit word-order pretraining for nlu by proposing a novel self-supervision task (tor), as well as a dedicated encoder architecture. the goal is to investigate if injecting syntactic biases into plm during pretraining would improves their awareness to language structure. while experiments on tor show promises in enhancing plm understanding of language structure, still many challenges remain in maintaining performances on word order insensitive tasks. we thereby highlight the importance of word-order pretraining objectives as an interesting research direction to explore in future.