Shuffled-token Detection for Refining Pre-trained R o BERT a (2021)

Abstract: State-of-the-art transformer models have achieved robust performance on a variety of NLP tasks. Many of these approaches have employed domain agnostic pre-training tasks to train models that yield highly generalized sentence representations that can be fine-tuned for specific downstream tasks. We propose refining a pre-trained NLP model using the objective of detecting shuffled tokens. We use a sequential approach by starting with the pre-trained RoBERTa model and training it using our approach. Applying random shuffling strategy on the word-level, we found that our approach enables the RoBERTa model achieve better performance on 4 out of 7 GLUE tasks. Our results indicate that learning to detect shuffled tokens is a promising approach to learn more coherent sentence representations.

## introduction the method of pre-training natural language models has been shown to greatly improve model performance on a wide range of nlp tasks @xcite @xcite . state-of-the-art models that utilize transformers and deep bi-directional representations of text such as bert, roberta, and albert @xcite @xcite have achieved superior results by pre-training on general, large corpora to learn rich representations from unlabeled data. particularly helpful in low training data resource scenarios, unsupervised pretraining has become the first step for many language models to build powerful linguistic representations before fine tuning for downstream target tasks. bert style models use masked language modeling (mlm) and sometimes next sentence prediction, as pre-training tasks. while these tasks have been shown to produce transferable sentence representations for many nlp tasks, using additional domain-agnostic pre-training tasks such as sentence shuffling may improve model performance. in a seminal cognitive psychology study it has been demonstrated that humans have a well trained ability to parse shuffled sentences @xcite . moreover, it has been shown that pre-trained models sometimes overlook word order while making predictions @xcite , and encouraging models to capture word order improves the classification performance. shuffling as a pre-training task may therefore help expand transformer models to achieve even better performance on nlp tasks. drawing inspiration from recent work in reconstructing shuffled text @xcite , we propose that pre-training the roberta model with a token modification discrimination head on randomly shuffled sentences provides constructive learning objective, which helps the model learn coherent representations and facilitate model recognition of the key pieces of a sentence and their association. to substantiate the argument, we design experiments to examine the model performance of roberta with the proposed approach. the results demonstrate that pre-training the model with shuffled sentences enhances the scores of a majority of glue tasks. ## related work shuffling sentences and words has often been used as a downstream task to evaluate model performance. one relevant example is the work by @xcite to develop a semi-character rnn model that surpasses previous spell-check methodologies on the cmadbrigde uinervtisy effect, where humans can easily reconstruct the shuffled token. @xcite explored the performance of bert on a shuffled sentence downstream task and highlighted some induced bias in the model that is the cause of incorrect predictions for noisy inputs. while the authors propose removing the induced bias from the representations to improve results, they do not consider the possibility of pre-training the model with shuffled sentences. the use of un-ordered or noisy data in model training itself has proven effective. a number of studies have focused on using shuffled input to create useful sentence representation vectors for language models. @xcite developed the skip-thoughts method to accomplish the task of reconstructing sentence order from a shuffled input. the authors used an encoder-decoder rnn model at the sentence level that allows a sentence to predict the adjacent sentences. @xcite explored how sentence ordering tasks can help models learn text coherence. using an rnn based approach, they train models to identify the correct ordering of sentences and show that models learn both document structure and useful sentence representations during this task. @xcite employed discourse based learning objectives to help models understand discourse coherence. specifically, given some sentences, they ask the model to predict if the sentences are in order, or if one sentence comes next to a set of sentences, or to predict the conjunction that joins the sentences. they showed that using these objectives to train models achieves significant reduction in computational training costs and is also effective when using unlabeled data. there are a number of papers that focus on wordlevel shuffling, as opposed to sentence-level shuffling. @xcite developed the sequential denoising autoencoder (sdae) method, where a sentence is corrupted using a noise function determined by free parameters. after a certain percentage of words have been corrupted, an lstm encoder-decoder model is tasked with predicting the original sentence from the corrupted version. the authors demonstrate training with noisy inputs allowed sdae to significantly outperform regular sae models, which did not introduce word-level-noise factors. one closely related paper in the field of computer vision leverages the use of shuffled input in model training. @xcite employ a cnn model that is trained to solve jigsaw puzzles to determine correct spatial representation. their results show that using shuffled input helps models learn that images are made up of different parts, and their relationship to the whole. finally, a variety of studies demonstrate that further pre-training performed after the general purpose bert pre-training leads to better model results instead of simply fine-tuning downstream. domain specific pre-training, such as biobert @xcite , story ending prediction by trans-bert @xcite , and video caption classification by videobert @xcite are all examples where expanding the pre-training tasks for bert has achieved enhancement in model performance. transbert in particular demonstrates that further pre-training using targeted supervised tasks achieves better results than relying only on the unsupervised pre-training in bert. ## methodology consider a sequence of tokens x. we first obtain x shuffled from x by shuffling a set of tokens of x. given x shuffled , we detect if tokens are shuffled or not by using a token modification discrimination head on top of the roberta base model. our choice of the discriminative head is motivated by the recent success of electra @xcite . ## conclusion and future work in this paper, we examine the performance of roberta model with token modification discrimination head on detecting randomly shuffled tokens. we have demonstrated that detecting shuffled tokens is indeed a challenging yet advantageous task, which allows the model to learn coherent representations of the sentences. in this work, we start with pre-trained roberta base model and train it further on the shuffled token detection task. for future work, the model can be further explored by expanding the shuffling strategy. one possible strategy is part of speech (pos) shuffling, which randomly permutes specific pos tokens such as nouns or verbs. instead of detecting shuffled tokens, another objective would be to predict the original positions of the shuffled tokens. yet another objective that can be explored is combining our proposed loss with the masked language modeling loss. we would also like to study our approach when applied to other pre-trained models such as albert and electra.