Better Pre-Training by Reducing Representation Confusion (2023)

Abstract: In this work, we revisit the Transformer-based pre-trained language models and identify two different types of information confusion in position encoding and model representations, respectively. Firstly, we show that in the relative position encoding, the joint modeling about relative distances and directions brings confusion between two heterogeneous information. It may make the model unable to capture the associative semantics of the same distance and the opposite directions, which in turn affects the performance of downstream tasks. Secondly, we notice the BERT with Mask Language Modeling (MLM) pre-training objective outputs similar token representations (last hidden states of different tokens) and head representations (attention weightsof different heads), which may make the diversity of information expressed by different tokens and heads limited. Motivated by the above investigation, we propose two novel techniques to improve pre-trained language models: Decoupled Directional Relative Position (DDRP) encoding and MTH pre-training objective. DDRP decouples the relative distance features and the directional features in classical relative position encoding. MTH applies two novel auxiliary regularizers besides MLM to enlarge the dissimilarities between (a) last hidden states of different tokens, and (b) attention weights of different heads. These designs allow the model to capture different categories of information more clearly, as a way to alleviate information confusion in representation learning for better optimization. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of our proposed methods.

## introduction the paradigm of pre-training on large-scale corpus and fine-tuning on specific task datasets has swept the entire field of natural language processing (nlp). bert @xcite is the most prominent pre-trained language model, which stacks the encoder blocks of transformer @xcite and adopts mlm and next sentence prediction (nsp) pre-training tasks, achieving the sota results in 2018. after that, a large number of pre-trained language models (plms) @xcite @xcite @xcite that optimize the transformer structure and pre-training objectives have emerged, which further improves the performance of the pre-trained language models on multiple downstream tasks. in this work, we identify two different types of information confusion in language pre-training, and explore two conceptually simple and empirically powerful techniques against them as follows: extensive experiments on the glue benchmark show that ddrp achieves better results than classi-cal relative position encoding @xcite on almost all tasks without introducing the additional computational overhead and consistently outperforms prior competitive relative position encoding models @xcite . moreover, our proposed mth outperforms mlm by a 0.96 average glue score and achieves nearly 2x pre-training speedup on bert base . both ddrp and mth are straightforward, effective, and easy to deploy, which can be easily combined with existing pre-training objectives and various model structures. our contributions are summarized as follows: ## related work in recent years, pre-trained language models have made significant breakthroughs in the field of nlp. bert @xcite , which proposes mlm and nsp pre-training objectives, is pre-trained on large-scale unlabeled corpus and has learned bidirectional representations efficiently. after that, many different pre-trained models are produced, which further improve the effectiveness of the pretrained models. roberta @xcite proposes to remove the nsp task and verifies through experiments that more training steps and larger batches can effectively improve the performance of the downstream tasks. albert @xcite proposes a cross-layer parameter sharing technique to lower memory consumption. xl-net @xcite proposes permutation language modeling to capture the dependencies among predicted tokens. electra @xcite adopts replaced token detection (rtd) objective, which considers the loss of all tokens instead of a subset. tupe @xcite performers query-key dot product with different parameter projections for contextual information and positional information separately and then added them up, they also add relative position biases like t5 @xcite on different heads to form the final correlation matrix. deberta @xcite separately encodes the context and position information of each token and uses the textual and positional disentangled matrices of the words to calculate the correlation matrix. ## method in this section, we analyze in turn two different types of information confusion that exist in the real plms: (i) the paradigm of utilizing a single parametric vector of relative position embedding to represent both relative distance and direction. ## conclusion in this work, we analyze and identify potential information confusion in the relative position encoding and model representations, respectively, and design two novel techniques to address these problems: ddrp (decoupled directional relative position) encoding and mth (mlm with tcd and hcd) pre-training objectives. specifically, ddrp decouples relative distance features and directional features to eliminate unnecessary randomness in the self-attention module. mth utilize tcd and hcd as two regularizers to supervise the model to always maintain a certain level of critical thinking. the experimental results show that ddrp achieves better performance compared with various relative position encoding models and mth outperforms mlm by a large margin. we believe that reducing information confusion in representation learning may have broader application scenarios, and leave this area of exploration for future work.