D ec BERT : Enhancing the Language Understanding of BERT with Causal Attention Masks (2022)

Abstract: Since 2017, the Transformer-based models play critical roles in various downstream Natural Language Processing tasks. However, a common limitation of the attention mechanism utilized in Transformer Encoder is that it cannot automatically capture the information of word order, so explicit position embeddings are generally required to be fed into the target model. In contrast, Transformer Decoder with the causal attention masks is naturally sensitive to the word order. In this work, we focus on improving the position encoding ability of BERT with the causal attention masks. Furthermore, we propose a new pre-trained language model DecBERT and evaluate it on the GLUE benchmark. Experimental results show that (1) the causal attention mask is effective for BERT on the language understanding tasks; (2) our DecBERT model without position embeddings achieve comparable performance on the GLUE benchmark; and (3) our modification accelerates the pre-training process and DecBERT w/ PE achieves better overall performance than the baseline systems when pre-training with the same amount of computational resources.

## introduction in recent years, transformer model proposed by @xcite has supplanted the widelyused lstm @xcite as an indispensable component of many nlp systems. there are two branches of model variant: transformer encoder and transformer decoder. the encoder-based language models, e.g., bert @xcite , roberta @xcite and deberta @xcite , have achieved great success on many natural language understanding benchmarks (e.g. glue @xcite and superglue @xcite ). the decoder-based language models such as gptfamily @xcite @xcite have shown superior performances on natural language generation. all of them utilize the multi-head self-attention (mha) mechanism @xcite . since mha is designed as an order-invariant mechanism @xcite , transformer encoder without the help of position embeddings should share the same intuitions with the bag-of-word model. on the other hand, in transformer decoder, the causal attention masks make the mha different from that of the transformer encoder. specifically, @xcite have proved that mha with such attention masks is not permutation equivalent, indicating that transformer decoder is sensitive to word order. it is noticed that several studies focus on enriching the position information of bert to improve the performance of natural language understanding @xcite @xcite @xcite , e.g., introducing extra learnable parameters to trace the word order. previous analysis also indicate that the lower layers of bert tend to capture rich surface-level language structural information such as position information @xcite . in this paper, to improve the language understanding of bert, we propose to enrich the position information in the lower hidden layers instead of introducing extra learnable positional parameters. to this end, we firstly design analysis experiments to examine the effectiveness of causal attention masks in terms of capturing position information. then we propose a new pre-trained language model decbert by adding the causal attention masks into the lower layers of bert (e.g., the first two layers) to enhance the position encoding ability. in this way, our proposed model is naturally sensitive to word order. then we pre-train our decbert as a masked language model, following the same objective as bert. to verify whether our modification can help bert trace word order, we also make a comparison with a variant of our decbert that excludes any position embeddings. the experimental results show that decbert w/o pe has 77 times (4.59 vs. 353.97) lower valid ppl score than bert w/o pe and achieves comparable performance with bert w/ pe on downstream tasks, corroborating the effectiveness of our modification. furthermore, decbert w/ pe achieves better performances than bert on most downstream tasks when pre-training with the same amount of time and computational resources. by analyzing the pre-training process, we find that our modification can also accelerate pre-training. the contributions of this work are summarised as follows: ## background: transformer transformer is a neural network model proposed by @xcite , which relies on the multi-head self-attention (mha) mechanism. input layer. due to the order-invariance of mha @xcite , a token embedding is added with a position embedding as the input of transformer encoder or decoder: in the later work, @xcite choose to use a learnable p e matrix. ## methodology in this section, we first analyze the relationship between transformer decoder and position embeddings (section 3.1). based on this analysis, we inject the causal attention masks into bert to create our new pre-trained language models, decbert (section 3.2). ## related work the previous works @xcite @xcite @xcite indicate that the self-attention mechanism of transformer encoder is permutation equivalent, so it needs to use the position embedding. @xcite have proved that decoder's self-attention is not permutation equivalent, indicating that decoder is not a bag-of-word model as encoder, but they do not conduct further analysis on decoder's position encoding ability. apart from the analysis, @xcite train the transformer language models with speech dataset. they find that models without position embeddings have lower perplexity scores. @xcite introduce a new linear transformer language model with fast weight memories @xcite , which has lower perplexity without position encodings on the wikitext-103 dataset. furthermore, an explosion of work focuses on proposing a better method to add the position information into the pre-trained language model. @xcite give a comprehensive introduction of different position encodings methods of transformer. they divide position encodings into three approaches. one line of such work is to add position embeddings to the input before it is fed to the actual transformer model @xcite @xcite @xcite @xcite . the second line of work directly modify the attention matrix @xcite @xcite @xcite . the last one combine the first two approaches together. however, all of them focus on introducing an extra set of parameters to trace the word order. our work chooses to make use of the causal attention masks. most similar to our modification in section 3.2, im and cho (2017) propose a self-attention based model which achieve better performance on snli task @xcite without the help of explicit position encodings. however, their models are different from the standard transformer and use extra local attention masks to control the information flow. with the popularity of the transformer model in the computer vision field, some works propose different methods to make vision transformer know word order implicitly @xcite @xcite , but all of them modify the models with convolution neural network @xcite . ## conclusion in this work, we introduce a new pre-trained model, called decbert, adopting the causal attention masks to enhance the language understanding of bert. we conduct a series of experiments to verify the effectiveness of our models. experimental results indicate that our proposed models achieve better performance than bert on most downstream tasks when pre-training with the same amount of data and computational resources. moreover, our analysis also indicates that our models can accelerate the pre-training process.