U ni D rop: A Simple yet Effective Technique to Improve Transformer without Extra Cost (2021)

Abstract: Transformer architecture achieves great success in abundant natural language processing tasks. The over-parameterization of the Transformer model has motivated plenty of works to alleviate its overfitting for superior performances. With some explorations, we find simple techniques such as dropout, can greatly boost model performance with a careful design. Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unites three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play different roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classification benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for the classification even using strong pre-trained RoBERTa as backbone.

## introduction in recent years, transformer @xcite has been the dominant structure in natural language processing (nlp), such as neural machine translation @xcite , language modeling @xcite ) and text classification @xcite . to further improve the model performance, there has been much effort in designing better architectures or introducing external knowledge into transformer models @xcite @xcite @xcite , which increases computational costs or requires extra resources. despite the effectiveness of above strategies, the over-parameterization and overfitting is still a crucial problem for transformer. regularization methods such as weight decay @xcite , data augmentation @xcite , dropout @xcite , parameter sharing @xcite are all widely adopted to address overfitting. among these regularization approaches, dropout @xcite , which randomly drops out some hidden units during training, is the most popular one and various dropout techniques have been proposed for transformer. for example, @xcite propose layerdrop, a random structured dropout, to drop certain layers of transformer during training. @xcite alternatively propose drophead as a structured dropout method for regularizing the multi-head attention mechanism. both of them achieved promising performances. one great advantage of dropout is that it is free of additional computational costs and resource requirements. hence we ask one question: can we achieve stronger or even state-of-the-art (sota) results only relying on various dropout techniques instead of extra model architecture design or knowledge enhancement? to this end, in this paper, we propose unidrop to integrate three different-level dropout techniques from fine-grain to coarse-grain, feature dropout, structure dropout, and data dropout, into transformer models. feature dropout is the conventional dropout @xcite that we introduced before, which is widely applied on hidden representations of networks. structure dropout is a coarse-grained control and aims to randomly drop some entire substructures or components from the whole model. in this work, we adopt the aforementioned layerdrop @xcite as our structure dropout. different from the previous two dropout methods, data dropout @xcite is performed on the input data level, which serves as a data augmentation method by randomly dropping out some tokens in an input sequence. we first theoretically analyze different regularization roles played by the three dropout techniques, and we show they can improve the generalization ability from different aspects. then, we provide empirical evaluations of the unidrop approach. we conduct experiments on neural machine translation with 8 translation datasets, and text classification task with 8 benchmark datasets. on both sequence generation and classification tasks, experimental results show that the three dropouts in unidrop can jointly improve the performance of transformer. the contributions of this paper can be summarized as follows: ## background feature dropout (fd) and structure dropout (sd) are highly coupled with model architecture. therefore, we briefly recap transformer and refer the readers to @xcite for details. as shown in figure 1a , transformer is stacked by several identical blocks, and each block contains two sub-layers, which are multi-head selfattention layer and position-wise fully connected feed-forward layer. each sub-layer is followed by an addnorm operation that is a residual connection add @xcite and a layer normalization @xcite . multi-head attention sub-layer consists of multiple parallel attention heads, and each head maps the query q and a set of key-value pairs k, v to an output through a scale dot-product attention: ## unidrop in this section, we first introduce the details of the three different levels of dropout techniques we study, feature dropout, structure dropout and data dropout. then we provide the theoretical analysis of these dropout methods on the regularization perspectives. finally, we present our proposed unidrop approach for training transformer. ## experiments we conduct experiments on both sequence generation and classification tasks, specifically, neural machine translation and text classification, to validate the effectiveness of unidrop for transformer. ## analysis in this section, we use iwslt14 @xmath0, as well as the effects of different dropout components and dropout rates on unidrop. ## conclusion in this paper, we present an integrated dropout approach, unidrop, to specifically regularize the transformer architecture. the proposed unidrop unites three different level dropout techniques from fine-grain to coarse-grain, feature dropout, structure dropout, and data dropout respectively. we provide a theoretical justification that the three dropouts play different roles in regularizing transformer. extensive results on neural machine translation and text classification datasets show that our transformer+unidrop outperforms the standard transformer and various ablation versions. further analysis also validates the effectiveness of different dropout components and our two-stage data dropout strategy. in conclusion, the unidrop improves the performance and generalization of the transformer without additional computational cost and resource requirement.