Multi-Path Transformer is Better: A Case Study on Neural Machine Translation (2022)

Abstract: For years the model performance in machine learning obeyed a power-law relationship with the model size. For the consideration of parameter efficiency, recent studies focus on increasing model depth rather than width to achieve better performance. In this paper, we study how model width affects the Transformer model through a parameter-efficient multi-path structure. To better fuse features extracted from different paths, we add three additional operations to each sublayer: a normalization at the end of each path, a cheap operation to produce more features, and a learnable weighted mechanism to fuse all features flexibly. Extensive experiments on 12 WMT machine translation tasks show that, with the same number of parameters, the shallower multi-path model can achieve similar or even better performance than the deeper model. It reveals that we should pay more attention to the multi-path structure, and there should be a balance between the model depth and width to train a better large-scale Transformer.

## introduction the large-scale neural network has achieved great success at a wide range of machine learning tasks @xcite @xcite . among them, deep models show great potential to deal with complex problems @xcite . by stacking more layers, deeper models generally perform better than shallower models since they provide more non-linearities to learn more complex transformations @xcite . compared to increasing the model depth, broadening the model width can also benefit the model by providing richer features in a single layer. there are two ways to broaden the model width: 1) scaling the matrix dimensions, such as turning the model configuration from transformer-base to transformer-big @xcite . however, both the number of parameters and computational costs will increase quadratically, making such model training and deployment difficult; 2) adopting the multi-path structure @xcite . the expressive power of such models can be improved by fusing abundant features obtained from different paths, and the parameters and computations will only increase linearly with the number of paths. the multi-path structure has proven to be quite important in convolutional networks for computer vision tasks @xcite . but in transformer, this type of structure has not been widely discussed and applied @xcite . in this paper, we continue to study the multi-path structure and adopt a sublayer-level multi-path transformer model. as shown in fig. 1 , the original multi-path models ■ significantly outperform wide models ▲ that scale the matrix dimensions. to make better use of the features extracted from different paths, we redesign the multi-path model by introducing three additional operations in each layer: 1) a normalization at the end of each path for regularization and ease of training; 2) a cheap operation to produce more features; 3) a learnable weighted mechanism to make the training process more flexible. to demonstrate the effectiveness of our method, we test it on the transformer-base and transformerdeep configurations. the experiments are run on 12 wmt machine translation benchmarks, including wmt14 english↔{german, french} and wmt17 english↔{german, finnish, latvian, russian}. experiments on the most widely used @xmath0.65 higher bleu points with the same depth as the transformer-base model. what's more, a shallower multi-path transformer can achieve 29.68 bleu points, which is even higher than the 48-layer transformer-deep model of the same size. it inspires us that, model width is as important as model depth, instead of indefinitely stacking more layers, we should pay more attention to the multi-path structure. ## discussion model depth or width which is more important becomes a hot topic in recent years @xcite @xcite @xcite . in general, one model can benefit more from increasing the depth @xcite @xcite , the reasons can be summarized as follows: 1) expressivity, deep models have better non-linear expressivity to learn more complex transformations. 2) efficiency, both the number of parameters and the computational complexity will be changed quadratically corresponding to the model width (referring to scaling the matrix dimensions) while linearly with the model depth, thus the cost of increasing width is often much higher than that of depth. for tasks such as computer vision, the model depth can even reach hundreds or thousands of layers @xcite . for the transformer model, @xcite even train a transformer model with 1,000 layers. however, the training process of deep models is not as simple as scaling the number of layers. when the model becomes too deep, the degradation problem caused by the back propagation will be exposed @xcite . to seek new solutions to further improve largescale neural networks, here we adopt the parameterefficient multi-path structure. as shown in fig. 1 , the multi-path models significantly outperform wide models that scale the matrix dimensions. from fig. 8 and fig. 9 we can see that, although the multi-path model does not achieve its ideal efficiency because of computational libraries support, it still takes less training cost than wide models. the above discussions show that multi-path is a better option to broaden the model width, and the width of one model is as important as its depth for the purpose of improving capacity. ## conclusion in this work, we construct a sublayer-level multipath structure to study how model width affects the transformer model. to better fuse features extracted from different paths, three additional operations mentioned in section 3 are introduced. the experimental results on 12 machine translation benchmarks validate our point of view that, instead of indefinitely stacking more layers, there should be a balance between the model depth and width to train a better large-scale transformer.