P artial F ormer: Modeling Part Instead of Whole for Machine Translation (2024)

Abstract: The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multi-head attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer’s capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/zhengkid/PartialFormer.

## introduction the transformer model @xcite has emerged as a cornerstone in the natural language processing (nlp) domain, overshadowing convolutional neural networks @xcite and recurrent neural networks @xcite by virtue of its minimal inductive bias, superior scalability, and proficiency in modeling sequences. nonetheless, its substantial computational and parametric requisites pose significant challenges to its deployment and training, warranting an ongoing trend in the research community toward eliminating redundant parameters and computations @xcite @xcite ; figure 1: illustration of our idea. @xcite @xcite in transformer. while these attempts represent significant strides in enhancing the efficiency of the transformer architecture, they largely neglect an equally critical component: the feed-forward network (ffn) that constitutes a substantial part of the transformer's computational and parametric footprint, due to the inherent large feature space and hidden dimension. previous studies @xcite @xcite have simplified ffns by naively reducing their hidden dimensions, often at the expense of expressive power. this leads to a question: is the current formulation of lightweight ffns truly optimal? to answer this concern, we turn to the insights provided by @xcite , who depicted ffns as a collection of key-value memories, where the number of memories is equal to the number of hidden dimensions in ffns. this finding underscores the significance of hidden dimension in ffns. drawing inspiration from this finding and the successful application of large hidden sizes in ffns as evidenced by meta's 4b model @xcite foot_0 , we hypothesize that an efficient lightweight ffn is not merely about parameter reduction. rather, it should aim to maintain or even increase the hidden dimension while judiciously reducing the number of parameters involved. the literature on animal cognition provides some clues for designing lightweight and expressive ffns. research on animals' behavior has shown that group animals such as insects, fish, and some birds can emerge with some incredible abilities to deal with some complex tasks, though each individual owns poor abilities @xcite . this concept resonates with the ai community's "swarm intelligence" paradigm @xcite , which emphasizes the power of collective decision-making. this biological prior motivates us to integrate swarm intelligence principles into the ffn design process. to this end, we propose partialformer, an innovative approach to transformer architecture. at the heart of partialformer lies the novel concept of @xcite . conceived as an ensemble of streamlined ffns operating in concert, each pg-ffn produces lower-dimensional hidden features. despite their reduced individual dimensions, the aggregated output of these pg-ffns either matches or surpasses the hidden dimensions of traditional, larger ffns, as empirically substantiated in figure 1 . moreover, we further equipped partialformer with a head scaling strategy tailed for efficiently scaling, and a residual-like attention calculation for stable optimization. these techniques empower partial-former to efficiently utilize parameters within the same parameter budget. our main contributions are as follows: ## preliminary: transformer in this section, we present some prior knowledge about the transformer. the transformer block consists of a multi-head self-attention and a feedforward network. let @xmath0. each multi-head self-attention component owns h heads. for simplicity, we omit layer normalization and residual connections. multi-head self-attention mhsa aims to model the global dependency among tokens. mhsa computes as follows: feed-forward network feed-forward network is responsible for improving the expressiveness of the whole representation space by adopting an "expansion-activation-reduction" mapping strategy. it computes as follows: ## experimental setups we assess partialformer's performance across both machine translation and abstractive summarization tasks foot_1 . more details are given in appendix a. dataset. for the machine translation task, we selected 9 datasets involving wmt'14 english-german (en-de), wmt'14 english-french (en-fr), wmt'16 english-romanian (en-ro), and six translation tasks from wmt'17 benchmark. we preprocessed the raw data following the standard strategy. for the abstractive summarization task, we utilized the widely-used cnn-dailymail dataset. we followed the same preprocessing approach as described in @xcite . we applied type model n -m d dk h param bleu comet-22 sbleu ## related work lightweight transformers several strands of research have been dedicated to enhancing the parameter efficiency of the transformer architecture, each taking a distinct approach to the problem at hand. the first category aims to mitigate redundancy directly through architectural innovations, employing more efficient transformation operations @xcite , integrating disparate yet synergistic patterns @xcite , or leveraging neural architecture search techniques @xcite . another avenue of research explores weight sharing as a means of improving parameter efficiency, exemplified by the universal transformer's cross-layer parameter sharing strategy @xcite . moreover, @xcite introduced an ordinary differential equation-inspired weight-sharing approach to achieve higher performance. different from them, our study focus on the design of lightweight ffn. multi-branch transformer the multi-branch strategy is widely used in transformer design. weighted transformer @xcite employs a multi-branch ffn, while multi-attentive transformer @xcite , multi-units transformer @xcite , and multi-path transformer @xcite extend this concept to different components of the transformer. our partialformer can be viewed as a pure multibranch architecture based on natural subspaces. ## conclusion in this paper, we present partialformer, a new parameter-efficient transformer architecture that offers an alternative approach to the design of the lightweight ffn. by employing multiple small ffns and leveraging matrix factorization techniques, partialformer effectively reduces the number of parameters in the ffn. moreover, we propose two innovative operations to further efficiently enhance the model capabilities. experimental results across various machine translation tasks showcase the significant performance improvements achieved by partialformer, while maintaining comparable parameter consumption.