Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine Translation (2022)

Abstract: Transformer has been demonstrated effective in Neural Machine Translation (NMT). However, it is memory-consuming and time-consuming in edge devices, resulting in some difficulties for real-time feedback. To compress and accelerate Transformer, we propose a Hybrid Tensor-Train (HTT) decomposition, which retains full rank and meanwhile reduces operations and parameters. A Transformer using HTT, named Hypoformer, consistently and notably outperforms the recent light-weight SOTA methods on three standard translation tasks under different parameter and speed scales. In extreme low resource scenarios, Hypoformer has 7.1 points absolute improvement in BLEU and 1.27 X speedup than vanilla Transformer on IWSLT’14 De-En task.

## introduction transformer @xcite is one of the most effective models in nmt. transformer could achieve better performance by stacking more layers and increasing the embedding dimension @xcite . however, this leads to a huge scale of parameters and a slow inference speed. various works have been recently proposed to compress transformer, such as light-weight variants @xcite , neural architecture search @xcite , parameter sharing @xcite , low-rank approximation @xcite @xcite , knowledge distillation @xcite , layer reassignment @xcite @xcite . following the line of low-rank approximation, existing works to compress nmt models (or related deep transformer models like bert) 1) are specific to a part of sub-components, e.g., selfattention networks @xcite or embedding layer @xcite ; and 2) show limited or no potential for speedup during inference @xcite @xcite . however, unlike the cloud-based system, it needs real-time feedback and faster inference in edge devices (e.g., smartphones and iots) even with constrained resources. although it is challenging, we believe that low-rank approximation has the potential for compressing and accelerating nmt models in edge devices. from a general point of view, this paper rethinks low-rank approximation methods, especially two representative methods namely matrix factorization (mf) and tensor-train decomposition (tt). the former is arguably time-efficient while the latter is parameter-efficient. however, (1) mf will encounter a low-rank bottleneck with a high compression ratio, potentially leading to a moderate performance drop @xcite . (2) transformer compression based on high tt-rank tt decelerates inference speed due to its quadratic computational complexity. interestingly, tt was claimed to retain a full matrix rank by @xcite while the expressive power of mf is bounded to rank r (see to avoid performance degradation while preserving its parameter efficiency of low-rank tt, we propose to compensate low-rank tt with an auxiliary dense projection matrix, resulting in a hybrid decomposition (figure 2e ), called hybrid tensor-train (htt) decomposition, which is inspired by the hybrid form of mf @xcite . htt can retain the full matrix rank, since its dense part is full matrix rank in definition, and its tt part is theoretically full matrix rank referring to tt embedding @xcite . the full matrix rank indicates that the expressivity of the original matrix can be retained in its decomposed form. thus, it is expected that htt decomposition can maintain the performance of original matrix, meanwhile with fewer operations and parameters. based on htt decomposition, we present a transformer-based nmt models, called hypoformer in a knowledge distillation fashion. specifically, we replace the dense layers of subcomponents (like embedding and self-attention) with htt, which can compress and accelerate the vanilla transformer. we evaluate hypoformer with three standard machine translation tasks (iwslt'14 de-en, wmt'16 en-ro, and wmt'14 en-de) on raspberry pi arm cpu and intel cpu. hypoformer consistently outperforms recent lightweight stateof-the-art methods @xcite @xcite under various parameter scales and speed scales. achieving similar performance on three tasks, hypoformer can compress transformer base model @xcite 2.9 ∼ 4.5×, and speedup it 1.9 ∼ 3.3× on intel cpu and 1.8 ∼ 3.9× on raspberry pi arm cpu. notably, when the parameter scale is extremely low, hypoformer achieves a 7.1 higher bleu score and 1.27× speedup than transformer on iwslt'14 de-en task. the contributions of this work are as follows: (1) based on the analysis of previous low-rank approximation methods @xcite , we propose a novel hybrid tensor-train (htt) decomposition, which can retain full matrix rank while with fewer operations and parameters. (2) applying htt decomposition to compress and accelerate transformer, hypoformer consistently outperforms the recent light-weight sota methods @xcite with a higher compression ratio, speedup, and better bleu score. (3) in edge devices with extremely limited resources, hypoformer can retain 95% performance with 1/12 parameters, setting a new sota in extremely-small nmt models. ## preliminaries tensor representation first, we introduce the tensor diagrams @xcite ). an n-order tensor can be denoted as a node with n legs. each leg is labeled by the length (an associated positive integer) of the corresponding dimension, and the orientation of the leg does not matter. as shown in figure 1 , a vector @xmath0, a matrix @xmath1, shown in figure matrix factorization is a common and effective technique to compress deep neural networks. to compress the fully connected layer, the weight matrix @xmath2, as shown in figure 2(b ). the compression ratio of matrix factorization (mf) is related to the mf rank r; the larger r is, the smaller the compression ratio and the bigger the capacity. tensor-train decomposition @xcite gives a method to represent the high-order tensor with a low-rank approximation. in tensor-train (tt) decomposition, each element in the original tensor can be represented as the product of two vectors and a series of matrices. for the weight matrix @xmath3, where @xmath4, it can be reshaped as a 2n-order tensor @xmath5. then, the element in t is defined as follow: ## hybrid tensor-train decomposition in this section, we first discuss the limitation of mf and tt, as well as their expressivity comparison in the case with limited parameters. then, we introduce hybrid tensor-train decomposition, which can retain full matrix rank but with fewer operations and parameters than the original matrix. figure 2 : the operation of different decomposition using tensor graphical representations. for a given feature representation denoted as @xmath6, there show five ways to implement an i × j linear projection for the given x. the hybrid decomposition 1) conducts both a dense linear projection and mf/tt layer on an identical x; and 2) concatenates such output to obtain j-dimensional output. α denotes the ratio of dense layer, where α ∈ [0, 1]. ## hypoformer in this section, we first show the overall structure between the transformer and our hypoformer in figure 3 . then, we introduce the three hypoformer sub-layers including htt embedding, htt self-attention, and lmf feed-forward network. ## related work low-rank approximation is a common and effective technique to compress deep neural networks. matrix factorization, which expresses a matrix @xmath7. hybrid matrix factorization @xcite ) is an extension of mf, which inspired our hybrid tensor train decomposition. tensor-train decomposition is one part of our methods, which has proved its effectiveness in compression in previous works @xcite @xcite . light-weight transformer is an increasing demand for the deployment of transformer in resource-constrained scenarios. recent works explore different methods to compress and accelerate model #params #total s (intel) bleu 1 transformer 31.6 m 36.8 m 1.0 × 34.5 2 + admin 31.6 m 36.8 m 1.0 × 34.9 3 + 12-2 31.6 m 36.8 m 1.9 × 35.4 4 + hypoformer 7.2 m 9.8 m 2.3 × 33.3 5 + hypoformer † 7.2 m 9.8 m 2.3 × 35.2 6 + htt emb † 31.6 m 34.2 m 2.0 × 36.0 7 + htt san † 21.7 m 24.3 m 2.1 × 35.5 8 + lmf ffn † 22.2 m 24.8 m 2.1 × 35.4 ## conclusion in this paper, we present an htt decompositionbased transformer, named hypoformer, which can compress and accelerate transformer. we explore the application of hypoformer on edge devices. in three standard translation tasks, hypoformer is consistently better than recent lightweight sota methods under various parameter and speed scales. in edge devices with extremely limited resources, hypoformer can retain 95% performance with 1/12 parameters, setting a new sota in extremely-small nmt models.