Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020)

Abstract: This paper introduces a method that efficiently reduces the computational cost and parameter size of Transformer. The proposed model, refer to as Group-Transformer, splits feature space into multiple groups, factorizes the calculation paths, and reduces computations for the group interaction. Extensive experiments on two benchmark tasks, enwik8 and text8, prove our model’s effectiveness and efficiency in small-scale Transformers. To the best of our knowledge, Group-Transformer is the first attempt to design Transformer with the group strategy, widely used for efficient CNN architectures.

## introduction character-level language modeling has become a core task in the field of natural language processing (nlp) such as classification @xcite , sequence tagging @xcite , question answering @xcite , and scene text recognition @xcite , with its simplicity on generating text and its adaptability to other languages. most previous approaches had consisted of recurrent neural networks (rnns), but they have suffered from high learning complexity caused by inherently long character sequences. recently, transformer @xcite have shown promise in addressing this problem and have become a standard way in general language modeling @xcite . transformers have achieved higher performance but have also grown in size by building deeper and wider networks. transformerxl @xcite and gpt-2 @xcite , for instance, contain 277m and 1542m parameters, respectively. however, this trend toward a large size model for performance is not suitable for edge device applications that require small memory sizes and fast real-time responsiveness, such as auto-correction and auto-completion @xcite . contrary to the recent trend, character-level language models need to be scaled down while minimizing performance degradation due to capacity loss. a simple way to get a lightweight transformer is to reduce its width and depth, directly related to the model complexity. however, the width reduction loses representation power of high dimensional feature space, and the depth reduction brings the lower capacity that stacks diverse dependencies between local information. to compensate for the losses, knowledge distillation @xcite tries to optimize a scale-downed model with a large teacher network, and weight-sharing @xcite stacks a unified layer multiple times. they have shown promising results, but they still require a scale-downed model as a target model, or a unified layer for iterative usage. in this paper, we introduce a lightweight transformer, referred to as group-transformer, with lower model complexity without any modification to its width and depth. our method utilizes group-wise operations, inspired by the group convolution approaches @xcite that have effectively compressed huge image processing models. the basic concept of group convolution is to partition the feature maps into multiple groups and process them individually, rather than connecting them. figure 1 shows a brief overview of our proposed model utilizing the group strategy. replacing all fully connected operations to group-wise operations reduce the model complexity since only a few connections remain as intra-group connections. beside, group-transformer employes lightweight inter-group operations to compensate for the information loss of inter-group correlations. the mutually exclusive calculation of the group strategy compromises performance, but modeling the interactions for all group pairs might be over-parameterized. our inter-group operations share a common feature over groups in attention layers and utilize a low-rank approximation in feed-forward layers to model the inter-group information flows with a few calculations. we conducted extensive experiments on two benchmark datasets, enwik8 and text8, and found that group-transformer showed better performance when compared against transformers with a comparable number of parameters under 10m. furthermore, when scaling down transformer, group-transformer shows promising results comparing to other scale-down methods. we provide further analysis to identify the contributions of our proposed modules in detail. to the best of our knowledge, group-transformer is the first attempt to build a lightweight transformer with the group strategy. ## group-transformer figure 2a shows the overall architecture of group-transformer. it consists of a group embedding (bottom grey box), which embeds a character into grouped features, group attention (yellow box), which contains attention modules to identify dependencies in the time domain, and group feed-forward layer (green box), which re-configures the grouped features. as can be seen, when an input character is given, group-transformer converts the input into multiple group representations (blue dots and red dots), processes and merges them to predict the next character. figure 2b and 2c show group-wise information flow (blue and red arrows) and inter-group information flow (grey arrow) in the sub-modules. without the intergroup information flows, the grouped features are processed independently. we observed that inter-group modeling ensures that the groups become aware of the others and prevents different groups hold the same information. the following subsections describe the architectural details of the sub-modules and their relations. for a simple description, we describe the processes for a single time step. ## conclusion recently, remarkable progress has been made in character-level language modeling by transformer. however, the models generally have improved performance in proportion to the size of the parameters, and training and reasoning about them required high computational costs. we argue that an efficient scale-down method is required because big models cannot be used in a limited computational environment. group-transformer has been developed to meet the requirement. the proposed model reduces computations and parameter sizes of transformer while keeping its feature dimension. when applying group-transformer on enwik8 and text8, we found that group-transformer achieves better performances than transformer-xl in small-scale experimental settings. further analysis has proved the effectiveness of the group strategy to reduce computational resources.