Hierarchical Transformers Are More Efficient Language Models (2022)

Abstract: Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences, which allows them to produce long coherent outputs: entire paragraphs produced by GPT-3 or well-structured images produced by DALL-E. These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them hierarchical. We use the best performing upsampling and downsampling layers to create Hourglass - a hierarchical Transformer language model. Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new state-of-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark.

## introduction transformer models @xcite are capable of solving many sequence modeling tasks, including classical nlp tasks @xcite , summarization @xcite , language modeling @xcite , code generation @xcite , or even music generation @xcite and image generation @xcite @xcite . one compelling feature of transformers is their ability to handle long contexts given as part of the input. this is particularly visible in tasks where the output depends on parts of the context that may not be * equal contribution. order determined by coin toss. the ability of transformers to handle long contexts comes at a price: each self-attention layer, at least in its original form, has complexity quadratic in the length of the context. when a stack of n transformer layers is used, both memory and time complexity is equal to o(l 2 n) where l is a sequence length and n number of decoder blocks. due to this limitation, vanilla transformers are infeasible to train on tasks with very long input sequences, for instance, on high-resolution images. this issue has been studied extensively, and a number of techniques were introduced that modify attention mechanism without changing overall transformer architecture @xcite @xcite . these sparse attention mechanisms reduce the complexity of self-attention but still force the model to operate on the sequence of the same length as the input. for generative transformer models, operating at the original scale of the input sequence is necessary, at least in the early and final layers, as the input must be processed at first and generated at the end (section 4.3). but forcing the models to operate at this granularity throughout the layer stack has both fundamental and practical shortcomings: to alleviate these issues, we propose to change the transformer architecture to first shorten the internal sequence of activations when going deeper in the layer stack and then expand it back before generation. we merge tokens into groups using a shortening operation (section 2.1) and so reduce the overall sequence length, and then up-sample them again combining with the sequence from earlier layers (section 2.3), the first part is analogous to the funnel-transformer architecture @xcite , and the whole architecture takes inspiration from u-nets @xcite . in contrast to both these architectures, the model we present is autoregressive, which is harder to ensure in hierarchical models than in vanilla transformers. the resulting model -which we call hourglassis an autoregressive transformer language model that operates on shortened sequences. it yields significant performance improvements for different attention types (fig. 6 , 7 ). we tested hourglass with transformer-xl @xcite and reformer @xcite blocks on enwik8 dataset. in both cases, it is not only better in terms of perplexity, but it is faster and uses less memory during training. we also propose a regularization technique for hierarchical transformers called shorten factor dropout which improves perplexity upon baselines trained with fixed shorten factor (see section 4.1). finally, hourglass achieves the new stateof-the-art among transformer models for image generation of imagenet32 (see tab. 3 ). ## model standard self-attention mechanism uses full tokenlevel sequence representations. in the hourglass, we bring efficiency to the model by utilizing shortening, which allows us to use the transformer layers on inputs with significantly smaller lengths. a high-level overview of our proposed model architecture is shown in figures 2 and 3. attention type in the vanilla layers and shortened layers is a configurable parameter. by default we use relative attention defined in transformer-xl @xcite . any attention module can be used -we show significant efficiency gains when applying hourglass also for lsh @xcite ) attention (see section 3.2 and fig. 7 ). ## experiments in this section, we present experimental results of hourglass. we start with a quick analysis of time and memory complexity of the approach (section 3.1). then we investigate the efficiency gains of applying hourglass to transformers with different attention types (section 3.2). finally, we use hourglass with relative attention parametrization from transformer-xl @xcite , evaluate it on three language modeling tasks, and compare the results with other models. (sections 3.3, 3.4) to show cross-domain generalization of our method, we train our model on one dataset related to natural language processing and two from the computer vision field. to ensure consistency in presenting configurations of our model, we introduce a notation describing hierarchy of our architecture: our model implementation is open source. figure 4 : an overview of our shortening approach. different colors denote token positions. initially, we shift right by one, which is a standard step in transformerlm. then, just before performing shortening, we additionally shift the tokens right by shorten factor -1 to preserve the autoregressive property of the model. ## ablations in this section, we start by introducing a training technique called shorten factor dropout (section 4.1), and then analyze hourglass's components de-imagenet32 bpd pixelcnn (van den @xcite 3.83 image transformer @xcite 3.77 axial transformer @xcite 3.76 hourglass 3.74 vdm @xcite 3.72 denseflow @xcite 3.63 imagenet64 bpd reformer @xcite 3.65 performer @xcite 3.64 hourglass 3.44 sparse transformer @xcite 3.44 routing transformer @xcite 3.43 combiner @xcite 3.42 vdm ( 2021) 3.40 denseflow ( 2021) 3.35 ## conclusion in this paper, we show how hierarchy can improve the efficiency of transformers in a language modeling setup. our proposed architecture, hourglass, significantly outperforms the baseline both in terms of perplexity reached at a given computation cost (figure 1 ) and empirical metrics like running memory (figure 6 ). hourglass achieves state-of-the-art results among autoregressive models on the ima-genet32 generation task and competitive results on other image generation and language modeling tasks. hourglass can be used with any attention type, which opens many directions for future research related to transformers capable of processing longer sequences. another line of future work might be related to advances in the shortening mechanism itself, for example, involving a dynamic pooling operation that could explicitly handle the problem of fixed-size groups in multi-stage shortening. we also leave open the problem of choosing the best hi-erarchy for a task. we conjecture that experiments with much longer contexts will provide better guidance for this choice and will benefit even more from the hourglass architecture.