L ittle B ird: Efficient Faster & Longer Transformer for Question Answering (2022)

Abstract: BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a limitation dealing with long inputs due to its attention mechanism. Longformer, ETC and BigBird addressed this issue and effectively solved the quadratic dependency problem.However we find that these models are not sufficient, and propose LittleBird, a novel model based on BigBird with improved speed and memory footprint while maintaining accuracy.In particular, we devise a more flexible and efficient position representation method based on Attention with Linear Biases(ALiBi). We also show that replacing the method of global information represented in the BigBird with pack and unpack attention is more effective.The proposed model can work on long inputs even after being pre-trained on short inputs, and can be trained efficiently reusing existing pre-trained language model for short inputs. This is a significant benefit for low-resource languages where large amounts of long text data are difficult to obtain.As a result, our experiments show that LittleBird works very well in a variety of languages, achieving high performance in question answering tasks, particularly in KorQuAD2.0, Korean Question Answering Dataset for long paragraphs.

## introduction transformer @xcite and pre-trained language models @xcite based on it have shown a lot of success in a wide variety of nlp tasks. however, the quadratic dependency problem that comes from the attention mechanism makes it impractical to process long documents. many techniques have been studied to overcome this problem and bigbird @xcite showed robust and state-of-the-art performance on various nlp downstream tasks. in this study, we propose a new model littlebird by analyzing and improving the shortcomings of bigbird. littlebird shows improved speed and memory footprint compared to bigbird while maintaining the overall accuracy of the question answering (qa) benchmarks and showing better accuracy in some of them. in this study, we propose three major improvements compared to bigbird. the first is the method for position representation. in bigbird, trainable positional embedding is used similar to bert @xcite , and in etc, relative positional encoding is used similar to t5 @xcite . however, trainable positional embedding cannot handle longer inputs than those used for training and the relative position encoding is relatively slow and uses extra memory and parameters @xcite . @xcite introduced the attention with linear biases (alibi) method that resolves these problems, but it was designed for causal language modeling, not autoencoding language modeling, which is typically useful for qa tasks. thus, we devise a new method based on the alibi that is fast, flexible, and also effective in qa tasks. the second is the method of capturing global information. bigbird introduces two ways of capturing global information, the random sparse attention and global tokens @xcite which attend to and be attended by all other tokens. however, the random attention method is practically slow compared to its time complexity because it requires to repeat gather and scatter operations at random positions. in addition, a relatively large number of (∼hundreds) global tokens are required to achieve the reported performance using only global tokens without a random attention method in etc. we show that replacing them with modified pack and unpack attention @xcite ) is more effective. the last is the efficient way to train a model for long sequences. we introduce a simple but effective method, padding insertion, which makes the model robust to long inputs while training on short inputs. we also propose a distillation method that can maximize the reuse of the pre-trained model for a short length and show that our model can be effectively pre-trained using these methods. our model shows a 12∼29% reduction in peak memory usage and a 6∼46% reduction in latency compared to various bigbird and etc model settings reported in the paper for 4k length document inference while showing better accuracy on several english qa benchmarks dev sets @xcite . our model achieves new state-of-the-art performance on korquad 2.0 @xcite , a korean long document qa benchmark. in addition, these results are obtained with littilebird pre-trained with only 2k sequence length. it shows that our novel positional representation method works well when the model is applied to the qa downstream task with a document longer than the sequence used in the pre-training phase. ## littlebird architecture in this section, we describe the architecture of lit-tlebird model. basically, the model can be viewed as a composition of several key ideas including sliding window attention from bigbird @xcite , linear bias to attention from alibi @xcite and pack and unpack attention from luna @xcite . ## experiment the goal of this section is to demonstrate the benefits of littlebird compared to other long transformer models. first we check if the proposed training method works well with the architecture of littlebird. following that, we consider qa tasks of english and korean, which requires longer sequence modeling. lastly we prove that padding insertion significantly improves accuracy and that the efficiency of our model by performing training and inference for various input lengths. ## conclusion we propose littlebird, which is more efficient in terms of memory and computational time than existing transformer models for long sequences, and its effective way to train. it combines a novel position encoding method, bialibi, and pack & unpack with sliding window attention to achieve high speed and accuracy, particularly in question answering tasks for long documents. the distillation and training method with padding insertion allows the model to be trained by reusing the existing pre-trained language model for short inputs and work well for long inputs even if trained on short inputs. we demonstrated through experiments that the accuracy of question answering improves as the model is fed a longer input, and we achieved state-of-the-art performance in korquad2.0 using littlebird.