Pointing to Select: A Fast Pointer- LSTM for Long Text Classification (2020)

Abstract: Recurrent neural networks (RNNs) suffer from well-known limitations and complications which include slow inference and vanishing gradients when processing long sequences in text classification. Recent studies have attempted to accelerate RNNs via various ad hoc mechanisms to skip irrelevant words in the input. However, word skipping approaches proposed to date effectively stop at each or a given time step to decide whether or not a given input word should be skipped, breaking the coherence of input processing in RNNs. Furthermore, current methods cannot change skip rates during inference and are consequently unable to support different skip rates in demanding real-world conditions. To overcome these limitations, we propose Pointer- LSTM, a novel LSTM framework which relies on a pointer network to select important words for target prediction. The model maintains a coherent input process for the LSTM modules and makes it possible to change the skip rate during inference. Our evaluation on four public data sets demonstrates that Pointer-LSTM (a) is 1.1x∼3.5x faster than the standard LSTM architecture; (b) is more accurate than Leap-LSTM (the state-of-the-art LSTM skipping model) at high skip rates; and (c) reaches robust accuracy levels even when the skip rate is changed during inference.

## introduction recurrent neural networks (rnns), such as long short-term memory (lstm) @xcite and gated recurrent unit (gru) @xcite , are popular architectural choices for many sequential text processing tasks. typically, rnns read in either all or a fixed-length slice of tokens sequentially, and output a distributed representation for each token. while this paradigm is essential for some applications such as machine translation @xcite , it may result in suboptimal or practically prohibitive processing times for long input sequences which are not uncommon in document classification @xcite and complex question answering @xcite . building on wider, established observations that (i) not all input tokens are equally important in many tasks @xcite ; (ii) text can be comprehended even through partial reading due to linguistic redundancy effects @xcite ; and (iii) attention, fixations, and saccades govern human reading in general @xcite , recent rnn optimisation efforts have focused on mechanisms to skip irrelevant, unimportant, or redundant words in the input @xcite @xcite @xcite . we refer to such models as selective rnns. in general, existing selective rnns use a decision-making component to predict whether or not a word should be skipped (skimmed). in a typical selective rnn workflow, the rnn's input steps are interrupted by the decision network to make a decision to copy the previous hidden state or calculate a new hidden state for the next time step, for example. interruptions of this kind make rnns unnecessarily complicated from the point of view of implementation and optimisation. furthermore, existing selective rnns make changes to the skip rate impossible during inference. however, real-world applications frequently call for flexible skip rates in order to optimise the trade-off between inference time (i.e. latency) and accuracy, amongst others. dynamic skip rates during inference can reduce the cost of iterative model retraining and make it easier to manage many differently parameterised models. this paper proposes pointer-lstm, a novel selective rnn framework which maintains a coherent rnn input procedure (i.e. no interruptions at each time step) and supports flexible inference skip rates. we draw inspiration from the pointer network @xcite @xcite which was designed to select and copy tokens from the input to the output. pointer-lstm similarly learns to select important words from the input. however, instead of merely copying the selected words to the output, pointer-lstm copies a proportion of the input words to a large lstm network for target prediction. more specifically, pointer-lstm uses a small bidirectional lstm (bilstm) network to produce a self-attention distribution over the entire input sequence; the attention distribution is then sampled to select top-k important words for target prediction by a large bilstm. the value of k in top-k is controlled by a pre-set skip rate r following @xcite . the main contribution of this paper is as follows: ## related work in this section, we first review some representative selective rnn frameworks, and then introduce the pointer network and its applications in nlp. ## our approach with faster long text classification as our goal, we investigate a selective lstm architecture which uses the attention mechanism as a pointer to select a controllable portion of important words. we regard word selection as a generation process and take the pointer-generator network for text summarisation @xcite as our starting point. we first introduce the pointer-generator network and then describe our modifications to it suitable for text classification (henceforth referred to as pointer-lstm). ## experiments we evaluate pointer-lstm on four benchmark data sets for long text classification. the state-of-theart selective rnn -leap-lstm @xcite -is used as our baseline foot_0 . we first compare the accuracy of our models and their speed-up over the standard lstm architecture foot_1 . we then investigate the robustness of pointer-lstm in cases where the skip rate is changed during inference, focusing on accuracy at different skip rates. ## conclusion and future work in this paper, we propose a novel selective rnn framework for long text classification -pointer-lstm -which can select important words from the input sequence more effectively than recent state-of-the-art methods. pointer-lstm uses a small attentive lstm network to produce an importance distribution over all input tokens using a pre-set skip rate to select important words as the input to a large lstm network for target prediction. this architectural design offers two advantages over existing selective rnn frameworks in that (i) pointer-lstm maintains a coherent, uninterrupted input process for the lstm components, and (ii) the skip rate can be adjusted during inference. our experiments on four long text classification data sets demonstrate that pointer-lstm offers higher accuracy and faster inference than the state-of-the-art selective rnn (leap-lstm) at a high skip rate. the pointer-lstm's accuracy also remains high when the skip rate is changed during inference. compared to leap-lstm, pointer-lstm offers a more stable control over the skip rate during training. in many industry scenarios under heavy resource constraints (such as memory-limited or cpu-only hardware, limited training data, and latency requirements), not all large-scale deep architectures can be deployed as-is. our skipping model, which can scan any long sequence and keep important words while reducing information loss, does not suffer from the inherent maximum input length limitation which hamstrings many a deep architecture. we will explore alternative attention mechanisms to select important words in future work. owing to the fact that the input into the main model can be condensed significantly after word selection, we will explore non-rnn networks (for example, convolutional neural networks and transformer variants) as the main model for target classification.