Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU (2021)

Abstract: Incremental processing allows interactive systems to respond based on partial inputs, which is a desirable property e.g. in dialogue agents. The currently popular Transformer architecture inherently processes sequences as a whole, abstracting away the notion of time. Recent work attempts to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. However, this approach is computationally costly and does not scale efficiently for long sequences. In parallel, we witness efforts to make Transformers more efficient, e.g. the Linear Transformer (LT) with a recurrence mechanism. In this work, we examine the feasibility of LT for incremental NLU in English. Our results show that the recurrent LT model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental (full sequence) quality. We show that the performance drop can be mitigated by training the model to wait for right context before committing to an output and that training with input prefixes is beneficial for delivering correct partial outputs.

## introduction one fundamental property of human language processing is incrementality @xcite . humans process language on a word-by-word basis by maintaining a partial representation of the sentence meaning at a fast pace and with great accuracy @xcite . the garden path effect, for example, shows that language comprehension is approximated incrementally before committing to a careful syntactic analysis @xcite @xcite . the notion of order along the time axis during computation is a key aspect of incremental pro-cessing and thus a desirable property both of cognitively plausible language encoders as well as in applications such as interactive systems @xcite ). rnns, for example, are inherently able to process words sequentially while updating a recurrent state representation. however, the transformer architecture @xcite , which has brought significant improvements on several nlp tasks, processes the input sequence as a whole, thus prioritising parallelisation to the detriment of the notion of linear order. one way to employ non-incremental models in incremental settings is resorting to an incremental interface, like in @xcite , where a complete recomputation of the available partial input happens at each time step to deliver partial output. @xcite examined the output stability of non-incremental encoders in this restart-incremental fashion. while qualitatively feasible, this procedure is computationally costly, especially for long sequences, since it requires as many forward passes as the number of input tokens. in parallel, there is ongoing research on ways to make transformers more efficient, e.g. the linear transformer (lt) introduced by @xcite . besides being more efficient, lts can be employed with a recurrence mechanism based on causal masking that turns them into models similar to rnns. in this work, we examine the suitability of using lts in incremental processing for sequence tagging and classification in english. we also inspect the use of the delay strategy @xcite @xcite to examine the effect of right context availability on the model's incremental performance. our hypothesis is that recurrence will allow lts to be better in incremental processing as it captures sequence order. as lts use an approximation of softmax attention, we also expect a performance drop compared to the standard transformer while being faster in the incremental setting due to its linear time attention. ## related work in recent years, neural approaches, including transformer-based architectures @xcite , have become more popular for incremental processing. given that transformer models are not inherently incremental, employing them for incremental processing demands adaptation. in simultaneous translation, for instance, @xcite proposed to use an incremental encoder by limiting each source word to attend to its predecessors and recompute the representation for previous source words when there is new input. @xcite introduced an average embedding layer to avoid recalculation when using an incremental encoder, while exploiting right context through knowledge distillation. an investigation of the use of non-incremental encoders for incremental nlu in interactive systems was conducted by @xcite . the authors employed bert @xcite for sequence tagging and classification using restart-incrementality, a procedure with high computational cost. the computational cost of a restart-incremental transformer can be reduced with more efficient models or even avoided if an inherently incremental transformer architecture existed. recent works have proposed modifications that could help achieve that. for instance, by approximating the softmax attention with a recurrent state @xcite @xcite . the linear transformer model @xcite can be viewed as an rnn when the attention is causal (see also, very recently, @xcite . ## conclusion we studied the use of transformer encoders for incremental processing and concluded that it is possible to deploy them as incremental processors with certain trade-offs. with recurrent computation, the linear transformer (lt) has inferior nonincremental performance compared to the regular transformer and the lt with restart-incrementality. however, it has the great advantage of being much more efficient for incremental processing, since recomputation at each time step is avoided. the output of the recurrent lt is generally more stable for sequence classification and monotonic for tagging. its non-incremental performance drop can be mitigated by introducing delay, which also improves the incremental metrics. it is also beneficial to train such model with input prefixes, allowing it to learn more robust predictions.