TAPIR : Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model (2023)

Abstract: Language is by its very nature incremental in how it is produced and processed. This property can be exploited by NLP systems to produce fast responses, which has been shown to be beneficial for real-time interactive applications. Recent neural network-based approaches for incremental processing mainly use RNNs or Transformers. RNNs are fast but monotonic (cannot correct earlier output, which can be necessary in incremental processing). Transformers, on the other hand, consume whole sequences, and hence are by nature non-incremental. A restart-incremental interface that repeatedly passes longer input prefixes can be used to obtain partial outputs, while providing the ability to revise. However, this method becomes costly as the sentence grows longer. In this work, we propose the Two-pass model for AdaPtIve Revision (TAPIR) and introduce a method to obtain an incremental supervision signal for learning an adaptive revision policy. Experimental results on sequence labelling show that our model has better incremental performance and faster inference speed compared to restart-incremental Transformers, while showing little degradation on full sequences.

## introduction incrementality is an inseparable aspect of language use. human speakers can produce utterances based on an incomplete message formed in their minds while simultaneously continuing to refine its content for subsequent speech production @xcite . they also comprehend language on (approximately) a word-by-word basis and do not need to wait until the utterance finishes to grasp its meaning @xcite . as observed by @xcite , a natural option for neural network-based incremental processing would be rnns @xcite , as they have essential properties the alert citizens the suspect person fear input controller incremental output (recomputed) output hypotheses det noun noun verb det noun noun det noun adj verb noun adj adj incremental pos tags from: https://demos.explosion.ai/displacy ? @xmath0: they keep a recurrent state, are sensitive to the notion of order and are able to accept partial input and produce an output at each time step. ideally, an incremental processor should also be able to revise its previous incorrect hypotheses based on new input prefixes @xcite . however, rnns are unable to do so as their output is monotonic. the transformer architecture @xcite has been the de facto standard for many nlp tasks since its inception. nevertheless, it is not designed for incremental processing as the input sequences are assumed to be complete and processed as a whole. a restart-incremental interface @xcite can be applied to adapt transformers for incremental processing @xcite , where available input prefixes are recomputed at each time step to produce partial outputs. such an interface also provides the capability to revise existing outputs through its non-monotonic nature. although feasible, this method does not scale well for long sequences since the number of required forward passes grows with the sequence length. 2 the revision process is also not effective as it occurs at every time step, even when it is unnecessary. revision is crucial in incremental processing, as it is not always possible for a model to be correct at the first attempt, either because the linguistic input is provided in its inherent piecemeal fashion (as shown in figure 1 ) or because of mistakes due to poor approximation. one way to improve the output quality is the delay strategy @xcite , where tokens within a lookahead window are used to disambiguate the currently processed input. however, it can neither fix past hypotheses nor capture long-range influences e.g. in garden path sentences. in this work, we propose the two-pass model for adaptive revision (tapir), which is capable of adaptive revision, while also being fast in incremental scenarios. this is achieved by using a revision policy to decide whether to write (produce a new output) or revise (refine existing outputs based on new evidence), whose mechanism is described in §3. learning this policy requires a supervision signal which is usually not present in non-incremental datasets @xcite . in §4, we tackle this issue by introducing a method for obtaining action sequences using the linear transformer (lt) @xcite . as silver labels, these action sequences allow us to view policy learning as a supervised problem. experiments on four nlu tasks in english, framed as sequence labelling foot_2 , show that, compared to a restart-incremental transformer encoder, our model is considerably faster for incremental inference with better incremental performance, while being comparable when processing full sequences. our in-depth analysis inspects tapir's incremental behaviour, showing its effectiveness at avoiding ill-timed revisions on correct prefixes. ## related work there has been increasing interest to explore neural network-based incremental processing. @xcite proposed a dialogue state tracker using lstm @xcite to incrementally predict each component of the dialogue state. @xcite introduced an incremental anaphora resolution model composed of a memory unit for entity tracking and a recurrent unit as the memory controller. rnns still fall short on non-incremental metrics due to their strict left-to-right processing. some works have attempted to address this issue by adapting bil-stms or transformers for incremental processing and applying it on sequence labelling and classification tasks @xcite and disfluency detection @xcite . our revision policy is closely related to the concept of policy in simultaneous translation, which decides whether to wait for another source token (read action) or to emit a target token (write action). simultaneous translation policies can be categorised into fixed and adaptive. an example of a fixed policy is the wait-k policy @xcite , which waits for first k source tokens before alternating between writing and reading a token. an adaptive policy on the other hand, decides to read or write depending on the available context and can be learned by using reinforcement learning techniques @xcite or applying monotonic attention @xcite @xcite . the memory mechanism is a key component for revision policy learning as it stores representations which, for instance, can be used to ensure that the action is correct @xcite . it also absorbs asynchronies that may arise when each component in an incremental system has different processing speed @xcite . the memory can be internal as in rnns, or external such as memory networks @xcite and the neural turing machine @xcite . revision in incremental systems has been previously explored. in simultaneous spoken language translation, @xcite proposed a scheme that allows re-translation when an asr component recognises a new word. @xcite evaluated streaming translation against re-translation models that translate from scratch for each incoming token and found that re-translation yields a comparable result to streaming systems. zheng et al. (2020) proposed a decoding method for simultaneous translation that overgenerates target words at each step, which are subsequently revised. one way to achieve revision is by employing a two-pass strategy. @xcite proposed a deliberation network for machine translation, composed of encoder-decoder architecture with an additional second-pass decoder to refine the generated target sentence. in dialogue domains, this strategy is also used to improve the contextual coherence and correctness of the response @xcite and to refine the output of retrieval-based dialogue systems @xcite . furthermore, the two-pass approach is commonly utilised in streaming asr to improve the initial hypothesis @xcite @xcite . the aforementioned works shared a common trait, as they used a fixed policy and performed revision either for each incoming input or when the input is already complete. our approach differs in that our model learns an adaptive policy that results in more timely revisions. contemporaneous to our work, @xcite proposed a cascaded uni-and bidirectional architecture with an additional module to predict when to restart. the module is trained with a supervision signal obtained from comparing the model's prediction against the ground truth. their approach is effective in reducing the required computational budget. ## model to address the weaknesses of rnn-and transformer-only architectures for incremental processing ( §1), we introduce a two-pass model for adaptive revision named tapir, which integrates advantages of both models and is based on the deliberation network @xcite . its architecture, depicted in figure 2 , consists of four components as follows: taking the inspiration from grave et al. (2017), we use caches, transformer reviser ȳt-2 ȳt-1 ȳt ht-2 yt-2 xt-2 ht-1 yt-1 xt-1 ht yt xt . . . . . . . . . . . . memory ctrl write revise input buffer output buffer 1st pass 2nd pass (optional) figure 2: tapir computes a candidate output using an rnn at each time step. then the controller decides whether to write by adding the new output to the output buffer or to take a revise action, which can edit the output buffer after observing the effect of the new input on past outputs with the help of the memory. during incremental inference, tapir computes a candidate output y t for the most recent input x t as the first pass. then, based on x t and the memory, it decides whether to take a write (add y t to an output buffer) or revise (perform a second pass to recompute all existing outputs) action. the action is defined by a revision policy π θ , which models the effect of new input on past outputs. at each time t, π θ makes use of processed inputs @xmath1. 4 it is parameterised by the controller hidden state k t with a non-linear function g: ## supervision signal for revision during incremental sentence comprehension, a revision or reanalysis occurs when disambiguating material rules out the current sentence interpretation. in figure 1 , noun is a valid label for suspect at @xmath2, but person at @xmath3, forcing a reanalysis to adj instead. training tapir's controller requires a sequence of write/revise actions expressed as the supervision signal a lt in equation ( 12 ), capturing when revision happens. this signal then allows us to frame the policy learning as a supervised learning task (as in the work of @xcite ). if we have the sequence of output prefix hypotheses at each step, as shown in figure 1 , we know that the steps when revisions have occurred are @xmath4, 7}. we can then construct the sequence of actions we need. the first action is always write as there is no past output to revise at this step. for @xmath5, the action can be determined by comparing the partial outputs at time step t (excluding y t ) against the partial outputs at time step t -1. if no edits occur, then the partial outputs after processing x t should not change, and a write action is appended to the sequence. if any edits occur, we append a revise action instead. intermediate human judgements about when to revise are not available, so we need to retrieve that from a model. it is possible obtain this information from a restart-incremental transformer, by comparing how the prefix at t differs from prefix at t -1. however, as shown by @xcite , the signal captured using this approach may lack incremental quality due to the missing recurrence mechanism. using a recurrent model is advisable here, as it can capture order and hierarchical structure in sentences, which is apparently hard for transformers @xcite @xcite . but it is difficult to retrieve this signal using vanilla rnns because its recurrence only allows a unidirectional information flow, which prevents a backward update of past outputs. therefore, we opt for the linear transformer (lt) @xcite , which can be viewed both as a transformer and as an rnn. 5 to generate the action sequences, we first train the action generator lt with causal mask to mimic an rnn training. afterwards, it is deployed under restart-incrementality on the same set used for training with the mask removed. we collect the sequence of partial prefixes for all sentences and use it to derive the action sequences. ## experiments datasets. we evaluate tapir on four tasks in english, for nlu and task-oriented dialogue, using seven sequence labelling datasets: alarm, reminder & weather @xcite and mit movie @xcite . we evaluate tapir on these three relevant di-mensions. for (i), we use similarity and diachronic metrics foot_5 proposed by @xcite and adapted in madureira and schlangen (2020): edit overhead (eo, the proportion of unnecessary edits over all edits), correction time score (ct, the average proportion of time steps required for an output increment to settle down), and relative correctness (rc, the proportion of output prefixes that match with the final output). aspect (ii) is analysed by benchmarking the incremental inference speed. for (iii), we use the f1 score adapted for the iob sequence labelling scheme, except for pos tagging, which is evaluated by measuring accuracy. rather than trying to beat the state-of-the art results, we focus on analysing the incremental abilities of models whose performances are high enough for our purposes. as a reference model, we use a transformer encoder applied in a restartincremental fashion, which implicitly performs revision at every step. we follow @xcite and @xcite by evaluating partial outputs with respect to the final output, to separate between incremental and nonincremental performance. delay strategy. to inspect the effect of right context on the model's performance, we use the delay strategy @xcite with a lookahead window of size 1 and 2, computing a delayed version of eo and rc @xcite . the output for the reference model is delayed only during inference, as in @xcite . for tapir, the same treatment would not be possible as it contains an rnn that must be able to recognise the output delay. thus, we follow the approach of @xcite : during training and inference, the label for input x t is expected at time step t + d, where d is the delay. implementation. for the reviser component, we choose transformer (trf) and linear transformer (lt) encoders trained with full attention. 7 the reference model is trained with cross entropy loss similar to the reviser. all models are trained with the adamw optimiser @xcite . we use 300-d glove embeddings @xcite , which, for the reference model and the reviser, are passed through an additional linear projection layer. the probability threshold τ is set to 0.5. we report results for a single run with the best hyperparameter configuration. see appendix for details about the set-up and experiments. ## results and analysis incremental. figure 3 depicts the incremental evaluation results. for the no-delay case, tapir performs better compared to the reference model. we also observe that the delay strategy helps improve the metrics. it improves the results for tapir, in general, but a longer delay does not always yield a better incremental performance. we suspect this happens for two possible reasons: first, if we consider the case where the delay is 1, tapir has already achieved relatively low eo (< 0.1) and high rc (> 0.85). this, combined with its nonmonotonic behaviour, might make it harder to further improve on both incremental metrics, even if a longer delay is allowed. second, a longer delay means that our model needs to wait longer before producing an output. in the meantime, it still has to process incoming tokens, which might cause some difficulty in learning the relation between the input and its corresponding delayed output. as a consequence, we have mixed results when comparing eo and rc for the delayed version of the reference model and tapir. their differences are, however, very small. tapir achieves low eo and ct score, which indicates that the partial output is stable and settles down quickly. rc is also high, which shows that, most of the time, the partial outputs are correct prefixes of the final, non-incremental output and would be useful for downstream processing. ## conclusion we proposed tapir, a two-pass model capable of performing adaptive revision in incremental scenarios e.g. for dialogue and interactive systems. we also demonstrated that it is possible to obtain an incremental supervision signal using the linear transformer (lt), in the form of write/revise action sequences, to guide the policy learning for adaptive revision. results on sequence labelling tasks showed that tapir has a better incremental performance than a restart-incremental transformer, in general, while being roughly comparable to it on full sentences. the delay strategy helps to improve incremental and non-incremental metrics, although a longer delay does not always yield better results. the ability to revise adaptively provides our model with substantial advantages over using rnns or restart-incremental transformers. it can fix incorrect past outputs after observing incoming inputs, which is not possible for rnns. looking from the aspect of efficiency, our model is also better compared to restart-incremental transformers as the recomputation is only performed when the need for it is detected. tapir is consequently faster in terms of inference speed.