Efficient Encoders for Streaming Sequence Tagging (2023)

Abstract: A naive application of state-of-the-art bidirectional encoders for streaming sequence tagging would require encoding each token from scratch for each new token in an incremental streaming input (like transcribed speech). The lack of re-usability of previous computation leads to a higher number of Floating Point Operations (or FLOPs) and higher number of unnecessary label flips. Increased FLOPs consequently lead to higher wall-clock time and increased label flipping leads to poorer streaming performance. In this work, we present a Hybrid Encoder with Adaptive Restart (HEAR) that addresses these issues while maintaining the performance of bidirectional encoders over the offline (or complete) and improving streaming (or incomplete) inputs. HEAR has a Hybrid unidirectional-bidirectional encoder architecture to perform sequence tagging, along with an Adaptive Restart Module (ARM) to selectively guide the restart of bidirectional portion of the encoder. Across four sequence tagging tasks, HEAR offers FLOP savings in streaming settings upto 71.1% and also outperforms bidirectional encoders for streaming predictions by upto +10% streaming exact match.

## introduction state-of-the-art text encoding methods assume the offline setting, where the entire input text is available when encoding it. this differs from the streaming setting where the input text grows over time (such as transcribed speech or a typed query) @xcite @xcite . processing streaming input incrementally can enable suggestions on partial inputs @xcite , reduced final latency @xcite and lead to more interactive nlu agents @xcite . existing state-of-the-art bidirectional encoders (such as @xcite ) are computationally expensive for streaming processing. when a new token is received, these models require a restart, i.e., re-computation of representation of each token by re-running the bidirectional layer @xcite . this adds to the computational cost (i.e., higher flops) during streaming and leads to higher wall clock time. another limitation of these encoders is poorer generalization to partial inputs @xcite , stemming from these models being trained only on complete (and offline) inputs. despite these disadvantages, such bidirectional models offer better offline performance than unidirectional models across several nlp tasks like sequence tagging @xcite . we address these issues by proposing hear -hybrid encoder with adaptive restarts, where a separate adaptive restart module (arm) predicts when to restart the encoder. the encoder in hear is a hybrid encoder that reduces the computational cost of running the models in streaming settings. in a hybrid encoder, the earlier layers are unidirectional and the deeper layers are bidirectional. this design allows early contextualization, and limits the need for restart to the later layers. while hybrid encoder reduce streaming computational overhead, restarting them at every new token may not be required. thus, we propose a lightweight adaptive restart module (arm) to guide restarts, by predicting whether restarting the bidirectional layers of the hybrid encoder will benefit the model performance. this module is also compatible with fully bidirectional encoders. following are our key contributions: ## streaming sequence tagging in the streaming sequence tagging task, we assume that at time 1 ≤ @xmath0, we have received the first t tokens x @xmath1, • • • x t ) as input from a stream of transcribed speech or user-typed input. 1 the model then predicts the tags for all t tokens ŷt = (ŷ 1,t , ŷ2,t • • • ŷt,t ). the models predictions ŷn over the offline input @xmath2 however, in the streaming settings, we are concerned with predicted label sequences over all of the prefixes during training we only have access to the offline ground-truth label sequence over the offline input sequence y * n , even though ground truth labels for tokens may change as additional context is received in the future timesteps. ## streaming sequence tagging with hear in order to motivate hear, consider how existing bidi encoders would be used in streaming sequence tagging. at time t and the input sequence x t , the model restarts to predicts the label sequence ŷt i.e. it re-computes all its layers for all tokens, without leveraging any of its previous computation or any previously predicted labels. consequently, running a typical o(n 2 ) encoder would have o(n 3 ) computations in streaming settings, over n input tokens. therefore, naively using existing bidi encoders for streaming settings is highly inefficient. previous works tackle inefficiency by modifying the streaming model to infer only once for each word, after k future tokens have been received @xcite . this leads to a k-delayed output with possibility for revisions after additional future words have been received. however, this may lead to poor performance for tasks involving long-range dependency (e.g., srl) and a higher output lag. hear is a system consisting of a trained hy-brid encoder model and an arm that is trained over the hybrid encoder. the early unidirectional layers in hybrid encoder, reduces the computational cost of restart of the encoder. the arm guides when to restart the hybrid encoder. it saves computational cost by keeping the model from restarting at each timestep and also improves streaming performance by avoiding unnecessary label flips stemming from avoidable restarts. figure 2 shows running of hear in streaming settings over an example. for each new token in the streaming input, all the unidirectional layers and a part of first bidirectional layer is ran for the token, to obtain its unidirectional encodings and updated cache. these are then used by arm to predict whether to restart the bidirectional layer or not. if the bidirectional layer is to be restarted, then we obtain updated labels for all the tokens received in the stream. otherwise, the auxiliary predictor is ran over the unidirectional encoder for the current token to obtain its label and the labels from previous timestep is copied for all other tokens. we now formally introduce hybrid encoder ( §3.1), followed by arm ( §3.2). ## experimental setup datasets and tasks. we consider four common sequence tagging tasks -slot filling over snips @xcite , semantic role labelling (srl) over ontonotes @xcite , and named entity recognition (ner) and chunking (chunk) over @xcite . the standard train, dev and test splits are used for all datasets. ## experimental results in this section, we provide empirical results to answer the following questions: 4 shows the offline f1 scores of the hybrid encoder, unidi and bidi encoders across the four tasks. the hybrid encoder has similar offline f1 as bidi encoders. in fact, on ner, it even outperforms it by a margin of 0.8 f1 score. as expected, the unidi model performs poorly compared to the bidi encoder across all the tasks. from here on, we omit unidi encoder. ## analysis and ablations in this section, we perform various analysis pertaining to the hybrid encoder and arm. ## related work streaming (or incremental) setting has been widely studied in machine translation and parsing, dating as far back as two decades @xcite . specifically, for incremental parsing, there are two broad approaches that have been studied: transition-based and graphbased. transition-based incremental parsers allow for limited backtracking and correcting parsing over prefixes by keeping track of multiple parse segments @xcite or via beams search @xcite . such methods can fail on garden-path sentences and long-range dependencies and only work proficiently with a large beam. graph-based parsers incrementally assign scores to edges of the graph, discarding those edges that cause conflicts to tree-structure of the graph. @xcite proposed an attach-juxtapose system to grow the tree, requiring restart at each new token over streaming input. @xcite improved on its efficiency by proposing a information bottleneck. however, these methods rely on the structured nature of parsing output can not be adapted to incremental sequence tagging tasks without restarting at each token. recently, madureira and schlangen (2020) benchmarked the modern encoders on streaming sequence labelling and observed poor streaming performance of pretrained transformer models. they explored improvement strategies by adopting techniques from other streaming streaming like chunked training @xcite , truncated training @xcite (training model on heuristically-aligned partial input-output pairs), and prophecy @xcite (autocompleting input using a separate language model). they observed performance degradation from truncated training unless used with prophecies. however, running a language model at each timestep for prophecy is computationally infeasible. therefore, unlike our approach, neither methods can improve performance feasibly. previous works have attempted to improve computation efficiency in bidi encoders. monotonic attention moves away from o(n 2 ) soft attention overhead by restricting attention to monotonically increase across timesteps @xcite @xcite . however, unlike hear, such attempts can't maintain offline performance. recently, @xcite used linear transformer @xcite as unidirectional model using masking for streaming sequence tagging and classification. this approach performs well under the assumption of delayed output @xcite , a relaxation, where the model waits for additional tokens before predicting. furthermore, the unidirectional model could not revise its output, rendering the model incapable of handling long-range dependency or tasks that go backward like srlan ability common to any model with some bidirectionality, such as hear. similar drawbacks were in the partial bidirectional encoder, a bidirectional attention with restricted window, proposed by @xcite . improving efficiency through adaptive computing has been independently studied for reasoningbased tasks @xcite ), text generation models @xcite and diffusion models @xcite . these works are restricted to offline settings and can be readily incorporated within the proposed overall approach of hear. ## conclusion we propose hear for sequence tagging in streaming setting where the input is received one token at a time to the model. the encoder in our model is hybrid encoder where early layers are unidirectional and later are bidirectional. it reduces the computational cost in streaming settings, by reducing the need of restart only to the later bidirectional layers while preserving the offline performance of the model. hear additionally consists of an arm to predict when to restart the model. using arm leads to reduced number of restart of the encoder, leading to better streaming performance and further savings in computation. compared to bidi encoders, our model, hear (hybrid encoder + arm) reduces the computation by upto 71% in streaming settings while maintaining the performance of the bidi encoders across various sequence tagging tasks. hear improved streaming em by upto +10.0% and reduced unnecessary edits by upto -12.0%.