Streaming Sequence Transduction through Dynamic Compression (2025)

Abstract: We introduce STAR (Stream Transduction with Anchor Representations), a novel Transformer-based model designed for efficient sequence-to-sequence transduction over streams. STAR dynamically segments input streams to create compressed anchor representations, achieving nearly lossless (12x) compression in Automatic Speech Recognition (ASR) and outperforming existing methods. Moreover, STAR demonstrates superior segmentation and latency-quality trade-offs in simultaneous Speech Translation, optimizing latency, memory footprint, and quality.

## introduction sequence transduction, also referred to as sequenceto-sequence modeling, has shown remarkable success across various domains, including speech translation @xcite @xcite and automatic speech recognition @xcite @xcite . traditionally, these models operate under the assumption of fully observing input sequences before generating outputs. however, this requirement becomes impractical in applications necessitating low latency or real-time output generation such as simultaneous translation @xcite @xcite . the concept of streaming sequence transduction @xcite @xcite @xcite , or stream transduction, arises to address this challenge. unlike traditional sequence transduction, stream transduction operates on partially observed input sequences while simultaneously generating outputs. this requires deciding when to initiate output generation, a task inherently tied to identifying critical triggers within 1 codes available at: https://github.com/ steventan0110/star figure 1 : when yield is triggered, the current segment's information is compressed into an anchor representation to generate the next output. the input sequence. triggers mark moments when sufficient input information has been received to initiate output generation, thus minimizing latency. consequently, they partition the input sequence into discrete segments, with outputs accessing only information preceding each trigger. locating these triggers poses a significant challenge. prior approaches have explored methods that employ fixed sliding windows to determine triggers @xcite , or learning models to predict triggers @xcite ), yet timing remains a complex issue. beyond reducing latency, another challenge for stream transduction is how to efficiently represent historical information while optimizing memory usage. prior work @xcite @xcite has mostly focused on improving the efficiency of transformer but does not investigate streaming scenarios. reducing the memory footprint for streaming systems introduces additional complexity as models must determine when certain information becomes less relevant for future predictions. in this work, we propose stream transduction with anchor representations (star), a novel ap-proach designed to maximize the benefits of stream transduction, optimizing both generation latency and memory footprint. star dynamically segments the input stream into buffers that contain similar levels of information. then, it introduces the concept of anchors, which aggregate a buffer of information (multiple vector representations) into single-vector anchor representations. once an anchor representation is yielded, it triggers the generation process to yield another token. we present a learning strategy to train star end-to-end so that the model learns to dynamically select anchor positions with the following objectives: (1) anchor positions are selected such that each segment contains the right amount of information for generating the next output; (2) anchor representation effectively compress the information of its preceding segment. for example, in fig. 1 , the model triggers yield at index 3 (which makes it an anchor position), compressing the information of the current chunk @xmath0, x 2 , x 3 ) into anchor representation z 1 to generate output y 1 . such a process repeats each time yield is triggered. to summarize, our contributions are as follows: (1) we propose star that dynamically segments and compresses input streams, tradingoff among latency, memory footprint, and performance for stream transduction; (2) we validate the effectiveness of our approach on well-established speech-to-text tasks. our results show that star greatly outperforms existing methods, obtaining better compression ability and excelling in qualitylatency trade-offs. ## experiments: non-streaming compression we experiment on the non-streaming asr task to better demonstrate the effectiveness of our selection-based compression method, since we do not need to consider the quality-latency trade-off as in the streaming scenario. we compare our method with other common baselines like convolutional neural networks @xcite and continuous integrate and fire @xcite . ## related work end-to-end streaming speech-to-text for streaming/simultaneous speech-to-text tasks, learning speech representation and policies for read and yield is essential. previous methods like rnn-transducer @xcite and connectionist temporal classification (ctc) @xcite leverage monotonic alignment for low error rate transcription. recent work @xcite further extends transformers for streaming asr using modified attention and beam search. for speech translation, @xcite proposed the wait-k strategy with a fixed decision policy that read chunks of equal-length text for decoding @xcite adapted the wait-k strategy for simultaneous speech translation. instead of a fixed decision policy, simulspeech @xcite trained segmenters with ctc loss. @xcite also use ctc for guidance on word boundary learns to shrink the representation and proposes the wait-k-stride-n strategy that writes n tokens for each read action. @xcite and @xcite use cif to learn segmentation for the speech sequences and trigger the yield action whenever cif fire a new representation. additionally, @xcite and @xcite support a more adaptive strategy where dynamic read and yield are possible. however, even for such an adaptive strategy, a good decision policy still matters @xcite . efficient methods for transformers prior work studied efficient methods to scale transformers to long sequences @xcite , including sparse patterns @xcite , recurrence @xcite , kernelized attentions @xcite , etc. some of them can be applied in the streaming settings, such as streaming llms @xcite , compressive transformers @xcite ; @xcite proposed to apply knn to the attention to select a subset of past tokens, akin to the segmentation process in this paper. similar to the residual connection in our paper, nugget @xcite trains a scorer to select a subset of tokens to represent texts. more recently, @xcite and @xcite also combine context compression with efficient fine-tuning methods like lora @xcite to expand context length for large language models. speech representation traditionally, acoustic features are extracted by filter-bank features, melfrequency cepstral coefficients, or bottleneck features @xcite . more recent work relies on self-supervision to learn speech representations. for example, @xcite and @xcite learn acoustic representation by reconstructing the original audio. to learn semantic representation, masked language modeling, and contrastive learning objectives are popularized by widely used representations from hubert @xcite , w2v-bert @xcite and wav2vec @xcite . all these models use cnns as a building block to downsample speech signals/representations. ## conclusion and future work we introduce star, a model designed for dynamic compression and transduction of streams. star features a segmenter learned via encoder-decoder cross-attention and employs a selection-based compression approach. our experiments across multiple speech-to-text tasks confirm star's superior compression performance and latency-quality trade-off relative to established methods such as convolutional neural networks and continuous integrate-and-fire. in the future, we hope to extend this framework to facilitate streaming nonautoregressive generation. 13 b continuous integrate and fire figure 12: illustration of continuous integrate and fire. continuous integrate and fire @xcite predicts a score for each position and dynamically aggregates the semantic representation. as shown in fig. 12 , cif first computes a list of scores α similar to our proposed method. then, starting from the first position, it accumulates the scores (and representation) until reaching a pre-defined threshold foot_2 β. once reaching the threshold, it fire the accumulated representation and starts to accumulate again. as shown in fig. 12 , suppose we originally have representation @xmath1, • • • , z 6 ) with corresponding scores α = (α 1 , • • • , α 6 ). suppose we reach the threshold at @xmath2, i.e., α 1 + α 2 + α 3 >= β, then we fire the representation by taking the weighted average of score and representation here c 1 becomes the compressed representation for the region @xmath3, 3]. note that since α 1 + α 2 + α 3 >= β, we have residual score α 3r = α 1 + α 2 + α 3β, which is left for future accumulation, and we only use α 3l = α 3α 3r when weighting representation z 3 . more generally, suppose the previous fire occurs at position j and at current step i the accumulated score reaches the threshold, the aggregated representation is computed as to enforce the compression rate r, we follow @xcite to re-scale the predicted scores so that the threshold β is reached t y times when accumulating the scores: here σ is the sigmoid function and n is the normalization term (summation of un-scaled scores) and n * denotes the number of desired selections, i.e., n * = t y . we assume the input feature is longer than the output (t @xmath4, so re-scaling scores to yield t y means we employ a dynamic compression rate @xmath5. note that t y is only observed during training and we cannot re-scale s in test time. therefore, we adopt a length penalty loss @xcite during training to regularize the segmenter to ensure proper learning of segmentations: finally, our training objective is the combination of negative log-likelihood and length penalty loss: in practice, the segmenter is only trained for a few thousand steps (so is the length penalty loss) and we set γ = 0.01. our method is fundamentally different because of how we treat the scorer and how we perform compression. in cif, the compression is performed as an aggregation (weighted average) within each segmented block (decided by the scores and threshold). in star, we directly take out representations and we force the semantic encoder to condense information to those important positions. in other words, we did not explicitly perform aggregation like cif but expect the semantic encoder to learn such aggregation innately through training. another key difference is how the scorer is learned. in cif, the weighted average with scores and representation allows a gradient to flow through the scorer. for star, we inject the scores into crossattention to update the scorer. the major advantage of our approach is that the importance of position is judged by the attention from the decoder to the encoder representation, which helps segment the speech representation in the way that the text decoder perceives it. for more details, we direct readers to the prior work @xcite @xcite .