LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding (2025)

Abstract: Large Language Models (LLMs) are primarily designed for batch processing. Existing methods for adapting LLMs to streaming rely either on expensive re-encoding or specialized architectures with limited scalability. This work identifies three key mismatches in adapting batch-oriented LLMs to streaming: (1) input-attention, (2) output-attention, and (3) position-ID mismatches. While it is commonly assumed that the latter two mismatches require frequent re-encoding, our analysis reveals that only the input-attention mismatch significantly impacts performance, indicating re-encoding outputs is largely unnecessary. To better understand this discrepancy with the common assumption,we provide the first comprehensive analysis of the impact of position encoding on LLMs in streaming, showing that preserving relative positions within source and target contexts is more critical than maintaining absolute order. Motivated by the above analysis, we introduce a group position encoding paradigm built on batch architectures to enhance consistency between streaming and batch modes. Extensive experiments on cross-lingual and cross-modal tasks demonstrate that our method outperforms existing approaches. Our method requires no architectural modifications, exhibits strong generalization in both streaming and batch modes. The code is available at repository https://github.com/EIT-NLP/StreamingLLM.

## introduction large language models (llms) have revolutionized a multitude of tasks @xcite @xcite @xcite . however, research on llms has largely focused on batch-processing, where the entire input is processed at once @xcite . in contrast, human cognition operates incrementally, interpreting information as it ar-* corresponding author rives-a capability essential for real-time decisionmaking, interactive dialogue, and other latencysensitive applications @xcite . bridging this gap between batch-oriented llms and streaming-aware processing is vital for unlocking their potential in dynamic, real-world scenarios. a naive strategy to adapt llms for streaming involves iteratively re-encoding both new inputs and prior outputs with each incoming data segment @xcite @xcite , as illustrated in figure 1 @xcite . while this batch-streaming paradigm preserves compatibility with batch-processing architectures, it introduces prohibitive computational costs. existing efforts to optimize llms for streaming data typically fall into two categories: (1) directly encoding streaming data in arrival order @xcite , an interleavedstreaming paradigm, which introduces structural mismatches with batch-processing setups used in pre-training and degrades performance; (2) designing entirely new architectures tailored to the streaming mode @xcite @xcite , which is costly, lacks scalability, and fails to fully leverage pre-trained llm capabilities. furthermore, existing methods lack rigorous analysis of the fundamental discrepancies between batch and streaming processing modes. this work tackles these limitations by identifying three key mismatches in adapting batchoriented llms to streaming, as shown in figure 1: • input-attention mismatch:batch-streaming confines input tokens to attending only prior inputs, whereas interleaved-streaming permits attention to previously decoded outputs. building on the identification of these mismatches, we systematically studied their effects on llm performance. our analysis revealed that input-attention mismatch does affect streaming model performance. in contrast, output-attention and position-id mismatches have negligible effects. a common assumption is that streaming models require re-encoding of previously generated content to mitigate token position inconsistencies arising from the incremental nature of streaming setting @xcite @xcite , as shown in figure 1 @xcite . however, our empirical findings do not support this hypothesis. instead, we observe that re-encoding the output is not necessary foot_0 . this discrepancy with the common assumption raises a fundamental question: how does position encoding impact llms in streaming scenarios? and how should we design appropriate position encoding for streaming llms? existing research on positional encoding in llms has largely focused on static scenarios @xcite @xcite , while its role in streaming scenarios remains underexplored. we conducted a more in-depth analysis to further explore the impact of position encoding on streaming models. experimental results reveal that the absolute positional order of tokens has a negligible effect on model performance in streaming tasks. however, maintaining the internal relative order within the source and target sequences is significantly more important. based on the findings, we propose a grouped position encoding streaming paradigm built on batch architectures(groupstreaming), which groups input and output position ids to enable more consistent processing with the batch model. this strategy is not only computationally efficient but also generalizable across different tasks and model architectures. we validated its effectiveness on cross-lingual (machine translation) and cross-modal (automatic speech recognition) tasks, demonstrating that it significantly outperforms existing solutions, including llms with more complex streaming-optimized architectures. the main contributions of this study can be summarized as: (1) we systematically analyze the mismatches between batch and streaming processing in llms, providing deep insights into key factors affecting their adaptation to streaming. contrary to mainstream assumptions, our experiments reveal that position disorder is not the primary factor affecting llm streaming performance. (2) we conduct the first comprehensive study on the impact of position encoding in streaming scenarios, demonstrating that absolute positional order is unnecessary, while maintaining relative order within source and target contexts is more critical. (3) we introduce a group streaming paradigm for streaming llms. this method imposes no architectural constraints on batch-processing llms, allowing seamless application to any pre-trained llm while ensuring high scalability and adaptability to various real-world streaming tasks. ## streaming-batch mismatches llms are pre-trained in a batch-processing paradigm, where the entire input sequence @xmath0, . . . , x n ] is processed simultaneously to generate the output sequence @xmath1, . . . , y m ]. this paradigm assumes full input availability, allowing both self-attention and cross-attention mechanisms to operate over complete sequences. in contrast, streaming tasks require incremental processing, where inputs and outputs arrive and are processed in an interleaved manner over time. at any time step t, the model only has access to a partial input sequence x @xmath2, . . . , x t ] and generates a corresponding partial output sequence y t ′ = [y 1 , . . . , y t ′ ]. this shift from batch to streaming introduces three key mismatches: input-attention mismatch in batch-streaming mode, self-attention enforces a strict ordering, where each input token x i can only attend to prior inputs @xmath3. this is typically expressed as: this disrupts the model's pre-trained assumptions, as input tokens in batch mode never attend to output, potentially leading to degraded performance. ## impact analysis of mismatches applying batch-trained llms to streaming mode introduces structural mismatches. existing research has not systematically analyzed the nature of these mismatches between streaming and batchprocessing. we employ a stepwise ablation approach to systematically isolate each mismatch and assess its impact on streaming task performance. ## impact analysis of position encoding the above analysis suggests that positional mismatches do not significantly impact the performance of streaming tasks. to further elucidate this phenomenon, this section provides a detailed investigation into the impact of positional encoding on the performance of llms in streaming scenarios. ## discussion why llms? we apply the proposed groupstreaming approach to mainstream large language models and compare its performance against other decoder-only streaming models to highlight its advantages. to demonstrate the effectiveness of our method, we evaluate it on the en-fr and en-de translation tasks from the iwslt-17 dataset, as well as the asr task from the librispeech dataset. the baselines for text translation include simul-mask @xcite and dst @xcite , while the baselines for asr include caat @xcite and wav2vec-s @xcite . as shown in figure 4 , the vertical axis represents task-specific performance metrics-bleu for translation and wer for asr-while the horizontal axis indicates the model's average latency (al and laal), measured by the number of waited words in translation and the waiting time in asr. the results show that group-streaming llms consistently outperform specialized decoder-only baselines, typically achieving higher accuracy under the same latency conditions. ## related work streaming language and speech transformers a typical implementation of transformer-based streaming tasks adopts an incremental encoding strategy on the encoder side and an incremental decoding strategy on the decoder side @xcite @xcite . with the rise of large language models, researchers have begun exploring how to adapt decoder-only architectures for streaming tasks. among these approaches, batch-streaming models based on prompt structures attempt to approximate offline batch processing by re-encoding tokens during streaming inference @xcite @xcite . some studies suggest that position confusion in streaming environments is a key factor necessitating reencoding in llms @xcite . to address this issue, one line of research focuses on modifying the decoder-only architecture to enhance its adaptability to streaming tasks @xcite @xcite , while another emphasizes optimizing positional encoding-such as the alibi positional gemma2-2b phi3-mini-3.8b llama3.1-8b 0.0 10.0 20.0 30.0 40.0 50.0 60.0 bleu batch-processing translation task on iwslt-17 en-fr position empty @xmath4.5 = 128 = 256 = 512 figure 5: the bleu performance of batch-processing translation task on iwslt-17 en-fr dataset. position encoding in transformers position encoding @xcite @xcite ) is a crucial component of transformer models @xcite , designed to break the permutation-invariant nature of self-attention mechanisms. recent studies have demonstrated that decoder-only transformer models can still capture positional information even in the absence of explicit positional encoding @xcite . a plausible explanation is that causal attention masks enforce position-dependent token interactions, implicitly encoding positional information @xcite . related re-search has shown that in tasks such as speech modeling @xcite and language modeling @xcite , decoder-only transformers without positional encoding can achieve performance comparable to standard decoder-based transformers. furthermore, other studies @xcite have indicated that the generalization ability of transformers without positional encoding does not degrade significantly when handling varying context lengths. while significant progress has been made in understanding positional encoding in llms, existing research has primarily focused on static scenarios. in contrast, the role of positional encoding in streaming scenarios remains underexplored, where the dynamic modeling of positional information may follow different patterns and exert distinct effects. ## conclusion this work provides a systematic analysis of the mismatches that arise in adapting batch-trained llms to streaming tasks. we identify input-attention mismatch as the primary bottleneck, while outputattention and position-id mismatches have negligible impact, challenging the prevailing assumption that position inconsistencies necessitate frequent re-encoding. to clarify this, we conduct the first in-depth analysis of position encoding in streaming settings, showing that preserving strict absolute positions is unnecessary; instead, maintaining relative token order within source and target contexts is more critical. building on the insights, we propose the group streaming paradigm, a simple yet effective strategy that bridges the gap between streaming and batch modes without requiring re-encoding. the approach is model-agnostic and generalizable, achieving strong performance across both crosslingual and cross-modal streaming tasks.