D o CIA : An Online Document-Level Context Incorporation Agent for Speech Translation (2025)

Abstract: Document-level context is crucial for handling discourse challenges in text-to-text document-level machine translation (MT). Despite the increased discourse challenges introduced by noise from automatic speech recognition (ASR), the integration of document-level context in speech translation (ST) remains insufficiently explored. In this paper, we develop DoCIA, an online framework that enhances ST performance by incorporating document-level context. DoCIA decomposes the ST pipeline into four stages. Document-level context is integrated into the ASR refinement, MT, and MT refinement stages through auxiliary LLM (large language model)-based modules. Furthermore, DoCIA leverages document-level information in a multi-level manner while minimizing computational overhead. Additionally, a simple yet effective determination mechanism is introduced to prevent hallucinations from excessive refinement, ensuring the reliability of the final results. Experimental results show that DoCIA significantly outperforms traditional ST baselines in both sentence and discourse metrics across four LLMs, demonstrating its effectiveness in improving ST performance.

## introduction speech translation (st) involves translating spoken language into written text in a different language. despite significant progress in recent years @xcite @xcite @xcite , incorporating document-level context into st remains a major challenge due to the cross-modal nature of the task. this paper shifts the focus to documentlevel context 2 and examines how it can enhance machine translation (mt) when combined with automatic speech recognition (asr) in the cascaded st systems. in a traditional cascaded st system, as shown in figure 1 (a), the asr and mt models operate independently at the segment level. this leads to significant discourse-level issues due to the absence of inter-sentence context. these challenges become even more pronounced in st, where asr errors-such as misrecognizing entity pronouns or handling disfluencies-further complicate the translation process. incorporating document-level context offers two key advantages: first, it can potentially correct asr transcription errors by providing a broader understanding of the context; second, when integrated into the mt model, it helps address discourse phenomena such as entity inconsistencies, coreference resolution, and longrange dependencies @xcite @xcite @xcite . to fully leverage document-level context, we introduce docia-document-level context incorporation agent-an online framework specifically designed to improve st performance by incorporating document-level context. end-to-end st systems, which directly translate source-language speech into target-language text, can reduce the propagation of asr errors. however, these systems suffer from limited interpretability and the challenge of scarce parallel st data, making it expensive to develop a reliable and effective end-to-end solution. in contrast, the cascading approach-especially with the emergence of powerful large language models (llms) @xcite @xcite -provides a more efficient and flexible alternative. the cascading model enables modular optimization in st, allowing llms to be used to enhance performance at various stages of the process. as shown in figure 1 (b), docia takes full advantage of the scalability and flexibility inherent in the cascading approach by breaking the st process into four key stages: asr, asr refinement, mt, and mt refinement. document-level context is incorporated during the latter three stages (asr refinement, mt, and mt refinement), improving both transcription and translation through llm-based agents. crucially, the document-level context is updated online as each segment is processed, ensuring that the context remains current and relevant throughout the translation process. in addition, we employ two techniques -a multi-level context integration strategy and a refinement determination mechanism -to enhance the performance of docia. first, while documentlevel context can be beneficial, it often includes redundant information, with only a small portion being relevant to discourse issues @xcite . using all available context indiscriminately can even degrade st performance and increase computational overhead. to address this, we propose a multi-level context integration strategy that retains the advantages of document-level context while reducing redundancy. second, our two refinement stages are designed to resolve inter-segment inconsistencies using document-level context. in most cases, minimal adjustments are sufficient to address discourse-related issues, as extensive changes may introduce errors such as hallucinations or semantic distortions. to minimize these risks, we introduce a determination mechanism that ensures the refined text remains consistent with the original semantics, improving the output without introducing undesirable changes. overall, the main contributions of this paper are summarized as follows: ## docia: document-level context incorporation agent we propose docia, an online agent designed to enhance speech translation (st) by effectively leveraging document-level context. docia operates in a cascaded four-stage process: asr, asr refinement, translation, and translation refinement. document-level context is incorporated during the asr refinement, translation, and translation refinement stages (section 2.1). to optimize context utilization, we introduce a multi-level integration strategy, splitting the context into short-and longmemory components (section 2.2). to prevent hallucinations during refinement, we also propose an effective determination mechanism (section 2.3). ## experimentation in this section, we validate the effectiveness of docia on five st transation tasks. ## discussion and analysis in this section, we use the en ñ de and en ñ ru tasks, with llama-3.1-8b and gpt-4o-mini, as representative examples to explore how docia (i.e., docia a-m-p in ## conclusion inspired by the success of incorporating documentlevel context in text-to-text mt, we propose do-cia, an online llm-based agent designed to improve st performance by integrating documentlevel context. docia breaks the whole st process into four stages, producing the final translation in a cascading manner. additionally, we introduce a multi-level context integration strategy and a refinement determination mechanism to enhance docia's ability to utilize inter-segment context while minimizing hallucinations during refinement. experimental results across five st tasks, using four different llms, demonstrate that docia effectively addresses discourse issues from both the asr and mt stages, leading to significant improvements in overall st quality. pei zhang, niyu ge, boxing chen, and kai fan. 2019a. lattice transformer for speech translation. in proceedings of acl, pages 6475-6484. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2019b. bertscore: evaluating text generation with bert. in proceedings of iclr. ziqiang zhang, long zhou, junyi ao, shujie liu, lirong dai, jinyu li, and furu wei. 2022b. speechut: bridging speech and text with hiddenunit for encoder-decoder based speech-text pretraining. in proceedings of emnlp, pages 1663-1676. yan zhou, qingkai fang, and yang feng. 2023. cmot: cross-modal mixup via optimal transport for speech translation. in proceedings of acl, pages 7873-7887.