Investigating Hallucinations in Simultaneous Machine Translation: Knowledge Distillation Solution and Components Analysis (2025)

Abstract: Simultaneous Machine Translation (SiMT) generates target translation before receiving the whole source sentence and faces a serious hallucination problem. In contrast, traditional offline machine translation (OMT) models exhibit significantly fewer hallucinations. Motivated by this disparity, we propose Knowledge Distillation for SiMT (KD-SiMT), a simple yet effective method that utilizes the OMT model to mitigate hallucinations in SiMT. Experiments on Zh \rightarrow En and De \rightarrow En tasks demonstrate that KD-SiMT effectively reduces hallucinations and enhances the SiMT performance. Furthermore, we systematically investigate the deficiencies in SiMT models related to serious hallucinations and the effect of KD-SiMT. Specifically, we design targeted tasks and metrics to quantitatively evaluate the components in SiMT models from the perspectives of model structure and knowledge acquisition. Our analyses reveal that inaccurate source representations and imbalanced cross-attention are more likely to occur in SiMT models when generating hallucinations, while KD-SiMT alleviates these issues. Besides, we find that KD-SiMT equips SiMT models with sufficient faithfulness knowledge in training, thus reducing hallucinations.

## introduction simultaneous machine translation (simt) aims to generate target translation before receiving the whole source sentence, which acquires models to learn both translation ability and a read/write policy that decides between outputting a target word and waiting for a new source word @xcite @xcite @xcite . however, this complex training objective hinders the simt models from learning translation ability, thus triggering serious hallucinations (han * simt model and omt model. we choose wait-k @xcite and mma @xcite as representative simt models. this observed performance gap motivates our approach. in this paper, we propose knowledge distillation for simt (kd-simt), a simple yet effective method that utilizes omt models to reduce hallucinations in simt models. kd-simt enhances the simt model by incorporating additional supervised signals derived from the hidden representations and output probabilities of the omt model. experiments on @xmath0. furthermore, we decouple simt models into distinct components for individual analysis to identify deficiencies in simt models associated with hallucinations and to understand how kd-simt reduces these hallucinations. specific tasks and metrics are designed for quantitative assessment. our analyses involve examining the simt models from two primary perspectives: model structure and knowledge acquisition. from the perspective of model structure, we introduce the auto-encoder task and our defined contribution standard deviation (csd) to evaluate the encoder and decoder in simt models separately. our evaluation reveals that inaccurate source representations and imbalanced cross-attention assignment are more likely to occur in simt models, and kd-simt effectively addresses these issues within the respective components. from the perspective of knowledge acquisition, we categorize the acquired knowledge into fluency knowledge and faithfulness knowledge. we find that simt models acquire adequate fluency knowledge but limited faithfulness knowledge, which is crucial for faithful translation. kd-simt can provide sufficient faithfulness knowledge for simt models during training. our contributions can be summarized as follows: simt model structure existing simt models are commonly based on transformer @xcite , which are formalized as: ## knowledge distillation solution we propose kd-simt, which utilizes the omt model as the teacher and the simt model as the student to reduce hallucinations in simt. figure 2 provides an overview of our method. further details are introduced below. ## components analysis for simt models to further investigate hallucinations in simt, we decouple the simt model into components and analyze their performances on @xmath1. from the perspective of model structure, we separately evaluate the encoder and decoder in the simt model. from the perspective of knowledge acquisition, we categorize the knowledge obtained in the simt model into fluency knowledge and fluent knowledge for individual analysis. for each component, specific tasks and metrics are designed. with these metrics, we can identify the specific deficiencies in each component when the simt model generates hallucinations (sec.5.1) and ana-lyze the impacts of kd-simt on these components (sec.5.2). besides, ablation studies and case studies are also conducted to further validate the effectiveness of kd-simt in reducing hallucinations of simt models (sec.5.3 and 5.4). ## conclusion in this paper, we focus on investigating serious hallucinations in simt. we propose kd-simt to reduce hallucinations in simt models. experiments show that kd-simt is effective and achieves significant improvements. furthermore, we conduct component analysis to explore the deficiencies in simt models related to hallucinations and the effect of kd-simt on the simt models. from the perspective of model structure, we find that kd-simt enhances the source representations and cross-attention assignments in simt models. from the perspective of knowledge acquisition, our analyses suggest that kd-simt equips simt models with sufficient faithfulness knowledge during training, thus improving translation quality.