The HW - TSC ’s Simultaneous Speech-to-Text Translation System for IWSLT 2023 Evaluation (2023)

Abstract: In this paper, we present our submission to the IWSLT 2023 Simultaneous Speech-to-Text Translation competition. Our participation involves three language directions: English-German, English-Chinese, and English-Japanese. Our proposed solution is a cascaded incremental decoding system that comprises an ASR model and an MT model. The ASR model is based on the U2++ architecture and can handle both streaming and offline speech scenarios with ease. Meanwhile, the MT model adopts the Deep-Transformer architecture. To improve performance, we explore methods to generate a confident partial target text output that guides the next MT incremental decoding process. In our experiments, we demonstrate that our simultaneous strategies achieve low latency while maintaining a loss of no more than 2 BLEU points when compared to offline systems.

## introduction this paper describes the hw-tsc's submission to the simultaneous speech-to-text translation (simuls2t) task at iwslt 2023 @xcite . from a systems architecture perspective, current research on simultaneous speech-to-text translation (simuls2t) can be categorized into two forms: cascade and end-to-end. cascade systems typically consist of a streaming automatic speech recognition (asr) module and a streaming text-to-text machine translation (mt) module, with the possibility of incorporating additional correction modules. while integrating these modules can be complex, training each module with sufficient data resources can prove to be worthwhile. alternatively, an endto-end approach is also an option for simuls2t, where translations can be directly generated from a unified model with speech inputs. however, it is important to note that bilingual speech transla-tion datasets, which are necessary for end-to-end models, are still scarce resources. the current efforts in simultaneous speech-totext translation (simuls2t) concentrate on developing dedicated models that are tailored to this specific task. however, this approach has certain drawbacks, such as the requirement of an additional model, which typically involves a more challenging training and inference process, as well as heightened computational demands and the possibility of decreased performance when utilized in an offline environment. our approach for this study involves utilizing a sturdy offline asr model and a robust offline mt model as the foundation for our system. by modifying the onlinization approach of @xcite and introducing an enhanced technique that can be seamlessly integrated into the cascade system, we are able to demonstrate that our simultaneous system can perform at the similar level as the offline models under strict latency restrictions without any adjustments to the original models. furthermore, our system even surpasses previous higher latency iwslt systems. our contribution is as follows: ## related work simultaneous speech-to-text translation can be achieved through either a cascaded system or an end-to-end model, both of which can be (hybrid) in nature. while cascaded systems currently offer the highest quality in offline speech translation, endto-end speech translation provides a better tradeoff between quality and latency @xcite . end-to-end speech translation systems incorporate various techniques to enable simultaneous translation. for example, @xcite implements a wait-k model and utilizes meta-learning to address data scarcity, while @xcite employs a wait-info model that incorporates information entropy from both the original text and the translation into the model. additionally, @xcite utilizes a unidirectional encoder with monotonic cross-attention to constrain dependence on future context. in addition, some research has focused on detecting stable hypotheses. for instance, @xcite proposed the hold-n strategy, which identifies the best hypothesis in the beam and removes the last n tokens from it. similarly, @xcite introduced the la-n strategy, which identifies the matching prefixes of two consecutive chunks. additionally, like the la-n strategy, @xcite developed the sp-n strategy, which identifies the longest common prefix among all items in the beam of a chunk. our work directly addresses this issue. ## results from ## conclusion in this paper, we report on our work in the iwslt 2023 simultaneous speech-to-text translation evaluation. we propose an onlinization strategy that can be applied to cascaded systems and demonstrate its effectiveness in three language directions. our approach is simple and efficient, with asr and mt modules that can be optimized independently. our cascade simultaneous system achieves results that are comparable to offline systems. in the future, we plan to further explore the direction of end-to-end systems.