The HW - TSC ’s Speech to Speech Translation System for IWSLT 2022 Evaluation (2022)

Abstract: The paper presents the HW-TSC’s pipeline and results of Offline Speech to Speech Translation for IWSLT 2022. We design a cascade system consisted of an ASR model, machine translation model and TTS model to convert the speech from one language into another language(en-de). For the ASR part, we find that better performance can be obtained by ensembling multiple heterogeneous ASR models and performing reranking on beam candidates. And we find that the combination of context-aware reranking strategy and MT model fine-tuned on the in-domain dataset is helpful to improve the performance. Because it can mitigate the problem that the inconsistency in transcripts caused by the lack of context. Finally, we use VITS model provided officially to reproduce audio files from the translation hypothesis.

## introduction in this year, there is only one track in the speech to speech translation task which is the english to german translation (en-de) @xcite . the audio files in english are given in the dataset, and we are required to produce the audio files in german. in recent research of speech to speech task, there are basically two paradigms with respect to the system architecture, which are cascade and end-to-end. and the cascade pipeline composed by an asr model, a mt model and a tts model is commonly used, because this system is more mature than end-to-end one. the advantage of this pipeline is that each module of the system can be a state-of-the-art one trained on sufficient independent corpora. it also allows us to perform experiments with different combinations of asr models, mt models and tts models. but compared to end-to-end system, this cascade system may not capture all information like accent of speakers, emotion, etc. end-to-end system like s2ut is introduced in @xcite , which can be directly trained on speech to speech dataset with the help of text generation as the auxiliary task. however, we didn't adopt this approach due to the insufficiency of available corpora. for the asr model, we tried conformer @xcite , @xcite and u2 @xcite , and obtained three types of asr results. in translation, inconsistency of translation of same words in the context is a common difficulty. this is caused by the flaw of conventional translation that treats each sentence independently in a documents, ignoring surrounding contexts. for example, a family name in english can be translated in different ways in chinese. because chinese transcripts comes from transliteration, and there are lots of words share same pronunciation but different spelling. this may cause the ambiguity in transcripts, which is hard for readers to understand. to solve the problem, we propose the context-aware reranking strategy in translation, essentially an approach to adapt sentence-level mt models into document-level translation scenarios. it aims to generate the best candidate by taking previous contexts into account and reranking with scores estimated by all models. ## conclusion in the paper, we elaborate the cascade system for this speech to speech task. there are several strategies we applied to improve the system, including domain-tag prefix and the context-aware reranking strategy. we did some experiments to verify the reliability of those strategies for a cascade system, and we also made some analysis from the theoretical level. in the future, we are going to explore the feasibility of the end-to-end system, since it might reduce the negative impact of information loss on system performance.