Tagged End-to-End Simultaneous Speech Translation Training Using Simultaneous Interpretation Data (2023)

Abstract: Simultaneous speech translation (SimulST) translates partial speech inputs incrementally. Although the monotonic correspondence between input and output is preferable for smaller latency, it is not the case for distant language pairs such as English and Japanese. A prospective approach to this problem is to mimic simultaneous interpretation (SI) using SI data to train a SimulST model. However, the size of such SI data is limited, so the SI data should be used together with ordinary bilingual data whose translations are given in offline. In this paper, we propose an effective way to train a SimulST model using mixed data of SI and offline. The proposed method trains a single model using the mixed data with style tags that tell the model to generate SI- or offline-style outputs. Experiment results show improvements of BLEURT in different latency ranges, and our analyses revealed the proposed model generates SI-style outputs more than the baseline.

## introduction simultaneous speech translation (simulst) is a technique to translate speech incrementally without waiting for the end of a sentence. since simulst should work in small latency against the input speech, monotonic translation following the word order of the source language is preferable. however, making translation monotonic is not trivial especially for distant language pairs with different word orders, such as english and japanese. most recent simulst studies still use parallel corpora only with offline translations and potentially have the limitation to work in a monotonic way. a prospective approach to this problem is to use si data to train a simulst model for mimicking simultaneous interpretation (si). there are several si data resources developed so far for english-japanese @xcite @xcite . despite these efforts, si data are still very small compared to bilingual data based on offline translations. using such scarce si data to fine-tune an offline translation model causes overfitting on the small si data. training a model using mixed data of offline and si data is another option to mitigate the problem of data scarcity, but the simple data mixture causes confusion between the output styles of offline translation and si. in this paper, we propose a method to train a simulst model using mixed data of si and offline translation with style tags to tell the model to generate si-or offline-style output selectively. it has the advantage of sharing two different styles in a single model and generating si-style outputs by putting the si-style tag in the decoding, which are leveraged by offline translation data. experiment results using must-c and small si data showed improvements of bleurt by the proposed method over the baselines in different latency ranges. further analyses revealed that the proposed model generates more appropriate si-style outputs than baselines. ## related work there have been many studies on simultaneous translation for text and speech in decades @xcite @xcite . most recent approaches are based on deep neural networks and have evolved with the technologies of neural machine translation (nmt) @xcite and neural speech recognition (asr) @xcite . an important advantage of the neural simulst methods @xcite is their end-to-end modeling of the whole process, which improves the efficiency compared to a cascade approach. such an end-to-end simulst model is trained using speech translation corpora such as @xcite , but these corpora are usually based on offline translation due to the lack of large-scale si data. for the english-japanese language pair, there 6 しかしこの経済(6)危機や私の( 8 )国での( 7 )出来事について(1)私は( 4 )男性に( 5 )非があると(3)言うつもりは(2)ありません (4)男性の、(5)せいだけでは(2)ありません、私どもの(8)国の、金融(6)崩壊の、(5)責任は、 and (1)i'm (2)not here to (3)say that (4)men are to (5)blame for the (6)crisis and what (7)happened in my (8)country. si target source offline target have been some attempts for the development of si corpora @xcite @xcite . however, the amount of such si corpora is still very limited compared to offline translations. we tackle this problem by using a larger-scale offline translation corpus. this condition can be seen as domain adaptation from resource-rich offline translation to resource-poor simultaneous translation. in a typical domain adaptation scenario, an out-of-domain model is finetuned using in-domain data @xcite , but it tends to overfit to the small in-domain data @xcite . as another adaptation approach, tag-based nmt works to control the politeness of translations @xcite and to enable zero-shot multilingual nmt @xcite . this tagbased approach has been extended to multi-domain fine-tuning @xcite and mixed finetuning @xcite . these studies fine-tune nmt models using mixed data of in-domain and out-of-domain corpora. tagged back-translation @xcite is an application of the tagbased approach to well-known back-translationbased data augmentation. it distinguishes source language sentences from parallel corpora and those obtained from back-translation to handle possible back-translation noise in the training of an nmt model. our work is motivated by these tag-based methods and tackles the scarcity of si data. ## differences between offline translation and simultaneous interpretation there is a large style difference between si and offline translation. figure 1 shows an example of offline translation and si transcript in japanese for a given english source sentence. the solid lines in the figure represent word correspondences. in this figure, we can find: these points suggest important differences between offline translation and si; si focuses on the simultaneity of the interpretation to deliver the contents as early as possible and to maintain the interpreter's working memory. the word order difference between english and japanese poses a serious difficulty in si, as mentioned in the literature @xcite . thus, it is important to use si data to train a simulst model to improve its simultaneity. ## proposed method although training a simulst model using si data is necessary, we suffer from data scarcity in practice. we propose a method to use a relatively large offline translation corpus to mitigate for the si data scarcity for training a simulmt model. following the tag-based nmt studies, we put a style tag at the beginning of the target string in training and predict a specified tag forcibly at the first step in inference. in this work, we use two tags: <@xmath0. suppose we have an si transcript: 私は、買っ た。 ペンを、 for an english input: i bought a pen. as a training example. we put the si-style tag at the beginning of the si transcript as follows: this string is tokenized into subwords 1 : here, we assume we have a pre-trained sequenceto-sequence model such as mbart @xcite as a basis of the simulst model, as described later in the next section. the aforementioned style tags may not be included in the subword vocabulary of the pre-trained model and are tokenized further like "_< @xmath1, but it works in practice. ## discussions the results shown in figures 2 , 3 demonstrated the advantage of the proposed method in bleurt, but not in bleu. in this section, we discuss the results in detail to reveal which model works the best from the viewpoint of simulst. ## conclusion in this paper, we proposed an effective method to train a simulst model using mixed data of siand offline-style translations with style tags to tell the model to generate outputs in either style, motivated by the tag-based approach to domain adaptation. experiment results on english-to-japanese simulst demonstrated the advantage of the proposed method in bleurt and bertscore recall despite the inferior performance in bleu and bertscore precision due to over-translations and repetitions. future work includes an extension to other language pairs and further verification via human evaluation.