NAIST Simultaneous Speech-to-Text Translation System for IWSLT 2022 (2022)

Abstract: This paper describes NAIST’s simultaneous speech translation systems developed for IWSLT 2022 Evaluation Campaign. We participated the speech-to-speech track for English-to-German and English-to-Japanese. Our primary submissions were end-to-end systems using adaptive segmentation policies based on Prefix Alignment.

## introduction this paper describes naist's submissions to iwslt 2022 @xcite simultaneous speech translation track. we participated the speech-to-speech track for english-to-german (en-de) and english-to-japanese (en-ja) using our end-to-end simultaneous machine translation (simulmt) systems. simulmt based on neural machine translation (nmt) has achieved a large success in recent years. there are two different simulmt approaches depending on the policy that determines read (waiting for speech input) and write (writing text output) actions: fixed and adaptive. fixed policies are usually implemented by simple rules @xcite @xcite . they are simple yet often effective, but they sometimes make inappropriate decisions due to large word order differences, pauses, and so on. in contrast, adaptive policies decide read or write actions flexibly taking current context into account @xcite @xcite . they can be more effective than fixed policies in end-to-end speech-to-speech simulmt because it is difficult to define fixed policies for speech input. in our systems, we use bilingual prefix alignment @xcite , which extracts alignment between bilingual prefix pairs in the training time, for prefix-to-prefix translation in simulmt. the bilingual prefix alignment is applied to extract prefix pairs of source language speech and target language translations. we also use the prefix pairs to train a boundary prediction model for an adaptive speech segmentation policy. our system showed some improvements against wait-k baselines on the development data, in all the latency regimes in both en-de and en-ja. ## simultaneous speech translation based on bilingual prefix alignment we developed simultaneous speech translation (simulst) based on offline speech translation (st). our simulst system translates an incrementallygrowing source language speech prefix into the target language. when the system detects a segment boundary in source language speech, the latest segment is translated taking its input and translation history into account. the st model is basically the same as an offline one, and we used it to translate an input prefix speech segment from the beginning. however, we constrained the translation prefix by the results in the previous time step. the constraint is implemented by a forced decoding with a given translation prefix. figure 1 shows an example of whole translation process, but we input the speech prefixes with fixed number of frames. please refer to @xcite for details of bilingual prefix alignment. for this system, we need an st model using an st corpus consisting of source language speech segments and corresponding translations in the target language. we then fine-tune the offline st model with prefix pairs of source language speech and target language translations obtained using bilingual prefix alignment. we also need a boundary predictor to segment source language speech adaptively as simulmt policies. in this section, we present how to extract prefix pairs (2.1) and build the boundary predictor (2.2). ## primary system we developed simulst systems for two language pairs: english-to-german (en-de) and englishto-japanese (en-ja). we implemented both our systems based on fairseq foot_0 @xcite . ## experiments we conducted comparative experiments with waitk @xcite . for baseline wait-k, we tried k ranging from 1 to 19 at two intervals for en-de and 5 to 31 at two intervals (excluding 29) for en-ja. ## conclusions in this paper, we described our simulst systems in english-to-german and english-to-japanese. the proposed method uses prefix alignment data to finetune the offline st model and train boundary predictor that judges when to read and write. our models achieved some improvements compared to the wait-k baselines in every latency regime in both english-to-german and english-to-japanese.