NAIST E nglish-to- J apanese Simultaneous Translation System for IWSLT 2021 Simultaneous Text-to-text Task (2021)

Abstract: This paper describes NAIST’s system for the English-to-Japanese Simultaneous Text-to-text Translation Task in IWSLT 2021 Evaluation Campaign. Our primary submission is based on wait-k neural machine translation with sequence-level knowledge distillation to encourage literal translation.

## introduction automatic simultaneous translation is an attractive research field that aims to translate an input before observing its end for real-time translation similar to human simultaneous interpretation. starting from early attempts using rule-based machine translation @xcite and statistical methods using statistical machine translation @xcite , recent studies successfully applied neural machine translation (nmt) into this task @xcite @xcite . the simultaneous translation shared task in the iwslt evaluation campaign started on 2020 with english-to-german @xcite speechto-text and text-to-text tasks, and a new language pair of english-to-japanese has been included on 2021 only in text-to-text task. english-to-japanese is much more challenging than english-to-german due to the large language difference in addition to data scarsity. we developed an automatic text-to-text simultaneous translation system for this shared task. we applied some extensions to a standard wait-k nmt in the training time: sequence-level knowledge distillation and target-side chunk shuffling. however, these techniques showed mixed results in different latency regimes on the iwslt21 development set, so we configured the system differently for each latency regime. this paper describes the details of the system and the results on the development sets. we also describe our another attempt to include incremental constitutent label prediction that was not included in the primary system. ## simultaneous neural machine translation with wait-k let @xmath0, x 2 , . . . , x |x| be an input sequence in a source language and @xmath1, y 2 , . . . , y |y | be an output sequence in a target language. here, the input can be speech or text, but we assume the input is text because this paper discusses the text-to-text task. the task of simultaneous translation is to translate x to y incrementally. in other words, each output prediction of y is made upon partial input observations of x. suppose an output prefix subsequence y j 1 = y 1 , y 2 , ..., y j has already been predicted from prefix observations of the input x i 1 = x 1 , x 2 , ..., x i . when we predict the next output subsequence y j j+1 = y j+1 , ..., y j after further partial observations x i i+1 = x i+1 , ..., x i , the prediction is made based on the following formula: wait-k delays the decoding process in k input tokens @xcite . the wait-k model translates a token sequence of the source language x into that of the target language y as follows. the decoder has to predict an output token based on the attention over an observed portion of the input tokens. k is a hyperparameter for the fixed delay in this model; setting k larger causes longer delays, while smaller k would result in worse output predictions due to the poor context information. ## knowledge distillation knowledge distillation (kd) @xcite is a method that uses the distilled knowledge learned by a stronger teacher model in the learning of a weaker student model. when teacher distribution is q(y|x; θ t ), we minimize the cross-entropy with the teacher's probability distribution instead of reference data, as follows: we use skd for reduction of colloquial expressions in the spoken language corpus. such colloquial expressions are highly dependent on languages and difficult to translate by nmt, which usually generates literal translations. here, we firstly train a teacher, transformer-based offline nmt model using the training corpus and use it to obtain pseudo-reference translations in the target language. then, we train a student, transformerbased simultaneous nmt model using the pseudoparallel corpus with the original source language sentences and the corresponding translation results by the teacher model. the pseudo-references should consist of more literal and nmt-friendly translations, therefore the training of the student model becomes easier than that using the original parallel corpus. since we have to train simultaneous translation using less context information than an offline translation model, the skd would be helpful. this is motivated by the recent success of non-autoregressive nmt using skd @xcite . ## target-side chunk shuffling chunk shuffling is a kind of data augmentation that reorders japanese chunks (called bunsetsu). our motivation for this one is to encourage monotonic imt utilizing a characteristic of japanese as an agglutinative language, in which the order of bunsetsu chunks is not so strict. when we have a target language sequence @xmath2, . . . , t |t | in the training set, we apply greedy left-to-right chunking to it; t is divided as a chunk sequence @xmath3, . . . , c q , in which each chunk consists of k (i.e., delay hyperparameter in wait-k) tokens c @xmath4, . . . , t q k . note that the last chunk c q may be shorter than k according to the length of t . then, we choose to shuffle or keep the chunks in t with a probability p r , defined as a hyperparameter. we tried only the random shuffling with the fixed chunk size of k in this time; more linguistically-motivated chunk reordering would be worth trying as future work. ## another attempt: incremental next constituent label prediction we tried another technique described below in the shared task, but it was not included in our primary submission because it did not outperform the baseline. here, we also describe this for further investigation in future. for simultaneous machine translation, deciding how long to wait for input before translation is important. predicting what kind of phrase comes next is a part of useful information in determining the timing. in this study, we tried incremental next constituent label prediction (nclp). in smt-based simultaneous translation, @xcite proposed a method to predict unseen syntactic constituents to determine when to start train dev eval 2,762,408 27,903 21,941 we used huggingface transformers @xcite for our implementation of nclp with bert-base-uncased. we used penn treebank 3 @xcite for the nclp training and development sets, and naist-ntt ted talk treebank @xcite for the nclp evaluation set. however, the results by wait-k using nclp results as its input did not outperform the baseline wait-k, as shown in figure 2 . we can observe nclp-based wait-k gave smaller als with the same latency hyperparameter k. one possible problem of current nclp-based wait-k is that the length of an input length is doubled by the additional constitutent labels. since we ran wait-k-based simultaneous nmt for such an augmented input sequence, the decoder using nclp-augmented input has roughly half of the information compared to the decoder using original input if we use the same k. this forces the decoder to perform very aggressive anticipation with limited information from an input prefix. ## conclusion in this paper, we described our english-to-japanese text-to-text simultaneous translation system. we extended the baseline wait-k with the knowledge distillation to encourage literal translation and targetside chunk shuffling to relax the output order in japanese. they achieved some improvements on iwslt 2021 development set in certain latency regimes.