The U niversity of E dinburgh’s Submission to the IWSLT 21 Simultaneous Translation Task (2021)

Abstract: We describe our submission to the IWSLT 2021 shared task on simultaneous text-to-text English-German translation. Our system is based on the re-translation approach where the agent re-translates the whole source prefix each time it receives a new source token. This approach has the advantage of being able to use a standard neural machine translation (NMT) inference engine with beam search, however, there is a risk that incompatibility between successive re-translations will degrade the output. To improve the quality of the translations, we experiment with various approaches: we use a fixed size wait at the beginning of the sentence, we use a language model score to detect translatable units, and we apply dynamic masking to determine when the translation is unstable. We find that a combination of dynamic masking and language model score obtains the best latency-quality trade-off.

## introduction in spoken language translation (slt), there is often a need to produce translations simultaneously, without waiting for the speaker to finish. for example, we may be targeting live events such as conferences or meetings where excessive latency will disrupt the user experience. in order to achieve low latency slt, however, translation systems must be able to cope well with incomplete utterances, and we find that we need to trade off latency for translation quality. in research on simultaneous slt, we would like to understand how to produce the best possible trade-off between these two measures. in the iwslt 2021 shared task on simultaneous translation, the aim was to build and evaluate simultaneous slt systems at three different latency regimes (low, medium and high), as measured using the average lagging (al; @xcite ). ## overview of our submission we participated in the @xmath0. since we re-translate the incomplete input (know as a prefix) each time it is updated, our system will try to modify the translations produced from earlier prefixes. but as the task is evaluated using simuleval @xcite which does not permit the modification of committed output (also known as flickering), we use a simple approach to generate incremental output at each re-translation step. concretely, we apply a method inspired by the wait-k streaming approach @xcite in our re-translation system in the following manner. in the task, a simultaneous slt system is implemented as an agent which must choose between read (read more input) and write (append to the current translation hypothesis) operations. our overall approach is shown in algorithm 1. the agent first performs k consecutive read operations and then alternatively reads and writes until the full input sentence is read. once the input is consumed, the agent keeps performing write operations until it reaches the end of the translated sentence. the write operation involves re-translating the prefix s and finding the next output word w from output prefix t . if the output prefix t has a length longer than the committed hypothesis h, it picks the (i + 1)th word of t , else sends read signal to the agent, i being the length of the current hypothesis. algorithm 1 our re-translation approach require: nmt system φ, k 1: initialize: s ← {}, h ← {}, w ← ε 2: while w is not 〈eos〉 do our first method of detecting mus relies on the language model (lm) score. the agent keeps track of the language model (lm) score of the previous token and compares it with the score of the current token. if the lm score is higher than the previous token, it keeps reading more tokens and does a re-translation only when this condition is not met. here the lm score is the log probability of the current token given the context. though lm score doesn't guarantee to find meaningful unit every time but this simple approach shows it is better than the baseline approach in terms of bleu score. our second method of stabilising the retranslation approach is based on the idea of dynamic masking @xcite . the dynamic mask approach finds the stable part of the target prefix by comparing the translation of the current prefix, with the translation of an extension of the current prefix. the longest common prefix (lcp) of the two translations is taken as the stable part. figure 1 shows how dynamic masking works in general. @xcite showed that using dynamic mask could give a better flickerlatency trade-off than using a fixed mask, without affecting the translation quality of full sentences. for our iwslt submission, we generate the extended prefixes for dynamic mask simply by appending unk (i.e the unknown word symbol) to the prefix. in figure 2 , we show an example of how dynamic mask stabilises the translation, by masking the least stable part of the mt output. this translation-with-dynamic-mask provides a drop-in replacement for the mt system φ() in line 7 of algorithm 1, except when the agent has read the full input sentence, when we do not need to apply any mask. ## experimental details we use only the officially allowed iwslt 2021 data sets. the training data include high quality english-german parallel data from wmt 2020 @xcite , english-german data from @xcite , the ted corpus @xcite and opensubtitle @xcite . for development, we use the concatenation of iwslt test sets from 2014 and 2015. we test on iwslt 2018 test set and tstsource translation mt output prefix back in new york, zurück in new york, extension back in new york, unk damals in new york, in prefix back in new york, i damals in new york have ich extension back in new york, i unk damals in new york war i damals in new york figure 2: an example of dynamic mask applied during translation. for the first prefix, the translation of the prefix and its extension disagree, so no output is produced (i.e. all output is masked). for the second prefix, the translation is more stable. common from must-c.v2. as the there is a significant overlap between must-c.v2 and tst-20{14,15,18}, we remove the overlaps from the must-c.v2 training data before training. for preprocessing we rely only on sentence-piece tokenization @xcite ; no other preprocessing tools are applied. we use a shared vocabulary size of 32k. standard nmt models perform well when translation is done on a full sentence but as our approach is based on retranslation, we use training data that is a 1:1 mix of full sentences and prefix pairs @xcite . this ensures that our model can translate both full sentences and prefixes. to create prefix pairs, we first randomly choose a position in the source sentence and then take the proportionate length of the target sentence. along with that we also add modified prefix pairs in which the source side has a shorter target prefix appended with the source prefix. the purpose of these modified prefix pairs was to investigate an alternative type of stabilisation, where the previous target prefix is fed into the translation of the current source prefix, but in early testing this method did not work well, so we did not pursue it further. the validation data is also pre-processed similarly to the training set. note that this preprocessed validation set is used at training for early stopping and not for reporting the validation scores in the for training, we use the marian toolkit @xcite with the 'base' transformer architecture @xcite . first, we train a model using the aforementioned pre-processed training data and then fine-tune the model using must-c.v2 training data which is more of a domain specific data for simultaneous translation task. to train the language model for stabilisation, we use kenlm @xcite to train a 6-gram language model on the source-side training data. we have shown the number of sentences in each corpus in ## conclusion in this paper, we describe our submission to the iwslt 2021 shared task on simultaneous text-totext german-english translation. we work with a re-translation approach, enabling use to use an unmodified mt inference engine, together with an adaptation of wait k to trade off quality and latency. additionally we proposed two techniques (dynamic masking and lm score) to improve translation quality by reducing the potential for flicker. we find that the combination of the proposed approaches achieves the best al-bleu trade-off.