It Is Not As Good As You Think! Evaluating Simultaneous Machine Translation on Interpretation Data (2021)

Abstract: Most existing simultaneous machine translation (SiMT) systems are trained and evaluated on offline translation corpora. We argue that SiMT systems should be trained and tested on real interpretation data. To illustrate this argument, we propose an interpretation test set and conduct a realistic evaluation of SiMT trained on offline translations. Our results, on our test set along with 3 existing smaller scale language pairs, highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data. In the absence of interpretation training data, we propose a translation-to-interpretation (T2I) style transfer method which allows converting existing offline translations into interpretation-style data, leading to up-to 2.8 BLEU improvement. However, the evaluation gap remains notable, calling for constructing large-scale interpretation corpora better suited for evaluating and developing SiMT systems.

## introduction simultaneous interpretation (si) is a task of translating natural language in real time. simt systems are expected to generate interpreted text as if the text was produced by human interpreters while maintaining acceptable delay @xcite . however, most current simt systems are trained and evaluated on offline translations differing from real-life si scenarios where translations are flexibly paraphrased, without compromising the source message @xcite . for instance, in prior work attempted to build interpretation corpora in a small scale @xcite @xcite , or constructed speech interpretation training corpora for mt tasks @xcite . but, very little attempt has been made on empirically quantifying the evaluation gap. an exception is @xcite which incorporated interpretation data in the training stage of a statistical mt system, but the lack of training data and the scale of evaluation set resulted in a marginal bleu score difference. 2 we compile a genuine interpretation test set of 1k utterances from the european parliament (ep) plenary focusing on @xmath0. we examine the real performance gap of wait-k @xcite , a state-of-the-art simt system, on our test set along with 3 smaller scale @xcite translation and interpretation language-pairs and observe a drop of up-to 13.83 bleu score. in the absence of interpretation-style training data, we propose a simple and effective translation-tointerpretation (t2i) style transfer method to produce pseudo-interpretations from abundant offline translations. training on our t2i transferred data, we observe an improvement of ∼2.8 bleu score. our findings necessitate further developments towards constructing large-scale interpretation corpora, designing domain adaptive techniques and models more reflective of real-life interpretations. ## @xmath1 we provide an overview of our data construction and move full details in appendix a.1. collection. we crawled data from the ep plenary foot_1 between 2008 and 2012 foot_2 and downloaded 238 debates consisting of speech transcriptions, offline translations and interpretation videos. we used google speech api to transcribe the interpretation videos and normalize automatic speech recognition (asr) outputs, yielding 323-hour of transcriptions. cleaning, alignment, and segmentation. we removed duplicates and the dialogues with non-german source sentences, while using available offline translations to retrieve named entities; this resulted in 5,239 dialogues. we filtered out dialogues with interpretations less than 4 words, and call the resulting interpretations raw hereafter. we further removed cases whose sources contained either (1) less than 20 tokens, (2) less than 150 words and included pre-defined signals, or (3) a different number of sentences from the corresponding offline translations. and whose sources and offline translations had a different number of sentences. next a manual process was applied, including removals of dialogues with non-essential contents and truncation of interpretations whose first and last sentences did not match the corresponding offline translations (mostly due to imperfect audio segmentation). 987 dialogues foot_3 were thus retained, each of which having 14.5 sentences on average. we aligned translations with transcriptions (interpretations). for each dialogue, as the transcriptions may not be well segmented in the asr process, we identified sentences in the transcriptions with stanza @xcite , before segmenting them using dynamic programming. manual inspection revealed that there were a portion of mismatched pairs, which was due to occasional interpreting failure resulting from interpreters' accumulated cognitive load @xcite . we further removed pairs the lengths of whose source and target were far off, and call it clean, containing @xmath2, translation, interpretation>. ## t2i style transfer offline translated texts and online interpreted texts differ in various aspects, including lengths, sentence structure and lexicon; this is fundamentally contributed by the fact that interpreters use tactics to minimize delay and reduce the load of retention @xcite . for example, interpreters tend to break a source sentence into several smaller chunks (see @xcite for more tactics). yet, while exhibiting stylistic differences, both preserve the key source message. as will be seen ( §4.3) these differences amount to a significant evaluation gap. while the ideal solution is using human annotators to create interpretation training corpora, in the absence of resources, we propose a simple technique, t2i style transfer, to convert existing translation data into interpretation-style data with a style transfer model. training such a model would require paired translations and interpretations, which are not available in large quantities. rather, our proposed approach allows fulfilling the goal of style transferring abundant translation data to interpretation-like data both in supervised settings, where clean is leveraged, and unsupervised settings, where only raw is used. supervised training given that our clean set consists of roughly 4.2k triples, we opt for statistical mt systems which inherently require far less data for sequence-to-sequence mapping tasks compared to their neural counterparts. furthermore, conducting style transfer in the same language involves word replacement and ordering, which conforms with the behaviors of smt systems that chunk an input sequence into segments, translate, and reorder the translated chunks @xcite . more specifically, we employ two classic statistical mt methods: phrase-based smt (pbmt) @xcite and hierarchical phrase-based mt (hpbmt) @xcite . 6 a similar framework was tried by @xcite for text simplification. we will describe the t2i pipeline process for unsupervised settings, as both settings have a similar process with different data configurations. the main difference is that we use clean instead of raw, which will be detailed in §4.1. unsupervised training figure 1 shows the three stages of our t2i approach in unsupervised settings: the first stage is to convert interpretations in raw to translation-style data by applying roundtrip translation on interpretations, with pretrained nmt models @xcite . it is expected that the outputs after this round-tripping, denoted as translation-fb, sit close to the translation domain, thus achieving the effects of interpretationto-translation. the second stage is to train a style transfer model to learn the mapping between the data points in translation-fb and their corresponding interpretations in raw. lastly, we apply the trained style transfer model on offline europarl translations and produce interpretation-like sequences which we call pseudo-i. foot_5 4 experiments in this section, we present datasets details ( §4.1) followed by the descriptions of our baselines and style transfer models ( §4.2). we report results by underlining the performance gap between evaluation on translated and interpreted texts ( §4.3), and showing the effectiveness of our t2i style transfer both quantitatively and qualitatively ( §4.4). we followed the instructions in @xcite to preprocess data, and their hyperparameters for training all wait-k models. for style transfer models, we used the standard setup for both pbmt and hpbmt. foot_6 ## conclusion we investigated the simt evaluation gap when simt models were tested on interpretation vs translation, across four language pairs. to the best of our knowledge, this is the first work quantifying this gap empirically. to bridge the gap, we proposed a data augmentation style transfer technique to create parallel pseudo-interpretations from abundant offline translation data. our results show an improvement of 2.8 bleu score. we hope our work and the highlighted evaluation discrepancy can encourage further developments of datasets and models more reflective of real-world si scenarios.