Robustness of Multi-Source MT to Transcription Errors (2023)

Abstract: Automatic speech translation is sensitive to speech recognition errors, but in a multilingual scenario, the same content may be available in various languages via simultaneous interpreting, dubbing or subtitling. In this paper, we hypothesize that leveraging multiple sources will improve translation quality if the sources complement one another in terms of correct information they contain. To this end, we first show that on a 10-hour ESIC corpus, the ASR errors in the original English speech and its simultaneous interpreting into German and Czech are mutually independent. We then use two sources, English and German, in a multi-source setting for translation into Czech to establish its robustness to ASR errors. Furthermore, we observe this robustness when translating both noisy sources together in a simultaneous translation setting. Our results show that multi-source neural machine translation has the potential to be useful in a real-time simultaneous translation setting, thereby motivating further investigation in this area.

## introduction speech translation (st) suffers from automatic speech recognition (asr) errors, especially in challenging conditions such as non-native language speakers, background noise, named entities and specialized vocabulary usage @xcite @xcite . asr errors negatively impact translation quality, via the compounding of speech recognition and translation errors @xcite , thereby limiting the application of automatic speech translation in realistic settings. fortunately, there are multilingual settings where a source is simultaneously or consecutively interpreted into multiple languages. many documents are also dubbed or subtitled in offline mode. this simultaneous interpretation takes place either via human interpreters, in the form of dubbing or subtitling. in a situation where the same sentence is available in multiple languages, multi-source machine translation (mt) significantly improves translation quality, especially when the two sources used separately do not yield high quality translations @xcite @xcite . although not yet clearly verified, multi-source mt could be useful in settings where the sources complement each other. in other words, challenges in translation posed by using each source should be independent of one another. given that asr is noisy and that multiple sources can help overcome limitations of individual sources, this paper asks the following question: "can multi-source mt be leveraged for speech translation in a multilingual setting where an original source transcription and its simultaneously interpreted transcription are available?" we decompose this question into three parts where we hypothesize that (a) asr errors in the original and interpreted transcripts are independent, which makes them complementary, (b) multi-source mt is robust to transcription errors present in individual sources, and (c) the robustness of multi-source mt continues to hold in a simultaneous translation setting. we address each question in sections 3, 4, and 5, respectively. to prove our hypotheses, firstly, we verify on the europarl simultaneous interpreting corpus (esic, @xcite that original speech asr and interpreted speech asr are indeed complementary in terms of errors. secondly, we simulate transcription errors in a full sentence multi-source mt setting for english and german to czech translation. we clearly show that when both sources are noisy, using them together leads to significant improvements when compared to using them individually, in contrast to drops in quality when at least one of the sources is clean. for example, on esic test set with 15% wer noise in english source and 10% wer noise in german source, multi-sourcing performs 0.9 bleu score higher than english source. finally, we use both sources in a simultaneous translation setting and show that multi-source mt continues to be robust to transcription errors. our findings show that multi-source mt has strong potential in a simultaneous translation setting where multiple sources are available via asr or interpreted asr. we note that our current analysis is limited to the case where the multiple sources are aligned and hence available at the same time. this is a setting of, e.g., dubbed and subtitled videos where we would want to consider additional target languages. in simultaneous settings where one source is available with a delay, the synchronization of the sources would be a considerable problem, which we leave for future research. ## related work this paper mainly focuses on asr errors, multilingual multi-source translation and simultaneous translation. asr errors often propagate to mt in cascaded st systems and in a real time translation setting where asr systems are used. it is a major issue that affects translation quality. @xcite propose a method to tune the mt on the training data with artificial noise that mimics asr errors, via a unigram "lexical noise model" learned on automatic-gold transcript pairs. other authors propose similar methods for training @xcite @xcite . however, in this work, rather than mimicking asr noise during training, we complement a noisy source with another whose asr errors are provably independent of the first. specifically, we use the lexical noising of @xcite to simulate noise in multiple source languages and show the robustness of mt models, especially in a multi-source setting. multilingualism @xcite has been shown to improve translation quality in a variety of situations. in particular, multi-source machine translation has high potential for improving translation quality, but has been relatively underexplored. in the context of multi-source text-to-text translation, @xcite and @xcite showed that leveraging the same sentence in different languages improves translation quality as the two sources are expected to com-plement hard to translate phenomena in the other source. although this approach requires multiparallel sentence-aligned data, the missing sources can be obtained by mt @xcite or via simultaneous interpretation. rather than training a multi-source model, @xcite propose the "late averaging" which needs multilingual models trained on pairwise bilingual data, which we also focus on when evaluating multi-source models. late averaging is akin to ensembling via logits averaging, but with source sentences in different languages. these works do not consider transcription errors which are ubiquitous in speech translation, an aspect this paper focuses on. multi-sourcing in simultaneous translation settings has not been extensively explored. @xcite have explored simultaneous multipivot translation where a source is translated into a target language via multiple pivot languages, where the pivot languages are translated using multi-source translation. unlike them, we consider only one pivot language which is interpreted from the source and then use it together with the source to show that the translation quality into the target language improves. additionally, they do not consider the effect of transcription noise on translation, which we do. simultaneous translation approaches such as wait-k @xcite and local agreement (la-n, @xcite are commonly used, and we use the latter for our experiment. an alternative multi-sourcing approach is to select and use only one source. @xcite provide an analysis of using either the original, or its simultaneously interpreted equivalent as a source for simultaneous st. interpreting is delayed, but shorter and simpler than translationese. interpreters also segment their speech to sentences differently than the original speakers, so it is not easy to align segments. in any case, selecting sources will involve additional effort and thus we consider using multiple sources together to be a more effective approach. in this regard, multiple language sources, both as text and speech streams, could be used in asr @xcite as well as in pre-neural @xcite @xcite . @xcite use them for punctuation restoration. @xcite provide a broad analysis of benefits of multilingual mt. ## parallel source-interpreted asrs are independent we assume a multi-source setting with the original speech and its simultaneously interpreted equivalent as the two sources will improve robustness to asr errors if the errors in the two source streams complement each other. this is not obvious because, on the one hand, the asrs work independently where they are deployed for different languages, trained on different data, and the processing is fully independent. on the other hand, the content of the speeches is identical. interpreters' speech pacing also depends on the original speaker, and it may influence the quality of both asrs the same way. therefore, in this section, we analyze the dependency of asr errors in the source and interpreter, on 10-hour esic corpus @xcite to prove that the asr errors are indeed independent. methodology first, we processed asr for english original speakers and interpreters into czech and german. for english, we used the low-latency neural asr by @xcite . for german, we used an older hybrid hmm-dnn model trained using the janus recognition toolkit, which features a single-pass decoder @xcite . for czech, we used kaldi @xcite hmm-dnn model trained on czech parliament data @xcite . we then re-used the word alignments of gold transcripts between the original and interpretation as described in @xcite . 38% of tokens were aligned between english and czech interpretations, and 40% between english and german, see finally, we aligned gold and automatic transcripts using levenshtein edit distance. 1 we classified each token in the asr transcript as transcribed correctly or not, both for source and interpretations. ## multi-source speech translation having established that asr errors are independent, we now analyze whether multi-source neural machine translation (nmt) is robust to noisy sources. we focus on nmt for individual sentences, with gold sentence alignment of the sources and reference. it is a less realistic use-case than translating long speech documents without any sentence segmentation and alignment of the sources, but proving the robustness of multi-sourcing in this data for training, we use data from opus @xcite , aiming at a multiway model with english and german on the source side and czech as a target. we download all the data from opus, remove all sentences from iwslt, wmt, esic and other test sets, filter them by language identification, and then process with dual cross-entropy scoring (junczys-dowmunt, 2018) using the bilingual nmt models from @xcite . we select the top 30 million sentences for each language pair as training data, to prevent overfitting for either. it is also near the threshold that @xcite showed as optimal. for nmt validation and evaluation, we use the "revised transcript and translations" from esic @xcite . these are the texts that were originally uttered in the european parliament, transcribed, revised and normalized for reading and publication on the website, and then translated. they are analogous, but not identical, to the gold transcripts of the original and interpretations that we used in section 3. in addition to the version published by @xcite , we properly align the sentences in all the three languages. two documents were removed because they missed german translation. the corpus is of comparable size to a usual mt test set. see size statistics in for a contrastive evaluation, we use newstest11 @xcite . it contains 3003 sentences in 5 languages: english, german, czech, french and spanish, the same amount in each. newstest11 has references that were translated directly, not through an intermediate language. we also use three additional czech references of new-stest11 that were translated from german @xcite . multi-sourcing we convert marian models to pytorch to be used with the hugging face transformers @xcite library, in which we implement late and early averaging. for both single-and multi-sourcing, we use greedy decoding because beam search support is not implemented with multi-source. training details we train a multi-way nmt model using marian @xcite with english and german as sources, with language identification tokens, and czech as the target. we use two separate sentencepiece @xcite vocabularies, both sizes of 16 000. the source vocabulary is joint for german and english, and the target is only for czech. the model is a transformer base (6 layers, 512 embedding size, 8 self-attention heads, 2048 filter size) trained on 8 quadro p5000 gpus with 16 gb memory for 17 days, until convergence. ## simultaneous multi-source in the previous section, we experimented with offline translation with artificial asr noise and showed that multi-source models are indeed robust to noise. however, one important use case of speech translation is in a real time setting where simultaneous mt is used. we therefore evaluate the robustness of multi-source models in a simultaneous setting. ## conclusion we have investigated the robustness of multi-source nmt to transcription errors in order to motivate its use in settings where asrs for the original speech and its simultaneously interpreted equivalent are available. to this end, we first analyzed the 10-hour esic corpus and documented that the asr errors in the two sources are indeed independent, indicating their complementary nature. we then simulated transcription noise for english and german when translating into czech in single and multi-source nmt settings and observed that using multiple noisy sources is significantly better than individual noisy sources. we then repeated experiments in a simultaneous translation setting and showed that multi-source translation continues to be robust to noise. this robustness of multi-source nmt to noise motivates future research into simultaneous multi-source speech translation, where one source is available with a delay. we will also consider training models with simulated asr errors to further increase their robustness, especially in multisource settings.