Inverted Projection for Robust Speech Translation (2021)

Abstract: Traditional translation systems trained on written documents perform well for text-based translation but not as well for speech-based applications. We aim to adapt translation models to speech by introducing actual lexical errors from ASR and segmentation errors from automatic punctuation into our translation training data. We introduce an inverted projection approach that projects automatically detected system segments onto human transcripts and then re-segments the gold translations to align with the projected human transcripts. We demonstrate that this overcomes the train-test mismatch present in other training approaches. The new projection approach achieves gains of over 1 BLEU point over a baseline that is exposed to the human transcripts and segmentations, and these gains hold for both IWSLT data and YouTube data.

## introduction speech translation is an important field that becomes more relevant with every improvement to its component technologies of automatic speech recognition (asr) and machine translation (mt). it enables exciting applications like live machine interpretation @xcite and automatic foreign-language subtitling for video content @xcite . however, translation of speech presents unique challenges compared to text translation. traditional text translation systems are often trained with clean, well-structured text consisting of (source language, target language) sentence pairs gathered from text documents. this works well for translating written text, but for cascaded systems composed of @xmath0, errors from asr and automatic punctuation are amplified as they pass through the translation * equal contribution system. such systems suffer from three issues: 1) spoken language structure is different from written language structure and can include aspects like disfluencies and partial sentences, 2) asr systems are not perfect and introduce errors in the stage from speech to source transcript, and 3) mistakes from automatic punctuation systems can lead to unnatural sentence segments and boundaries @xcite @xcite . these problems can lead to poor translations and pose unique challenges for mt that are not readily addressed by current methods. in this work, we set out to make mt robust to the second and third issues in particular. we have developed an approach to train translation models that are robust to transcription errors and punctuation errors, by introducing errors from actual asr and automatic punctuation systems into the source side of our mt training data. this is similar in spirit to the method of @xcite , which introduces artificial sentence boundary errors into the training bitext. however, instead of artificial boundaries, our segmentation approach uses actual boundaries generated by the automatic punctuation system, which required the development of our inverted projection technique, and we also include errors from asr. for a small subset of our training set, we assume access to long-form source audio documents, their corresponding human transcriptions, and translations of those transcriptions. this makes it possible to compare the performance of a baseline model trained on the human transcription with a model trained on source sentences derived from applying asr transcription and automatic punctuation to the same audio. our primary contributions are first to show how to produce training data that captures the errors from automatic transcription and punctuation, which requires a non-trivial re-segmentation of the reference translation that we call inverted projec-tion; and second to show experimentally that it is actually more important to expose the mt system to segmentation errors than lexical transcription errors when aiming for speech-robust mt. ## background compounding errors from asr are known to cause problems when cascaded into mt @xcite . these issues are one of the main motivators for end-to-end modeling of speech translation @xcite @xcite . however, we consider end-to-end modeling out of scope for this study since we aim to benefit from the modularity that comes with a cascaded speech translation strategy. to improve a cascade's robustness to speech input, one can train the mt system with some combination of artificial errors, actual asr output, or long-form segmentation errors. we discuss each in turn. introducing artificial errors into the training set has the advantage of being efficient, and not necessarily tied to a specific asr system. one can add gaussian noise to the source embeddings @xcite or induce lexical substitutions that may be informed by phonetics @xcite . @xcite experiment with a noise model that can perform insertions, deletions and substitutions, but find little value in refining the substitutions to account for word frequency or orthographic similarity. more related to our efforts are those that use actual asr output. early experiments used asr output to replace the source side of parallel text during training @xcite . these did not perform well, likely because asr word error rates (wer) on the fisher corpus were more than 40%, resulting in an unreliable training signal. recently, @xcite showed that, given asr training corpora (coupled audiotranscription pairs), one can build a robust mt system by training with the normal mt objective on mt corpora, plus a mixture of: (1) an adversarial objective that tries to bring encoder representations for asr output close to those of human transcriptions; and (2) a normal mt objective that has asr output as source and machine translations of human transcripts as target. in an iwslt ted translation scenario, they showed large improvements (+2.5 bleu) using the second idea alone, which we take as a strong signal that there is much to be gained by training with asr output on the source side. we consider a long-form scenario where sentence boundaries for the input audio are not given at test time. as such, the method of @xcite to make mt robust to segment boundary errors is very relevant. they introduce artificial sentence boundary errors in their training bitext. they first fragment adjacent source sentences, and then produce analogous fragments in the target according to proportional token lengths. we draw inspiration from their approach when building the target sides of our inverted projections. ## methods our approach to producing mt systems that are robust to automatic transcription errors is to introduce errors from our asr system into our mt training data. throughout the discussion of our methods, we make use of both human (manual) and system (automated) transcriptions of the source audio. when discussing the target-side of our training data, we use instead the term "gold" to indicate a trusted reference translation. throughout our experiments, the gold standard is a human translation of the human transcript @xcite , though it could just as easily, and much less expensively, be a machine translation of the human transcript @xcite . we divide transcription errors into two categories: token and segment errors. a token error is any word that is transcribed incorrectly by asr, such as a homophone substitution or the omission of a mumbled word. meanwhile, segment errors are introduced by failing to correctly break the recognized text into sentence-like segments. a human transcription is expected to have error-free tokens and segments. with system tokens and human segments is the approach taken by others such as @xcite , resulting in token robustness. in the case of longform asr, the human segments can be projected onto the asr output. this is an effective approach for exposing the training model to token errors from asr, but it has an important disadvantage, as it results in a train-test mismatch because the human segments seen during training will not be available at inference time. we describe this approach in section 3.2 to provide a comparison to our approaches using system segments and to introduce some of the concepts and tools used in those approaches. the two approaches using system segments are the main innovations in this paper. introducing segment errors alone results in segment robustness (section 3.3), while segment and token errors together result in system robustness (section 3.4); that is, mt that is robust to the complete long-form transcription pipeline. we will show in the following sections how we can project system segments onto the source and target text; we call this an inverted projection. ## conclusions to aid text-based translation models to adapt to speech data, we introduced an inverted projection approach that projects automatically detected system segments onto human transcripts and then resegments the gold translations to align with the projected human transcripts. this approach overcomes the train-test mismatch present in previous attempts to train on long-form asr output by exposing mt training to both token and segment errors, exactly matching the source transcription pipeline used at test time. the results demonstrate a gain of over 1 bleu point on both iwslt data and youtube data. for future work, we aim to train models on languages with higher asr wer since our english wer is very low (5.5%). we also plan to experiment with mt targets during training to address the data bottleneck. and we also plan to investigate whether we can eliminate segmentation altogether with document-level speech translation.