Consistent Transcription and Translation of Speech (2020)

Abstract: The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. We make this common requirement explicit and explore the task of jointly transcribing and translating speech. Although high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from inconsistencies between both outputs that degrade the user experience. We introduce a methodology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further introduce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy.1

## introduction speech translation (st) is the task of translating acoustic speech signals into text in a foreign language. according to the prevalent framing of st (e.g., @xcite , given some input speech 1 we release human annotations of consistency under https://gi t h u b . com/apple/ml-transcript -translation-consistency-ratings . according to this formulation, st models primarily focus on translation quality, while transcription receives less emphasis. in contrast, practical st user interfaces often display transcripts to the user alongside the translations. a typical example is a two-way conversational st application that displays the transcript to the speaker for verification, and the translation to the conversation partner @xcite . therefore, there is a mismatch between this practical requirement and the prevalent framing as described above. while traditional st models often do commit to a single automatic speech recognition (asr) transcript that is then passed on to a machine translation (mt) component @xcite , researchers have undertaken much effort to mitigate resulting error propagation issues by developing models that avoid making decisions on transcripts. recent examples include direct models @xcite that bypass transcript generation, and lattice-to-sequence models @xcite that translate the asr search space as a whole. despite their merits, such models may not be ideal for scenarios that display both a translation and a corresponding transcript to users. in this paper, we replace eq. 1 by a joint transcription/translation objective to reflect this requirement: ŝ, @xmath0,@xmath1, t | x)} . figure 1 : example of lexical inconsistencies we encountered when generating transcript and translation independently. although the transcript correctly contains replay, the german translation (mistakenly) chooses ersetzen (english: replace). the inconsistency is explained by the acoustic similarity between replay and replace, which is not obvious to a monolingual user. figure 2 : illustration of surface-level consistency between english transcript and german translation. only translation 1 spells both named entities (bill gross and esolar) consistently, and the german translation solarthermaltechnologie (translation 1) is preferred over solarwärme-technologie (translation 2), by itself a correct choice but less similar on the surface level. on evaluation. first, besides translation accuracy, transcription accuracy becomes relevant and equally important. second, the issue of consistency between transcript and translation becomes essential. for example, let us consider a naive approach of transcribing and translating with two completely independent, potentially erroneous models. these independent models would expectedly produce inconsistencies, including inconsistent lexical choice caused by acoustic or linguistic ambiguity (figure 1 ), and inconsistent spelling of named entities (figure 2 ). even if output quality is high on average, such inconsistencies may considerably degrade the user experience. our contributions are threefold: first, we introduce the notion of consistency between transcripts and translations and propose methods to assess consistency quantitatively. second, we survey and extend existing models, and develop novel training and inference schemes, under the hypothesis that both joint model training and a coupled inference procedure are desirable for our goal of accurate and consistent models. third, we provide a comprehensive analysis, comparing accuracy and consistency for a wide variety of model types across several language pairs to determine the most suitable models for our task and analyze potential trade-offs. ## evaluation beyond accuracy-the need for consistency to better understand the desiderata of models that perform transcription and translation, it is helpful to discuss how one should evaluate such models. a first step is to evaluate transcription accuracy and translation accuracy in isolation. for this purpose, we can use well-established evaluation metrics such as word error rate (wer) for transcripts and bleu @xcite for translations. when considering scenarios in which both transcript and translation are displayed, consistency is an essential additional requirement. 2 let us first clarify what we mean by this term. definition: consistency between transcript and translation is achieved if both are semantically equivalent, with a preference for a faithful translation approach @xcite , meaning that stylistic, lexical, and grammatical characteristics should be transferred whenever fluency is not compromised. importantly, consistency measures are defined over the space of both well-formed and erroneous sentence pairs. in the case of ungrammatical sentence pairs, consistency may be achieved by adhering to a literal or word-for-word translation strategy. consistency is only loosely related to accuracy, and can even be in opposition in some cases. for instance, when a translation error cannot be avoided, consistency is improved at the cost of transcription accuracy by placing the backtranslated error in the transcript. because accuracy and error metrics assess transcript or translation quality in isolation, these metrics cannot capture phenomena that involve the interplay between transcript and translation. ## estimating consistency having argued for consistency as a desirable property, we now wish to empirically quantify the level of consistency between a particular model's transcripts and translations. to our knowledge, consistency has not been addressed in the context of st before, perhaps because traditional cascaded models have not been observed to suffer from inconsistencies in the outputs. therefore, we propose several metrics for estimating transcript/translation consistency in this section. in §7.3, we demonstrate strong agreement of these metrics with human ratings of consistency. ## models for transcription and translation we now turn to discuss model candidates for consistent transcription and translation of speech (figures 4 5 ). we hypothesize that there are two desirable model characteristics in our scenario. first, motivated by eq. 2, models may achieve better consistency by performing joint inference, in the sense that no independence assumption between transcript and translation are made. we call this characteristic coupled inference. second, shared representations through end-to-end (or joint) training may be of advantage in our scenario. we introduce several model variants, and also discuss whether they match these characteristics. ## consistency as training and inference objectives having surveyed models that are suitable for our task to various degrees, we next explore simple ways to further improve the consistency of the generated outputs through adjusting training or inference objectives. ## results we start by presenting empirical results across all four language pairs, and will then focus on english-german to discuss details. ## conclusion this paper investigates the task of jointly transcribing and translating speech, which is relevant for use cases in which both transcripts and translations are displayed to users. the main theme has been the discussion of consistency between transcripts and translations. to this end, we proposed a notion of consistency and introduced techniques to estimate it. we conducted a thorough comparison across a wide range of models, both traditional and end-to-end trainable, with regards to both accuracy and consistency. as important model ingredients, we found that a coupled inference procedure, where translations are conditioned on transcripts through attention, is particularly helpful. we also found that end-toend training improves consistency and translations but at the cost of degraded transcripts. we further introduced training and inference techniques that are effective at further improving consistency, which we found to also come with some trade-offs. future work should examine how consistency correlates with user experience in practice and establish specific trade-offs for various use cases. moreover, our techniques are applicable to other multitask use cases that could potentially benefit from consistent outputs.