Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? (2021)

Abstract: Five years after the first published proofs of concept, direct approaches to speech translation (ST) are now competing with traditional cascade solutions. In light of this steady progress, can we claim that the performance gap between the two is closed? Starting from this question, we present a systematic comparison between state-of-the-art systems representative of the two paradigms. Focusing on three language directions (English-German/Italian/Spanish), we conduct automatic and manual evaluations, exploiting high-quality professional post-edits and annotations. Our multi-faceted analysis on one of the few publicly available ST benchmarks attests for the first time that: i) the gap between the two paradigms is now closed, and ii) the subtle differences observed in their behavior are not sufficient for humans neither to distinguish them nor to prefer one over the other.

## introduction speech translation (st) is the task of automatically translating a speech signal in a given language into a text in another language. research on st dates back to the late eighties and its evolution followed the development of the closely related fields of speech recognition (asr) and machine translation (mt) that, since the very beginning, provided the main pillars for building the so-called cascade architectures. with the advent of deep learning, the neural networks widely used in asr and mt have been adapted to develop a new direct st paradigm. this approach aims to overcome known limitations of the cascade one (e.g. architectural complexity, error propagation) with a single encoder-decoder architecture that directly translates the source signal bypassing intermediate representations. until now, the consolidated underlying technologies and the richness of available data have upheld the supremacy of cascade solutions in industrial applications. however, architectural simplicity, reduced information loss and error propagation are the ace up the sleeve of the direct approach, which has rapidly gained popularity within the research community in spite of the critical bottleneck represented by data paucity. within a few years after the first proofs of concept @xcite , the performance gap between the two paradigms has gradually decreased. this trend is mirrored by the findings of the international workshop on spoken language translation (iwslt), foot_0 a yearly evaluation campaign where direct systems made their first appearance in 2018. on english-german, for instance, the bleu difference between the best cascade and direct models dropped from 7.4 points in 2018 @xcite to 1.6 points in 2019 @xcite . in 2020, participants were allowed to choose between processing a presegmented version of the test set or the one produced by their own segmentation algorithm. as reported in @xcite , the distance between the two paradigms further decreased to 1.0 bleu point in the first condition and, for the first time, it was slightly in favor of the best direct model in the second condition, with a small but nonetheless meaningful 0.24 difference. so, quoting @xcite , is the cascade solution still the dominant technology in st? has the direct approach closed the huge initial performance gap? are there systematic differences in the outputs of the two technologies? are they distinguishable? answering these questions is more than running an evaluation exercise. it implies pushing research towards a deeper investigation of direct st, finding a path towards its wider adoption in industrial settings and motivating higher engagement in data exploitation and resource creation to train the data-hungry end-to-end neural systems. for all these reasons, while ansari et al. (2020) were cautious in drawing firm conclusions, in this paper we delve deeper into the problem with the first thorough comparison between the two paradigms. working on three language directions (en-de/es/it), we train state-of-the-art cascade and direct models ( §3), running them on test data drawn from the must-c corpus @xcite . systems' behavior is analysed from different perspectives, by exploiting high-quality post-edits and annotations by professionals. after discussing overall systems' performance ( §4), we move to more fine-grained automatic and manual analyses covering two main aspects: the relation between systems' performance and specific characteristics of the input audio ( §5), and the possible differences in terms of lexical, morphological and word ordering errors ( §6). we finally explore whether, due to latent characteristics overlooked by all previous investigations, the output of cascade and direct systems can be distinguished either by a human or by an automatic classifier ( §7). together with a comparative study attesting the parity of the two paradigms on our test data, another contribution of this paper is the release of the manual post-edits that rendered our investigation possible. the data is available at: https://ict.fbk.eu/mustc-post-edits . ## background cascade st. by concatenating asr and mt components @xcite , cascade st architectures represent an intuitive solution to achieve reasonable performance and high adaptability across languages and domains. at the same time, however, they suffer from well-known problems related to the concatenation of multiple systems. first, they require ad-hoc training and maintenance procedures for the asr and mt modules; second, they suffer from error propagation and from the loss of speech information (e.g. prosody) that might be useful to improve final translations. research has focused on mitigating error propagation by: i) feeding the mt system with asr data structures (e.g. asr n-best, lattices or confusion networks) which are more informative than the 1-best output @xcite @xcite @xcite , and ii) making the mt robust to asr errors, for instance by training it on parallel data incorporating real or emulated asr errors as in @xcite @xcite @xcite . although the former solutions are effective to some extent, state-of-theart cascade architectures @xcite prefer the latter, as they are simpler to implement and maintain. ## overall systems' performance we compute overall performance results both on the pe-sets and on the must-c common test sets. our primary evaluation is based on the collected post-edits. we consider two ter-based foot_6 metrics: i) human-targeted ter (hter) computed between the automatic translation and its human post-edited version, and ii) multi-reference ter (mter) computed against the closest reference among the three available ones (two post-edits and the official reference from must-c). the latter metric better accounts for post-editors' variability, making the evaluation more reliable and informative. for the sake of completeness, in a bird's-eye view of the results shows that, in more than half of the cases, performance differences between cascade and direct systems are not statistically significant. when they are, the raw count of wins for the two approaches is the same (4), attesting their substantial parity. looking at our primary metrics (hter and mter), systems are on par on en-it and en-de, while for en-es the direct approach significantly outperforms the cascade one. this difference, however, does not emerge with the other metrics. indeed, bleu and ter scores computed against the official references are less coherent across metrics and test sets. for instance, on the en-it pe-set the cascade system significantly outperforms the direct one in terms of bleu score, while ter shows the opposite on must-c common. interestingly, the scores obtained using independent references can also disagree with those computed with post-edits. this is the case of en-es, where significant hter and mter reductions attest the superiority of the direct system, while most bleu and ter scores are still in favor of the cascade. on the one hand, primary evaluation scores suggest that the rapidly advancing direct technology has eventually reached the traditional cascaded approach. on the other, the highlighted incongruities confirm widespread concerns about the reliability of fully automatic metrics -based on independent references -to properly evaluate neural systems @xcite . this calls for deeper quantitative and qualitative analyses. those presented in the next sections investigate performance differences focusing on two main aspects: the impact of specific input audio properties ( §5), and the linguistic errors made by the systems ( §6). ## classifiers' verdict so far, our inquiry has been entirely driven by predefined assumptions (the importance of certain audio properties) and linguistic criteria (the focus on specific error types). this top-down approach, however, might fail to disclose important differences, which were not specifically sought after when analysing the two paradigms. this consideration motivates the adoption of the complementary bottom-up approach that concludes our comparative study by answering the question: is the output of cascade and direct systems distinguishable? understanding if and why discriminating between the two is possible would not only suggest new issues to look at. it would also highlight possible output regularities that, despite the similar overall performance, make one paradigm preferable over the other in specific application scenarios. to this aim, we set up a classification experiment, comparing the ability of humans to correctly identify the output of the two systems with the performance of an automatic text classifier. ## conclusion and final remarks there is a time when the possible transition from consolidated technological frameworks to new emerging paradigms depends on answering fundamental questions about their potential, strengths and weaknesses. a time when technology developers are faced with the choice of where to direct their future investments. five years after its appearance on the scene, the direct approach to st confronts the community with similar questions in relation to the traditional cascade paradigm that it aims to overtake. our investigation showed that, in spite of the known data paucity conditions still penalizing the direct approach, the two technologies now perform substantially on par. subtle differences in their behavior exist: overall performance being equal, the cascade still seems to have an edge in terms of morphology, word ordering and lexical diversity, which is balanced by the advantages of direct models in audio understanding and in capturing prosody. however, they do not seem sufficient and consistent enough across languages to make the output of the two approaches easily distinguishable, nor to make one model preferable to the other. back to our title, they no longer make a difference. we are aware that the generalizability of these results depends on several factors such as the considered languages, systems and benchmarks, as well as the human workforce deployed for the inquiry. here, with the help of professionals, we proposed multi-faceted quantitative and qualitative analyses, run on the output of state-of-the-art systems on three language pairs -though, by now, covering only the most-explored and data-favorable condition, which has english as source. although our findings hold for a specific scenario, in which free data were at our disposal (and to which we contribute back by releasing high-quality post-edits), they might not be generalizable to other (e.g. difficult, distant) languages and other (e.g. highly specialized) domains. nevertheless, we present them as a timely contribution towards answering a burning question within the st community. the maximum value (5 × 10 -4 ), after that it follows an inverse square root decay; dropout is set to 0.3. minibatches consist of 3072 tokens and update frequency is set to 4; the total number of iterations is 200k; the last 10 saved checkpoints (one out of 1k iterations) are averaged. the model uses label smoothing with a uniform prior distribution (0.1) over the vocabulary; source and target languages share a bpe vocabulary of 32k sub-words. the training data, whose statistics are reported in the three models are then fine-tuned on must-c training data (∼250k pairs, 4-5m english words) by continuing the training for 4k iterations on the adaptation data, with a learning rate reduced by a factor of 5. to mitigate error propagation and make the mt system more robust to asr errors, similarly to @xcite fine tuning is run on the concatenation of human and automatic transcripts of must-c, both paired with manual translations.