Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020)

Abstract: Over its three decade history, speech translation has experienced several shifts in its primary research themes; moving from loosely coupled cascades of speech recognition and machine translation, to exploring questions of tight coupling, and finally to end-to-end models that have recently attracted much attention. This paper provides a brief survey of these developments, along with a discussion of the main challenges of traditional approaches which stem from committing to intermediate representations from the speech recognizer, and from training cascaded models separately towards different objectives. Recent end-to-end modeling techniques promise a principled way of overcoming these issues by allowing joint training of all model components and removing the need for explicit intermediate representations. However, a closer look reveals that many end-to-end models fall short of solving these issues, due to compromises made to address data scarcity. This paper provides a unifying categorization and nomenclature that covers both traditional and recent approaches and that may help researchers by highlighting both trade-offs and open research questions.

## introduction speech translation (st), the task of translating acoustic speech signals into text in a foreign language, is a complex and multi-faceted task that builds upon work in automatic speech recognition (asr) and machine translation (mt). st applications are diverse and include travel assistants @xcite , simultaneous lecture translation @xcite , movie dubbing/subtitling @xcite , language documentation and crisis response @xcite , and developmental efforts @xcite . until recently, the only feasible approach has been the cascaded approach that applies an asr to the speech inputs, and then passes the results on to an mt system. progress in st has come from two fronts: general improvements in asr and mt models, and moving from the loosely-coupled cascade in its most basic form toward a tighter coupling. however, despite considerable efforts toward tight coupling, a large share of the progress has arguably been owed simply to general asr and mt improvements. 1 recently, new modeling techniques and in particular end-to-end trainable encoder-decoder models have fueled hope for addressing challenges of st in a more principled manner. despite these hopes, the empirical evidence indicates that the success of such efforts has so far been mixed @xcite . in this paper, we will attempt to uncover potential reasons for this. we start by surveying models proposed throughout the three-decade history of st. by contrasting the extreme points of loosely coupled cascades vs. purely end-to-end trained direct models, we identify foundational challenges: erroneous early decisions, mismatch between spokenstyle asr outputs and written-style mt inputs, and loss of speech information (e.g. prosody) on the one hand, and data scarcity on the other hand. we then show that to improve data efficiency, most endto-end models employ techniques that re-introduce issues generally attributed to cascaded st. furthermore, this paper proposes a categorization of st research into well-defined terms for the particular challenges, requirements, and techniques that are being addressed or used. this multidimensional categorization suggests a modeling space with many intermediate points, rather than a dichotomy of cascaded vs. end-to-end models, and reveals a number of trade-offs between different modeling choices. this implies that additional work to more explicitly analyze the interactions between these trade-offs, along with further model explorations, can help to determine more favorable points in the modeling space, and ultimately the most favorable model for a specific st application. ## chronological survey this chapter surveys the historical development of st and introduces key concepts that will be expanded upon later. foot_2 ## central challenges given the abundance of prior work, a clear picture on where we currently stand is needed. for purposes of identifying the key challenges in st research, this section will contrast the extreme cases of the loosely coupled cascade (cc in fig. 1 ) foot_3 against the vanilla direct model (di in fig. 1 ). 4 we emphasize that these models are only extreme points in a modeling space with many intermediate points, as we will see in §4. we assume appropriate speech features x as inputs. t, @xmath0, respectively, from the mt hypothesis space. @xmath1. ## modeling techniques let us now break apart modeling techniques from prior literature into four overarching categories, with the aim of exposing the st modeling space between the extreme points of vanilla direct models and loosely coupled cascades. ## applications and requirements while we previously described the task of st simply as the task of generating accurate text translations from speech inputs, the reality is in fact much more complicated. future work may exploit new modeling techniques to explicitly address the aspects drawn out below. ## discussion recent works on end-to-end modeling techniques are motivated by the prospect of overcoming the loosely coupled cascade's inherent issues, yet of the issues outlined in §2.1, often only the goal of avoiding early decisions is mentioned motivationally. while early decisions and data efficiency have been recognized as central issues, empirical insights are still limited and further analysis is needed. mismatched source-language and information loss are often not explicitly analyzed. we conjecture that the apparent trade-off between data efficiency and modeling power may explain the mixed success in outperforming the loosely coupled cascade. in order to make progress in this regard, the involved issues (early decisions, mismatched source-language, information loss, data efficiency) need to be precisely analyzed ( §3), and more model variants ( §4) should be explored. as a possible starting point one may aim to extend, rather than alter, traditional models, e.g. applying end-to-end training as a fine-tuning step, employing a direct model for rescoring, or adding a triangle connection to a loosely coupled cascade. we further suggest that more principled solutions to the different application-specific requirements ( §5) should be attempted. perhaps it is possible to get rid of segmentation as a separate step in batch delivery mode, or perhaps text as output medium can be used to visualize repairs more effectively. several of the application-specific requirements demand user studies and will not be sufficiently solved by relying on automatic metrics only. ## conclusion we started this paper with a chronological survey of three decades of st research, focusing on carving out the key concepts. we then provided definitions of the central challenges, techniques, and requirements, motivated by the observation that recent work does not sufficiently analyze these challenges. we exposed a significant space of both modeling ideas and application-specific requirements left to be addressed in future research. our hope is to encourage meaningful and generalizable comparisons on our quest toward overcoming the long-standing issues found in st models.