Tutorial: End-to-End Speech Translation (2021)

Abstract: Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation. Speech translation has attracted interest for many years, but the recent successful applications of deep learning to both individual tasks have enabled new opportunities through joint modeling, in what we today call ‘end-to-end speech translation.’ In this tutorial we introduce the techniques used in cutting-edge research on speech translation. Starting from the traditional cascaded approach, we give an overview on data sources and model architectures to achieve state-of-the art performance with end-to-end speech translation for both high- and low-resource languages. In addition, we discuss methods to evaluate analyze the proposed solutions, as well as the challenges faced when applying speech translation models for real-world applications.

## description machine translation (mt) and automatic speech recognition (asr) have been mainstays of the speech and natural language processing communities for decades. speech translation (st), the combination of both tasks to translate from speech in one language typically to text in another, has existed for nearly as long as either of these @xcite , attracting interest from both academia and industry. until very recently, however, research in this area involved a cascade of separately trained speech recognition and machine translation models, with main questions pertaining to intermediate representations and processing steps to best connect these models. the successful application of deep learning methods to speech and language processing has not only significantly improved the quality of models for both tasks @xcite , but has also enabled new opportunities through joint modeling of speech and translation in what is today referred to as end-to-end speech translation @xcite . by integrating ideas from machine translation and speech recognition, this research topic is at the intersection of speech and language processing, traditionally two separate communities. the paradigm switch to neural, end-to-end models has brought a significant increase in research interest and data resources for st. the yearly evaluation campaign organized by iwslt has seen large increases in participation in recent years @xcite , and this year brought the creation of a joint special interest group (sigslt) spanning the acl and isca communities. "simpler" sequence-to-sequence architectures have lowered the barrier to entry; where previously researchers wishing to work in this area typically needed to either have significant knowledge of both asr and mt or work in large collaborations, this is no longer the case. however, it remains the case that the best-performing models do draw on insights from both of these fields, and so we think that the time is ripe for a tutorial to better introduce the techniques to do cutting-edge research in st. this tutorial will summarize recent developments in end-to-end speech translation. we will start with discussion about the term 'end-to-end' foot_0 as well as a comparison to the traditional cascaded approach. in the subsequent sections, we will summarize ideas leveraged from automatic speech recognition (e.g. @xcite ) and machine translation (e.g. @xcite ) that are part of current state-of-the-art models, which are cur-rently demonstrated through evaluation campaigns like iwslt. a particular focus point of the tutorial will be the current data landscape, as well as techniques to exploit different resources @xcite to enable speech translation not just for the few high-resource languages for which multi-parallel speech, transcripts, and translations exist. after the survey of current state-of-the-art methods, we will present evaluation and analysis methods, and challenges when bringing these models from the lab to real-world environments. for example, one challenge of end-to-end models is their 'opaqueness'; with one joint system, it is more difficult to isolate causes of particular model behaviors and perhaps intervene, to avoid situations where key terms are translated in unexpected ways. further, most training examples used fixed, presegmented input with parallel sentences, while in most practical applications the audio is not segmented. this brings additional challenges both in processing and also scoring. finally, there are aspects of speech, such as speaker gender, accent, and prosody, which in cascaded systems the mt model did not have access to. we will touch on the impacts of some of these aspects, and provide greater detail about the specific example of gender bias mitigation @xcite . during the tutorial, we will highlight the present successes and challenges in end-to-end speech translation using examples from current state-ofthe-art systems. resources and teaching materials will be made available at https://st-tutorial. github.io . ## tutorial type this tutorial will cover cutting-edge research in the emerging field of end-to-end speech translation, and the aspects from speech and mt needed for this interdisciplinary research. the topic has not been previously covered in *cl tutorials. ## prerequisites we would assume acquaintance with basic knowledge of machine learning and sequence-tosequence models for machine translation, such as are covered in most introductory nlp courses. any programming examples will be shown in python. ## reading list jan niehues, maastricht university email: jan.niehues@maastrichtuniversity.nl website: https://dke.maastrichtuniversity. nl/jan.niehues/ jan niehues is an assistant professor at maastricht university. he received his doctoral degree from karlsruhe institute of technology in 2014 on the topic of "domain adaptation in machine translation." he has conducted research at carnegie mellon university and limsi/cnrs, paris. his research has covered different aspects of machine translation and spoken language translation. he has been involved in several international projects on spoken language translation e.g. the german-french project quaero, the h2020 eu project qt21 eu-bridge and elitr. currently, he is one of the main organizers of the spoken language track in the iwslt shared task. elizabeth salesky, johns hopkins university email: esalesky@jhu.edu website: https://esalesky.github.io elizabeth salesky is a phd student at johns hopkins university. she has previously studied at carnegie mellon university, been a research assistant at karlsruhe institute of technology, and worked at mit lincoln laboratory, focused on speech and text translation. her research focuses on speech translation for real-world and low-resource scenarios; e.g. phoneme features to reduce data dependence, disfluency removal in translating conversational speech, and learning robust representations. she has organized shared tasks on speech translation at iwslt. marco turchi, fondazione bruno kessler email: turchi@fbk.eu website: http://ict.fbk.eu/people/detail/ marco-turchi/ marco turchi is the head of the machine translation unit at fondazione bruno kessler (fbk). he received his phd degree in computer science from the u. of siena, italy in 2006. before joining fbk in 2012, he worked at the european commission, at the university of bristol, at the xerox research centre europe, and at yahoo research lab. his research activities focus on various aspects of sequence-to-sequence modelling applied to machine translation, speech translation and automatic post-editing. he is the co-organizer of the conference of machine translation, the spoken language translation workshop and the automatic post-editing evaluation campaigns. he has been involved in several eu projects such as smart, matecat, modernmt and qt21. he was the recipient of the amazon aws ml research awards on the topic of end-to-end spoken language translation in rich data conditions. he is the secretary of the isca sigslt interest group. matteo negri fondazione bruno kessler email: negri@fbk.eu website: https://ict.fbk.eu/people/detail/ matteo-negri matteo negri is a senior researcher in the machine translation unit at fondazione bruno kessler. he received his degree in philosophy of language from the university of turin, italy in 2000. his research interests are in the field of computational linguistics, particularly machine translation, spoken language translation, textual entailment and question answering. he worked in several eu projects (qt21, cracker, mmt, mate-cat, cosyne, qall-me) and co-organised conferences, workshops and evaluation campaigns in nlp and mt-related areas (including the conference on machine translation, the international workshop on spoken language translation and se-meval shared tasks). together with marco turchi, he was the recipient of an amazon aws ml research award on "end-to-end spoken language translation in rich data conditions."