ESP net- ST : All-in-One Speech Translation Toolkit (2020)

Abstract: We present ESPnet-ST, which is designed for the quick development of speech-to-speech translation systems in a single framework. ESPnet-ST is a new project inside end-to-end speech processing toolkit, ESPnet, which integrates or newly implements automatic speech recognition, machine translation, and text-to-speech functions for speech translation. We provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines for a wide range of benchmark datasets. Our reproducible results can match or even outperform the current state-of-the-art performances; these pre-trained models are downloadable. The toolkit is publicly available at https://github.com/espnet/espnet .

## introduction speech translation (st), where converting speech signals in a language to text in another language, is a key technique to break the language barrier for human communication. traditional st systems involve cascading automatic speech recognition (asr), text normalization (e.g., punctuation insertion, case restoration), and machine translation (mt) modules; we call this cascade-st @xcite @xcite . recently, sequence-to-sequence (s2s) models have become the method of choice in implementing both the asr and mt modules (c.f. @xcite ). this convergence of models has opened up the possibility of designing end-to-end speech translation (e2e-st) systems, where a single s2s directly maps speech in a source language to its translation in the target language @xcite . e2e-st has several advantages over the cascaded approach: (1) a single e2e-st model can reduce latency at inference time, which is useful for time-critical use cases like simultaneous interpretation. (2) a single model enables back-propagation training in an end-to-end fashion, which mitigates the risk of error propagation by cascaded modules. we believe the time is ripe to develop a unified toolkit that facilitates research in both e2e and cascaded approaches. we present espnet-st, a toolkit that implements many of the recent models for e2e-st, as well as the asr and mt modules for cascade-st. our goal is to provide a toolkit where researchers can easily incorporate and test new ideas under different approaches. recent research suggests that pre-training, multi-task learning, and transfer learning are important techniques for achieving improved results for e2e-st @xcite @xcite . thus, a unified toolkit that enables researchers to seamlessly mix-and-match different asr and mt models in training both e2e-st and cascade-st systems would facilitate research in the field. 1 espnet-st is especially designed to target the st task. espnet was originally developed for the 1 there exist many excellent toolkits that support both asr and mt tasks (see toolkit supported task example (w/ corpus pre-processing) pre-trained model asr lm e2e-cascade-mt tts asr lm e2e-cascade-mt tts st st st st espnet-st (ours) lingvo 1 ♣ ♣ ♣ asr task @xcite , and recently extended to the text-to-speech (tts) task @xcite . here, we extend espnet to st tasks, providing code for building translation systems and recipes (i.e., scripts that encapsulate the entire training/inference procedure for reproducibility purposes) for a wide range of st benchmarks. this is a non-trivial extension: with a unified codebase for asr/mt/st and a wide range of recipes, we believe espnet-st is an all-in-one toolkit that should make it easier for both asr and mt researchers to get started in st research. the contributions of espnet-st are as follows: ## example models to give a flavor of the models that are supported with espnet-st, we describe in detail the construction of an example e2e-st model, which is used later in the experiments section. note that there are many customizable options not mentioned here. ## conclusion we presented espnet-st for the fast development of end-to-end and cascaded st systems. we provide various all-in-one example scripts containing corpus-dependent pre-processing, feature extraction, training, and inference. in the future, we will support more corpora and implement novel techniques to bridge the gap between end-to-end and cascaded approaches.