N eur ST : Neural Speech Translation Toolkit (2021)

Abstract: NeurST is an open-source toolkit for neural speech translation. The toolkit mainly focuses on end-to-end speech translation, which is easy to use, modify, and extend to advanced speech translation research and products. NeurST aims at facilitating the speech translation research for NLP researchers and building reliable benchmarks for this field. It provides step-by-step recipes for feature extraction, data preprocessing, distributed training, and evaluation. In this paper, we will introduce the framework design of NeurST and show experimental results for different benchmark datasets, which can be regarded as reliable baselines for future research. The toolkit is publicly available at https://github.com/bytedance/neurst and we will continuously update the performance of with other counterparts and studies at https://st-benchmark.github.io/ .

## introduction speech translation (st), which translates audio signals of speech in one language into text in a foreign language, is a hot research subject nowadays and has widespread applications, like cross-language videoconferencing or customer support chats. traditionally, researchers build a speech translation system via a cascading manner, including an automatic speech recognition (asr) and a machine translation (mt) subsystem @xcite @xcite . cascade systems, however, suffer from error propagation problems, where an inaccurate asr output would theoretically cause translation errors. owing to recent progress of sequence-to-sequence modeling for both neural machine translation (nmt) @xcite @xcite and end-to-end speech recognition @xcite @xcite , it becomes feasible and efficient to train an end-toend direct st model @xcite @xcite . this end-to-end fashion attracts much attention due to its appealing properties: a) modeling without intermediate asr transcriptions obviously alleviates the propagation of errors; b) a single and unified st model is beneficial to deployment with lower latency in contrast to cascade systems. recent studies show that end-to-end st models achieve promising performance and are comparable with cascaded models @xcite . the end-to-end solution has great potential to be the dominant technology for speech translation, however challenges remain. the first is about benchmarks. many st studies conduct experiments on different datasets. @xcite evaluate the method on ted english-chinese; @xcite use libri-trans english-french and iwslt2018 english-german dataset; and @xcite show the results on covost dataset and the fr/ro portions of must-c dataset. different datasets make it difficult to compare the performance of their approaches. further, even for the same dataset, the baseline results are not necessarily kept consistent. take the libri-trans english-french dataset as an example. dong et al. (2021 2020 ) report 15.5 (detokenized bleu). the mismatching baseline results in an unfair comparison on the improvements of their approaches. we think one of the primary reasons is that the preprocessing of audio data is complex, and the st model training involves many tricks, such as pre-training and data augmentation. therefore a reproducible and reliable benchmark is required. in this work, we present neurst , a toolkit for easily building and training end-toend st models, as well as end-to-end asr and nmt for cascade systems. we implement state-ofthe-art transformer-based models @xcite and provide step-by-step recipes for feature extraction, data preprocessing, model training, and inference for researchers to reproduce the benchmarks. though there exist several counterparts, such as lingvo @xcite , fairseq-st @xcite and kaldi 1 style espnet-st @xcite , neurst is specially designed for speech translation tasks, which encapsulates the details of speech processing and frees the developers from data engineering. it is easy to use and extend. the contributions of this work are as follows: ## design and features neurst is implemented with both tensorflow2 and pytorch backends. in this section, we will introduce the design components and features of this toolkit. ## speech translation benchmarks we conducted experiments on several benchmark speech translation corpora using neurst and compared the performance with other open-source codebases and studies. though that would be an unfair comparison due to the different model structures and hyperparameters, the goal of neurst is to provide strong and reproducible benchmarks for future research. ## conclusion we introduce neurst toolkit for easily building and training end-to-end speech translation models. we provide straightforward recipes for audio data pre-processing, training, and inference, which we believe is friendly with nlp researchers. moreover, we report strong and reproducible benchmarks and will continuously catch up on advanced progress using neurst , which can be regarded as the reliable baselines for the st field.