ESP net- ST -v2: Multipurpose Spoken Language Translation Toolkit (2023)

Abstract: ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community. ESPnet-ST-v2 supports 1) offline speech-to-text translation (ST), 2) simultaneous speech-to-text translation (SST), and 3) offline speech-to-speech translation (S2ST) – each task is supported with a wide variety of approaches, differentiating ESPnet-ST-v2 from other open source spoken language translation toolkits. This toolkit offers state-of-the-art architectures such as transducers, hybrid CTC/attention, multi-decoders with searchable intermediates, time-synchronous blockwise CTC/attention, Translatotron models, and direct discrete unit models. In this paper, we describe the overall design, example models for each task, and performance benchmarking behind ESPnet-ST-v2, which is publicly available at https://github.com/espnet/espnet .

## introduction the objective of this project is to contribute to the diversity of the open-source spoken language translation ecosystem. toward this, we launched this espnet-st-v2 update in collaboration with researchers working on fairseq @xcite and torchaudio @xcite . this project focuses on: offline speech-to-text (st), simultaneous speech-to-text (sst), and offline speech-to-speech (s2st). these three spoken language translation tasks have drawn significant interest, as evidenced by rising iwslt 2 shared task participation. the st task can be considered a base form of spoken language translation. early approaches to st stemmed from coupling statistical automatic speech recognition (asr) @xcite ) and text-to-text translation (mt) @xcite , and this type of cascaded approach is still 1 please see our documentation for st/sst and s2st to get started. example models and tutorials are provided. given the recent swell in e2e st, sst, and s2st research, we have revamped espnet-st @xcite which previously only supported e2e st. in particular, this work: with this major update, espnet-st-v2 keeps pace with the interests of the community and offers a variety of unique features, making it a valuable complement to fairseq @xcite , neurst @xcite , and other spoken language translation toolkits. ## related works espnet-st-v2 follows a long line of open-source speech processing toolkits which can support spoken language translation @xcite @xcite @xcite . in ## espnet-st-v2 in this section, we first describe the overall design and then introduce a few key features. ## example models in this section, we introduce example models which are pre-built in espnet-st-v2 using the neural network components described in the previous section. these examples include state-of-the-art core architectures, as evidenced by prior studies and our performance benchmarking (presented in §5). the final encoder representations to be monotonic with respect to the target. ctc/attention models are jointly decoded using either label-synchronous (wherein the attention branch is primary) or timesynchronous (wherein the ctc branch is primary) beam search. for offline tasks, label-synchrony has shown greater performance @xcite . multi-decoder ctc/attention (mca) as shown in figure 2 , the multi-decoder decomposes st into two sub-tasks, logically corresponding to asr and mt encoder-decoder models, while maintaining e2e differentiability @xcite . this multi-decoder scheme is also combined with the ctc/attention scheme described in the blurb above, following @xcite . we use conformer encoders with hierarchical ctc for encoding speech and transformer encoders for encoding intermediate asr text. we use transformer decoders for both asr and st. during inference, the asr stage is decoded first and then the final mt/st stage is decoded; both stages use label-synchronous joint ctc/attention beam search. ## performance benchmarking in this section, we 1) compare open-source toolkits 2) compare our different example models and 3) compare our models with top iwslt shared task systems and state-of-the-art prior works. ## conclusion we presented espnet-st-v2 which now supports offline speech translation, simultaneous speech translation, and offline speech-to-speech translation. espnet-st-v2 will continue to grow to support the community's interests. future updates may include more new tasks, such as simultaneous speech-to-speech translation, and cross-toolkit integrations via torchaudio.