Fairseq S 2 T : Fast Speech-to-Text Modeling with Fairseq (2020)

Abstract: We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq’s careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based as well as Transformer-based models and open-source detailed training recipes. Fairseq’s machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T is available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text .

## introduction end-to-end sequence-to-sequence (s2s) modeling has witnessed rapidly increased applications in speech-to-text (s2t) tasks. it achieves state-of-theart performance on automatic speech recognition (asr) @xcite and leads to the recent resurgence of speech-totext translation (st) research @xcite . asr and st are closely related. there are recent attempts to combine the two tasks under the same s2s model architecture via multi-task learning @xcite . they also benefit from each other via transfer learning @xcite and are able to leverage additional supervision from machine translation (mt) and language modeling (lm). when supervised data is not abundant, self-supervised pretraining @xcite and semi-supervised training @xcite lowers the requirements on supervision and improves model performance. the increased connections among asr, st, mt and lm has called for all-in-one s2s modeling toolkits, and the use of large-scale unlabeled speech data sets the scalability requirements. in this paper, we introduce fairseq s2t, a fairseq @xcite extension for s2t tasks such as end-to-end asr and st. it follows fairseq's careful design for scalability and extensibility. we provide end-to-end workflows from data pre-processing, model training to offline (online) inference. we implement state-of-the-art rnn-based @xcite and transformer-based @xcite models and open-source detailed training recipes. fairseq's mt models and lms can be seamlessly integrated into s2t workflows for multi-task learning or transfer learning. to facilitate model evaluation, we add a collection of scorers as well as vizseq @xcite integration for visualized error analysis. fairseq s2t documentation and examples are available at https://github.com/pytorch/fairseq/ tree/master/examples/speech_to_text . with counterpart toolkits such as espnet @xcite and lingvo @xcite , fairseq s2t pursues the best integration, scalability and reproducibility. a detailed comparison of fairseq s2t with its counterparts can be found in ## conclusion we introduce fairseq s2t, a fairseq extension for speech-to-text (s2t) modeling tasks such as speech recognition and speech translation. it includes end-to-end workflows and state-of-theart models with scalablity and extensibility design. it seamlessly integrates fairseq's machine translation models and language models to improve s2t model performance. fairseq s2t documentation and examples are available at https://github.com/pytorch/fairseq/ tree/master/examples/speech_to_text .