fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021)

Abstract: This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built and their importance is shown empirically. To facilitate faster iteration of development and analysis, a suite of automatic metrics is included. Apart from the features added specifically for this extension, fairseq Sˆ2 also benefits from the scalability offered by fairseq and can be easily integrated with other state-of-the-art systems provided in this framework. The code, documentation, and pre-trained models will be made available at https://github.com/pytorch/fairseq/tree/master/examples/speech_synthesis .

## introduction speech synthesis is the task of generating speech waveforms with desired characteristics, including but not limited to textual content @xcite @xcite @xcite , speaker identity @xcite , and speaking styles @xcite @xcite . it is also more often referred to as text-to-speech (tts) when text is used as input to the system. along with automatic speech recognition (asr) and machine translation (mt), these language technologies have advanced rapidly over the past few years @xcite . traditionally, these tasks may be used in conjunction to form a system (e.g., combining the three for speech-to-speech translation), but they rarely leverage each other during training. as a result, each application used to have its own dedicated open-source toolkit, for example, equal contribution. kaldi @xcite and htk @xcite for asr, hts @xcite , merlin @xcite , straight @xcite , and world @xcite for speech synthesis, and moses @xcite for mt. recently, there are growing interactions among these systems in the learning process. for example, @xcite and @xcite propose to leverage speech synthesis systems to generate paired text and speech data for asr training; @xcite , @xcite , and @xcite chain asr and tts together to form a loop for semi-supervised learning with cycle-consistency loss; @xcite , @xcite , and @xcite demonstrate that it is possible to build an end-to-end system translating speech into text or speech in a target language. beyond text-based systems, there is also an emerging research topic that explores the use of units discovered from self-supervised speech representation learning @xcite @xcite to replace text for representing the lexical content in numerous applications, such as language modeling @xcite , speech resynthesis @xcite , image captioning @xcite , and translation @xcite . this line of research bypasses the need for text and makes technologies applicable even to unwritten languages. however, to interpret the output of such systems -a sequence of learned units, a unit-to-speech model is required. this brings up the need of a framework for broader speech synthesis systems that can alternatively take learned units as input. these research directions can benefit from having a single toolkit with different state-of-the-art language technologies. in this paper, we introduce fairseq s 2 , a fairseq @xcite extension for speech synthesis. fairseq is a popular open-source sequence modeling toolkit based on pytorch @xcite that allows researchers and developers to train custom models. it offers great support for training large models on large scale data, and provides a number of state-of-the-art models for language technologies. we extend fairseq to support speech synthesis in this work. in particular, we implement a number of popular text-to-spectrogram models, with interface to both signal processingbased and neural vocoders. multi-speaker variants of those models are also implemented. while speech synthesis often relies on subjective metrics such as mean opinion scores for benchmarking, we implemented a suite of widely used automatic evaluation metrics to facilitate faster iteration on model development. last but not least, we support a number of text and audio preprocessing modules, which allow developers to quickly build a new dataset from less curated in-the-wild data for speech synthesis. the main contribution of this work is threefold. first, we implement a number of state-of-the-art models and provide pre-trained checkpoints and recipes, which can be used by researchers as baselines or as building blocks in applications such as text-to-speech translation. second, we create pre-processing tools that enable developers to use customized data to build a tts model, and demonstrate the effectiveness of these tools empirically. lastly, as part of the fairseq codebase, this speech synthesis extension allows easy integration with numerous state-of-the-art mt, asr, st, lm, and selfsupervised systems already built on fairseq. we provide an example by building a unit-to-speech system that can be used for text-free research. the rest of the paper is organized as follows: section 2 describes the features of fairseq s 2 . experiments are presented in section 3. related work is discussed in section 4, and we conclude this work in section 5. ## features fairseq models fairseq provides a collection of mt @xcite , st @xcite , unsupervised speech pre-training and asr @xcite models that demonstrate state-of-the-art performance on standard benchmarks. they are open-sourced with pretrained checkpoints and can be integrated or extended easily for other tasks. speech synthesis extension fairseq s 2 adds state-of-the-art text-to-spectrogram prediction mod-els, tacotron 2 @xcite and transformer @xcite , which are ar with encoderdecoder model architecture. for the latest advancements on fast non-ar modeling, we provide fast-speech 2 @xcite as an example. all our models support the multi-speaker setting via pre-trained @xcite or jointly trained speaker embeddings @xcite . note that the former enables synthesizing speech for speakers unseen during training. for fastspeech 2, pitch and speed are controllable during inference. for spectrogram-to-waveform conversion (vocoding), fairseq s 2 has a built-in griffin-lim @xcite vocoder for fast model-free generation. it also provides examples for using external model-based vocoders, such as waveglow @xcite and hifi-gan @xcite . ## experiments we evaluate our models in three settings: singlespeaker synthesis, multi-speaker synthesis and multi-speaker synthesis using noisy data. ## conclusion this paper introduces fairseq s 2 , a fairseq extension for speech synthesis. we believe this extension will allow researchers and developers to more easily test novel ideas for language technologies by providing great support for scalability, integrability, and a wealth of tools for curating data as well as automatically evaluating trained systems.