P addle S peech: An Easy-to-Use All-in-One Speech Toolkit (2022)

Abstract: PaddleSpeech is an open-source all-in-one speech toolkit. It aims at facilitating the development and research of speech processing technologies by providing an easy-to-use command-line interface and a simple code structure. This paper describes the design philosophy and core architecture of PaddleSpeech to support several essential speech-to-text and text-to-speech tasks. PaddleSpeech achieves competitive or state-of-the-art performance on various speech datasets and implements the most popular methods. It also provides recipes and pretrained models to quickly reproduce the experimental results in this paper. PaddleSpeech is publicly avaiable at https://github.com/PaddlePaddle/PaddleSpeech .

## introduction speech processing technology enables humans to directly communicate with computers, which is an essential part of enormous applications such as smart home devices @xcite , autonomous driving, and simultaneous translation @xcite . open-source toolkits boost the development of speech processing technology by lowering the barrier of application and research in this area @xcite @xcite @xcite @xcite @xcite . however, the current prevailing speech processing toolkits presume that their users are experienced practitioners or researchers, so beginners might feel baffled when developing their exciting applications. for example, to prototype new speech applications with kaldi @xcite , the users have to be comfortable reading and revising the provided recipes written in bash, perl, and 1 demo video: https://paddlespeech.readthedocs.io/en /latest/demo_video.html python scripts and be proficient at c++ to hack its implementation. the more recent toolkits, such as fairseq s2t @xcite and neurst @xcite , become more flexible by building on general-purpose deep learning libraries. but their complicated code styles also make it timeconsuming to learn and hard to migrate from one to another. so, we have developed paddlespeech, providing a command-line interface and portable functions to make the development of speech-related applications accessible to everyone. notably, the chinese community has many developers eager to contribute to the community. however, nearly all deep learning libraries, such as pytorch @xcite and tensorflow @xcite , target the english community mainly, so it significantly increases the difficulty for chinese developers. paddlepaddle, as the only fully-functioning open-source deep learning platform targeting both the english and chinese community, has accumulated more than 500k commits, 476k models, and is used by 157k enterprises. so, we expect paddlespeech, developed with paddlepaddle can remove the barriers between the english and chinese communities to boost the development of speech technologies and applications. developing speech applications for the industry is not the same scenario as conducting research in academia. the research papers mainly focus on developing novel models to perform better on specific datasets. however, a clean dataset usually does not exist when applying a speech product. so, paddle-speech provides on-the-fly preprocessing for the raw audios to make paddlespeech directly usable in product-oriented applications. notably, some preprocessing methods are exclusive in paddlespeech, such as rule-based chinese text-to-speech frontend, which can significantly benefit the performance of synthesized speech. performance is the cornerstone of all applica1 . our main contributions in this paper are twofolds. ## design of paddlespeech figure 1 shows the software architecture of pad-dlespeech. as an easy-to-use speech processing toolkit, paddlespeech provides many complete recipes to perform various speech-related tasks and demo usage of the command line interface. getting familiar with the top level should be enough for building speech-related applications. the second level faces researchers in speech and language processing. the design philosophy of paddlespeech is model-centric to simplify the learning and development of speech processing methods. for a specific method, all computations of a specific model are included in two files under fundamental platforms common modules models & updaters & datasets recipes paddlepaddle other libraries kaldi, sclite, sox, openblas paddleaudio paddleaudio.features.spectrogram paddleaudio.features.logmelspectrogram utils utils/build_vocab.py utils/parse_options.sh paddlespeech cli/asr/infer.py t2s/exps/fastspeech2/train.py t2s/models/fastspeech2/fastspeech2.py t2s/models/fastspeech2/fastspeech2_updater.py datasets dataset/aishell/aishell.py dataset/librispeech/librispeech.py examples/demos examples/aishell/run.sh demos/metaverse/run.sh figure 1: software architecture of paddlespeech. paddlespeech/<@xmath0. foot_0 paddlespeech has implemented most of the commonly used and well-performing models. a model architecture is implemented in a standalone file named by the method. its corresponding training step and evaluation step are implemented in another updater file. generally, reading or hacking these two files is enough to understand or design a model. more advanced hacking on more fine data processing or more compli-cated training/evaluation loop is also available at paddlespeech/<@xmath1. the original datasets can be obtained by scripts in corresponding dataset/<@xmath2. paddlespeech supports distributed multi-gpu training with good efficiency. the standard modules, such as audio and text feature transformation and utility scripts, are implemented as libraries in the third level . the backend of paddlespeech is mainly paddlepaddle with some functions from third-party libraries as shown in the fourth level . paddlespeech provides multiple ways to extract multiple types of speech features from raw audios using paddleaudio and kaldi, such as spectrogram and filterbanks, which can be varied according to the needs of the tasks. ## experiments in this section, we compare the performance of models in paddlespeech with other popular implementations in five speech-related tasks, including sound classification, speech recognition, punctuation, speech translation, and speech synthesizing. the toolkit can reach sota on most tasks. all experiments in this section include details on data preparation, evaluation metrics, and implementation to enhance reproducibility. 3