F luent S peech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models (2023)

Abstract: Stutter removal is an essential scenario in the field of speech editing. However, when the speech recording contains stutters, the existing text-based speech editing approaches still suffer from: 1) the over-smoothing problem in the edited speech; 2) lack of robustness due to the noise introduced by stutter; 3) to remove the stutters, users are required to determine the edited region manually. To tackle the challenges in stutter removal, we propose FluentSpeech, a stutter-oriented automatic speech editing model. Specifically, 1) we propose a context-aware diffusion model that iteratively refines the modified mel-spectrogram with the guidance of context features; 2) we introduce a stutter predictor module to inject the stutter information into the hidden sequence; 3) we also propose a stutter-oriented automatic speech editing (SASE) dataset that contains spontaneous speech recordings with time-aligned stutter labels to train the automatic stutter localization model. Experimental results on VCTK and LibriTTS datasets demonstrate that our model achieves state-of-the-art performance on speech editing. Further experiments on our SASE dataset show that FluentSpeech can effectively improve the fluency of stuttering speech in terms of objective and subjective metrics. Code and audio samples can be found at https://github.com/Zain-Jiang/Speech-Editing-Toolkit .

## introduction recently, text-based speech editing @xcite @xcite @xcite @xcite has made rapid progress, and stutter removal is a critical sub-task in speech editing. there are various application scenarios for stutter removal, like short-form videos, movies, podcasts, youtube videos, and online lectures, since it provides great convenience for media producers. previous speech editing systems @xcite successfully enable the user to edit the speech recording through operations in the text transcript. some neural text-to-speech (tts) based methods @xcite achieve smooth transition at the boundaries of the edited region. and most recently, the mask prediction based methods @xcite learn better contextual information from the input melspectrogram and outperform previous approaches at speech quality and prosody modeling. however, the existing approaches only aim at modifying reading-style speeches, while removing stutters from spontaneous speeches remains a considerable challenge. when applied to the stutter removal task, previous efforts are still subject to the following limitations: 1) the generated mel-spectrogram is usually blurry and lacks frequency bin-wise details, resulting in unnatural sounds in the boundaries of the modified region; 2) when the speech recording is full of stutters, the edited speech is usually not robust due to the noise introduced by the discrepancy between text and stuttering speech content; 3) the stutter region should be manually determined one by one, which is costly and laborious for media producers. to tackle these challenges, we propose flu-entspeech, the first generative model to solve the stutter removal task, which automatically detects the stutter regions, removes them, and generates fluent speech with natural details. specifically, experiments on the vctk @xcite and @xcite dataset show that fluentspeech outperforms state-of-theart models on speech editing towards reading-style speech with fewer model parameters. and in the experiments on our newly collected sase dataset, fluentspeech enjoys much robustness against stuttering speech and demonstrates the ability to improve the fluency of stuttering speech significantly. the main contributions of this work can be summarized as follows: ## background in this section, we describe the background of speech editing and the basic knowledge of diffusion model. we also review the existing applications of diffusion model in speech tasks and analyze their advantages and disadvantages. ## fluentspeech this section presents our proposed fluentspeech, a stutter-oriented automatic speech editing model that solves the stutter removal task. we firstly overview the motivation and the architecture of fluentspeech. secondly, we describe the detailed designs of alignment modeling, context-aware spectrogram denoiser, and stutter predictor. finally, we describe the training objectives of fluentspeech, following with the illustration of training and inference procedures. ## conclusion in this work, we proposed fluentspeech, a stutteroriented automatic speech editing model for stutter removal. fluentspeech adopts a context-aware spectrogram denoiser to generate high-quality and expressive speeches with rich frequency details. to improve the robustness against stuttering speeches and perform automatic stutter removal, we propose a conditional stutter predictor that localizes the stutter region and injects the stutter embedding into the text hidden sequence to reduce the discrepancy between text and stuttering speech recording. we also contribute a novel stutter-oriented automatic speech editing dataset named sase, which contains spontaneous speech recordings with time-aligned stutter labels. experimental results demonstrate that fluentspeech achieves state-ofthe-art performance on speech editing for readingstyle speeches. moreover, fluentspeech is robust against stuttering speech and demonstrates the ability to improve the fluency of stuttering speech significantly. to the best of our knowledge, flu-entspeech is the first stutter-oriented automatic speech editing model that solves the automatic stutter removal task. our extensive ablation studies demonstrated that each design in fluentspeech is effective. we hope that our work will serve as a basis for future stutter-oriented speech editing studies.