TTS voice corpus reduction for audio-book generation (2020)

Abstract: Nowadays, with emerging new voice corpora, voice corpus reduction in expressive TTS becomes more important. In this study a spitting greedy approach is investigated to remove utterances. In the first step by comparing five objective measures, the TTS global cost has been found as the best available metric for approximation of perceptual quality. The greedy algorithm employs this measure to evaluate the candidates in each step and the synthetic quality resulted by its solution. It turned out that reducing voice corpus size until a certain length (1 hour in our experiment) could not degrade the synthetic quality. By modifying the original greedy algorithm, its computation time is reduced to a reasonable duration. Two perceptual tests have been run to compare this greedy method and the random strategy for voice corpus reduction. They revealed that there is no superiority of using the proposed greedy approach for corpus reduction.

## introduction in the text-to-speech (tts) framework, the main reasons to design a small speech database via a corpus reduction approach are generally the database scalability, the recording or the labeling cost reduction, and the elimination of destructive data @xcite . several studies could be found in the literature dealing with voice corpus optimization. the main purposes of these studies were the pruning of voice corpus in order to respect a parsimony constraint or to extract a more neutral voice, assuming the elimination of expressive voice parts would improve the naturalness of synthetic signals. the idea used by @xcite was to remove the least selected acoustic units to synthesize signals, using a unit-selection tts system, for a domain specific application in order to decrease the voice corpus size. this approach was compared with a greedy strategy based on kullback-leibler divergence (kld). the evaluations indicated better achievements for the adaptive domain pruning method than for the kld based one. some studies have been done in order to use voice data stemming from audio-books to provide input for tts systems. the main goal in this case is to improve naturalness of a neutral voice. an outlier-removal approach has been introduced by @xcite . the outlier, supposed to involve less natural-sounding speech, has been found out as hypo-articulated utterances and low mean f0 utterances in @xcite . in @xcite , it has been noticed that variety in a speech corpus degrades tts quality in general task. they discarded sentences based on acoustic features (extreme f0 patterns, too loud or barely audible sentences) and linguistic features (non-neutral style sentences such as quotation, interjections, utterances which start with lowercase, etc). although tts systems for a general task need neutral voice corpus @xcite , the audio-book generation needs an expressive speech synthesis. found data like available audio-books in the public domain can contain some destructive parts if directly used to build a voice for tts systems. the strategy consisting on selecting a cleaner subset of speech data may result higher synthetic quality and helps tts unit selection engine to speed up. in @xcite , two types of errors, misalignment and annotation errors, which degrade tts quality have been identified to be removed. however the objectives of previous works were mainly the improvement of the naturalness of synthetic signals, expressiveness plays also a crucial role in audio-book generation. the main aim of this study is audio-book generation using a recorded voice of book reading. although synthetic speech has less quality than natural one, the audio-book generation could be less costly using a tts system. thus, the voice corpus design problem considered here is defined as the script selection for recording in order to use the resulting signals as voice to synthesize the rest of the book. this problem has been investigated in previous studies @xcite ) by taking into account the linguistic information. this paper introduces a posterior strategy as a voice corpus reduction : the study is conducted directly from a fully recorded audiobook, instead of its textual content. the voice is a subset of this audio-book and its achievements is assessed by the quality of the synthetic vocalisation of a complementary part of the book. this approach permits to save recording phases and to test several script selections. moreover, voice corpus reduction methods can profit from acoustic and linguistic information. analyses of selected voice sub-corpus achievement and content could be helpful for script design based on only linguistic information before recording process in future works. the original recorded voice is composed of high quality expressive signals which is spoken by a professional speaker. as to offer an adequate expressiveness (from the perspective of later recording phase), the corpus reduction is done at utterance level. since the final product will be an audio-book mixing natural speech utterances (which compose the voice) and synthesized speech signals, and a bigger voice size generally provides a better synthesized speech quality, the goal of this study is to find the best trade-off between the signal quality of the audio-book and the voice size. this paper is organized as follows. first, section 2 describes the proposed greedy algorithm and its heuristics to achieve voice corpus reduction. this algorithm needs to evaluate synthetic signals and utterance candidates to remove without requiring human listeners and section 3 investigates an objective measure for ranking candidates. at last, the evaluation results of the proposed algorithm, in comparison with a random method as baseline, are detailed in section 4. ## framework in order to extract a reduced voice from the original voice corpus well-adapted to synthetically vocalize the content of the unselected sentences, two main requirements are needed : a practical heuristic for selecting a subset of the full corpus and an automatic evaluation method to assess the quality of synthetic signals by using the voice subcorpus. by considering the previous works @xcite @xcite , it has been shown that the greedy algorithm selects portions of a corpus in reasonable time, close to the optimal ones. the greedy algorithm (with spitting or agglomerative policy) is an iterative strategy and needs a score function to rank candidates. in each step of spitting (resp. agglomerative) greedy process, candidates with minimum (resp. maximum) utility are selected to be excluded from (resp. added to) the voice corpus under reduction (resp. construction). the utility score of each utterance represents the increasing gain of the richness in the voice corpus when this utterance is kept. this metric will be presented in section 3. the objective is to extract voice corpus from a fully recorded audio-book to be used by tts. this voice corpus selection should provide the highest synthetic quality for the rest of the book which is called synthetic part. the process, described in algorithm 1, is based on a spitting greedy approach and the initial voice corpus to reduce is the whole audio-book. a similar process has been implemented in @xcite : the authors proposed to agglomeratively select utterances which causes the least synthetic quality degradation for a target set of utterances. in the case of our problem, the target set is the rest of the book and is modified by the reduction process. these units in other utterances. these utterances are locked and should not be removed from voice corpus. at each step, the remaining unlocked utterances in voice corpus compose the candidate set for the next step. by removing utterances from voice corpus, a voice subset with a reduced size will be achieved. the spitting greedy process is continued until the candidate set is empty. for each reduction rate, the rest of the book should be synthesized and evaluated in terms of quality degradation. indeed a small change inside voice corpus could change the synthetic quality of the synthetic part. but since synthesizing the whole synthetic part each time is computationally expensive, the synthetic signal of selected candidates in each step will be used as approximation of the overall quality degradation. the idea behind this proposition is that the small change of voice corpus by removing an utterance could be ignored and only the quality degradation of final audio-book because of replacing recorded voice of the utterance by its synthetic signal would be taken into account. in the ideal scenario, in order to investigate the impact of tts voice corpus reduction on synthetic quality, all combinations of sub-sets should be evaluated perceptually for a given reduction rate. but this is not possible within a reasonable time. in @xcite , the measure for ranking utterances and the evaluation of quality are the same (tts costs). in this way, the quality evaluation process does not need additional computation (algorithm 1 by using the same score in line 10 and 14 follows this idea). this measure will be investigated in section 3. the selection algorithm and the computational problem of the proposed algorithm will be presented in section 4. the optimization process starts with a full audio-book as a initial corpus. in this study, the initial voice corpus contains 3339 utterances of a french expressive audio-book spoken by a male speaker. the overall length of the speech corpus is 10h44. more information on the annotation process can be found in @xcite . the irisa tts system @xcite , which is unit selection based, uses voice subcorpus for synthesising. ## objective measure for selection it is impossible to have all synthetic signals evaluated perceptually by listeners. thus an automatic measure is necessary for quality evaluation of synthetic utterances. this objective measure should be a good approximation of perceptual evaluation. the correlation coefficient or the ranking correlation coefficient could indicate the reliability of an objective measure. the unit selection tts costs is used in previous works @xcite @xcite as the synthetic quality indicator. in this study, the usage of tts global cost, which is a linear combination of concatenation and target cost, in unit selection tts is evaluated as the objective measure of synthetic quality. moreover this objective measure does not need any supplementary computation in the proposed greedy (the result of the line 6 in algorithm 1 can be used directly for line 10). beside tts costs, we propose other objective measures for quality evaluation of synthetic signals. some measures such as pesq @xcite and dynamic time warping (dtw) between two signals need the reference signal. basically they evaluate similarity between a test signal and its reference. three dtw based measures are proposed ; a dtw between mel-frequency cepstral coefficients (mfcc) features of the test signal and its natural pair, a dtw between mel-generalized cepstral coefficients (mgc) features of the test signal and its natural pair, and a dtw between mgc features of test signal which is synthesized signal using voice subcorpus and a reference signal which is synthesized signal using the whole of voice corpus. the third dtw calculates the degradation quality of an synthetic test signal from the highest possible synthetic quality using the tts. ## optimization strategy in this section, a modified greedy process based on algorithm 1 will be compared with a random selection for voice corpus reduction in terms of synthetic quality. ## conclusion the computational time is the main challenge in voice corpus reduction by greedy algorithm. by modifying the original spitting greedy, its complexity has been reduced to a reasonable time. however this approximation level probably costs lower efficiency and makes the solutions closer to the random strategy. in the first step, some objective measures like pesq, dtw between synthetic signal and voice signal, and tts global cost have been investigated. a perceptual listening test (dmos) has been designed to evaluate the synthetic signals using different voice subcorpus sizes. a higher correlation between objective measures and perceptual quality confirmed that tts global cost is the best available metric to estimate perceptual quality. hence the tts global cost has been employed in greedy algorithm for ranking candidates in each reduction step. by modifying the original greedy algorithm, the computational time reduced to a reasonable time. although these modifications cause some level of inefficiency. the proposed greedy has been compared with random strategy for different voice corpus sizes in a mushra test. it has revealed that after a certain size of voice (1 hour of our audio-book), the voice corpus is big enough and the difference of synthetic signals because of voice corpus size can not be distinguished perceptually. moreover it has not been observed any difference between random and proposed greedy. in order to evaluate the performance of the proposed greedy, an ab preference test has been run. the result of this listening test confirmed that listeners did not prefer the signals which are synthesized using voice subcorpus obtained with the proposed greedy.