Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems (2024)

Abstract: In recent years, there has been increased demand for speech-to-speech translation (S2ST) systems in industry settings. Although successfully commercialized, cloning-based S2ST systems expose their distributors to liabilities when misused by individuals and can infringe on personality rights when exploited by media organizations. This work proposes a regulated S2ST framework called Preset-Voice Matching (PVM). PVM removes cross-lingual voice cloning in S2ST by first matching the input voice to a similar prior consenting speaker voice in the target-language. With this separation, PVM avoids cloning the input speaker, ensuring PVM systems comply with regulations and reduce risk of misuse. Our results demonstrate PVM can significantly improve S2ST system run-time in multi-speaker settings and the naturalness of S2ST synthesized speech. To our knowledge, PVM is the first explicitly regulated S2ST framework leveraging similarly-matched preset-voices for dynamic S2ST tasks.

## introduction progress in deep learning and voice cloning technology has enhanced public access to robust aidriven voice cloning systems. these systems can help solve complicated speech-to-speech translation (s2st) tasks like automated video dubbing (auto-dubbing) by generating audio deepfakes @xcite @xcite . cloning systems are desirable for dynamic speech tasks because they can generate a clone from an input voice given an audio sample as short as a few seconds @xcite . currently, voice cloning technology is highly unregulated and can be harmful if misused or commercialized irresponsibly @xcite . as voice cloning systems can clone an arbitrary voice and do not require permission, they raise several privacy concerns @xcite . risks related to voice cloning technology include lack of informed consent, biometric privacy, and the spread of misinformation through deepfakes @xcite . robust regulations are necessary to mitigate these risks, protect individual rights, and prevent misuse @xcite @xcite . the risks of unregulated voice cloning technologies are compounded by a high demand for voice cloning-based products. pressure to capitalize on a newly budding market of cloning-based products can lead businesses to emphasize speed over careful and tested development. since voice cloning technology is so new, regulatory measures are required and in the process of being implemented, but not yet fully in place. given these challenges, it is crucial to integrate privacy regulations into aipowered voice cloning systems @xcite @xcite . to address the need for regulated voice cloning technology, we propose preset-voice matching (pvm), a regulated s2st framework. pvm bakes regulatory precautions into the s2st process by removing the explicit training objective of cloning an unknown input speaker's voice, and instead cloning a similar preset-voice of a consenting speaker. pvm can be easily installed on top of existing cascaded s2st pipelines, improving regulatory compliance. we find this process also decreases system run-time in multi-speaker auto-dubbing scenarios and improves speaker naturalness relative to stateof-the-art voice cloning systems when translating across our tested languages. the intention of this paper is to put forward a regulated pvm s2st framework that is robust against legislative changes and future liability concerns. we demonstrate pvm is desirable for s2st over current benchmark voice cloning frameworks due to its inherent safety, lower run-time in multispeaker scenarios, and enhanced speaker natural-ness. we show this by providing and testing a pvm algorithm which we call gemo-match. we hope this work inspires others to develop and tune the framework for different high-performance environments. our main contributions are as follows: the rest of this paper is organized as follows. background information is provided in section 2. our pvm framework and gemo-match algorithm are detailed in sections 3 and 4. relevant datasets are described in section 5. section 6 explains our experimental setup as well as the techniques, algorithms, and parameters used in the study. section 7 includes experimental results and analysis. we discuss potential future work towards pvm and conclude the paper in sections 8 and 9. we address pvm limitations in section 10. ## background information speech-to-speech translation (s2st) is typically achieved by direct translation or cascaded approaches @xcite . direct translation approaches use speech and linguistic encoder/decoders @xcite to directly translate speech signals from one language to another. cascading architectures split s2st into three sub-tasks, using separate but connected speech-to-text (stt), text-to-text (ttt), and text-to-speech (tts) modules @xcite . cascading architectures have been the traditional method for s2st. two common approaches for synthesizing speech from text are concatenative and parametric tts. concatenative tts combines pre-recorded clips from a database to form a final speech output @xcite . parametric tts attempts to model and predict speech variations given text and a reference voice @xcite . parametric deep learning methods have shown ubiquitous success spanning various industries from computer vision to text synthesis @xcite @xcite . as deep neural network (dnn) based tts methods can lead to natural and expressive synthesized voices, they are desirable for many speech tasks @xcite . wavenet is a benchmark dnn-based tts model @xcite . since its creation, there have been many advancements in sequenceto-sequence tts models trained to produce humanlike speech @xcite . wavenet performs speech synthesis by training on a set of human voices, conditioning on their unique speaker id to generate natural-sounding utterances in the voice of a selected speaker @xcite . recently, there have been models which aim to extend this behavior by cloning voices unseen in training, resulting in zero-shot voice cloning @xcite . cross-lingual voice cloning is difficult due to complexities in discriminating between languagespecific and speaker-specific features within a singular waveform, and mapping these features across different languages @xcite . additionally, training robust multilingual speech generation models requires vast amounts of processed language and speech data in multiple languages with a variety of utterances and speakers. the performance of these models depends on the data they are trained on @xcite . preset-voice tts methods generate speech from stored options of preset or pre-recorded voices. preset-voice methods are typically used in static or repetitive systems which do not require dynamic adaptive functionality. examples include pre-programmed transit operator dispatch messages, medical alert systems in healthcare, and emergency flight announcements @xcite @xcite . due to the static nature of current preset-voice methods, they have not previously been used for dynamic s2st tasks like auto-dubbing. such dynamic tasks require modelling different speakers across languages based on incoming media data @xcite . in addition to providing a regulated pvm framework, this work aims to extend the application of preset-voice tts methods to more dynamic settings. ## preset-voice matching framework this section explains our privacy regulated preset-voice matching (pvm) framework. pvm bakes privacy regulations into the s2st process by cloning a similar and prior consenting preset-voice, instead of the voice originally input to the s2st system. the pvm framework connects to cascading s2st architectures, performing additional computations alongside the stt, ttt, and tts modules. the pvm framework consists of 3 sub-modules. module 1, the similarity feature extraction module, extracts features from the inputted voice. it then uses the extracted features to match the input voice to the most similar preset-voice from the preset-voice library. module 2, the preset-voice library, contains a collection of consented targetlanguage preset-voices, partitioned by discrete feature codes depending on the pvm implementation. module 3, the tts module, generates tts in the target-language using the matched preset-voice from the preset-voice library. we describe these 3 modules below in detail. ## gemo-match algorithm in this section we describe gemo-match, an example pvm framework implementation. following a similar notion to (singh and prasad, 2023), gemo-match employs a hierarchical gender-dependent emotion classifier architecture trained with a gender-dependent training method. the process of splitting gender and emotion in emotion classification simplifies the emotion classification problem. as gemo-match is a pvm framework, it contains the 3 pvm modules: the similarity feature extraction module, the preset-voice library, and the tts module. these modules and their process are described below. ## dataset descriptions in this section, we describe the datasets used to test our framework. we experimented with two speech-emotion datasets: the ryerson audio-visual database of emotional speech and song (ravdess) (livingstone and russo, 2018), and the combined gender-dependent dataset (cgdd), which we curated by combining four benchmark speech datasets. to ensure compatibility with our gender-emotion based gemo-match algorithm, we split the ravdess dataset by gender and relabeled it with genderemotion pairs. further details on ravdess and cgdd are outlined in 5.1 and 5.2. ## experimental setup this section details the setup of each experiment, which show additional strengths of the pvm framework, beyond its inherent regulatory benefits. we demonstrate the effectiveness of pvm for s2st with gemo-match in terms of robustness, multilingual capability, and run-time. our experiments were run on a single tesla t4 gpu with 40 cores. we discuss each experiment in detail below. ## experimental results and analysis in this section, we discuss and analyze our experimental results. section 7.1 describes the results of the gemo-match robustness experiment, contained in tables 1 and 2. next, section 7.2 provides an analysis on the results in ## future work pvm is a general framework for regulated s2st that can be integrated into pre-existing cascaded s2st pipelines. the performance of pvm is directly dependent on the quality of the individual swappable components of the pipeline. consequently, the efficacy of any pvm implementation is expected to increase with general advancements in tts technology. there are many ways to improve the pvm framework, and we propose some ideas for future work. for future work, we propose a cascaded voice cloning tts system which uses an initial vocal encoder with learned weights to extract and compress relevant features from the input voice. the system would perform the classical cloning tasks on this encoded voice in a downstream, decoupled tts model. this would allow voices to be stored in the preset-voice library in their encoded formats rather than speech signals, likely decreasing run-time complexity. using a cascaded learning process, the tts module would learn to effectively clone and only synthesize voices encoded by the vocal encoder. during distribution of the system, the vocal encoder would not be published. in this way, the system could not be used to clone a voice outside of the pre-encoded preset-voices in the preset-voice library. gemo-match uses classifiers which depend on labeled data. this dependency motivates the development of alternative pvm instances capable of voice-matching without relying on labeled data. we posit that learned encodings can be used, akin to self-supervised learning mechanisms employed by transformer architectures, to extract robust internal representations of speech inputs @xcite . this would require a new training pipeline with an objective function for maximizing speaker similarity between the input voice and the matched voice. the resulting pvm system could use latent feature representations to perform voice matching, and training would not require labeled speech datasets. ## conclusion we proposed preset-voice matching, a novel framework that bakes regulatory precautions into the s2st process. pvm achieves this by removing the explicit objective of cloning an unknown input speaker's voice, and instead cloning a similar preset-voice of a consenting speaker. this paradigm is extensible to a variety of industry settings to regulate the behavior of s2st systems. quantitative experiments show pvm is a desirable framework compared to the tested benchmarks in terms of run-time and naturalness of multilingual translation output. additionally, we provided cgdd, a gender-dependent speech-emotion dataset. we then showed cgdd leads to better model generalization and robustness in terms of accuracy and precision compared to the benchmark ravdess dataset. we hope this work inspires others to create more privacy regulated s2st systems using the pvm framework.