Distinguishing Neural Speech Synthesis Models Through Fingerprints in Speech Waveforms (2024)

Abstract: “Recent advancements in neural speech synthesis technologies have brought aboutwidespread applications but have also raised concerns about potential misuse and abuse.Addressing these challenges is crucial, particularly in the realms of forensics and intellec-tual property protection. While previous research on source attribution of synthesizedspeech has its limitations, our study aims to fill these gaps by investigating the identifi-cation of sources in synthesized speech. We focus on analyzing speech synthesis modelfingerprints in generated speech waveforms, emphasizing the roles of the acoustic modeland vocoder. Our research, based on the multi-speaker LibriTTS dataset, reveals twokey insights: (1) both vocoders and acoustic models leave distinct, model-specific fin-gerprints on generated waveforms, and (2) vocoder fingerprints, being more dominant,may obscure those from the acoustic model. These findings underscore the presence ofmodel-specific fingerprints in both components, suggesting their potential significance insource identification applications.”

## introduction recently, neural speech synthesis (also known as text-to-speech, or tts) systems have made substantial advancements in generating highly realistic speech waveforms. these neural tts systems can mainly be divided into two categories. the first category, pipeline systems, generally comprise an acoustic model such as tacotron 2 @xcite , fastspeech 2 @xcite and gradtts @xcite , alongside neural vocoders such as parallel wave-gan @xcite , hifigan @xcite , style melgan @xcite and multiband melgan @xcite . the second category, end-to-end systems, also have been gaining traction in recent years. these systems, such as fastspeech 2s @xcite and vits @xcite , are capable of generating speech waveforms directly from text. while these advancements have found widespread application, they raise concerns about misuse in fraudulent activities, threatening security and privacy. consequently, researchers have aimed to develop robust methods for differentiating genuine human speech from synthetic counterparts @xcite . initiatives like the asvspoof challenges @xcite @xcite and the add challenges @xcite showcase advancements in audio anti-spoofing, including the use of front-end spectral features like linear frequency cepstral coefficients (lfccs) @xcite as well as backend classification models like res2net @xcite in anti-spoofing. there is a also growing interest in identifying the source of synthesized speech, especially in forensics and intellectual property protection. the add 2023 challenge @xcite includes a deepfake algorithm recognition track, addressing the recognition of synthesis tools, where various methods have been proposed @xcite . it is however worth noting that deepfake source attribution is a research topic that has not been extensively explored. pons et al. @xcite focused on upsampling artifacts but limited their investigation to visual representation without exploring exploitability for attribution. similarly, yan et al. @xcite studied vocoder fingerprints but overlooked finer parameter differences within vocoders of the same architecture. this underscores the need for more comprehensive investigations covering various factors to enhance our understanding of deepfake source attribution. notably, the traditional approach to tts usually involves a pipeline system, where an acoustic model generates mel spectrograms from text, which are then converted into speech waveforms by a vocoder; mainstream approaches to source attribution have largely overlooked the potential of these components to leave fingerprints on the synthesized speech by performing attribution on a pipeline-level basis. in the interest of filling this gap and fostering a more robust and nuanced comprehension of deepfake source attribution, we present our study. we aim to extract fingerprints from synthesized speech to identify their sources, focusing on acoustic model and vocoder fingerprints in pipeline tts systems. in particular, we put forth and seek to explain the following questions: (1) do acoustic models and vocoders leave model-specific fingerprints on the speech waveforms they generate? (2) when taking into consideration both the acoustic model and the vocoder, does one overshadow the other in terms of fingerprints? to answer these questions, we train various acoustic models and vocoders on the lib-ritts @xcite dataset to form tts pipelines. through waveform fingerprint extraction from these pipelines, we assess our hypothesis that both the acoustic model and vocoder leave model-specific fingerprints on the generated waveforms. our experiments confirm these fingerprints' existence, highlighting the vocoder's more prominent role, potentially masking acoustic model fingerprints. these findings suggest the potential utility of model-specific fingerprints in source identification applications. in this paper, we present our methodology in section 2, followed by our experimental setup and results in section 3, and then conclude with a discussion of our findings in section 4. ## experiments and results in this section, we outline our conducted experiments and the subsequent findings. we have undertaken a series of ten distinct experiments. the first five experiments are conducted to verify the existence of vocoder (v1-v3) and acoustic model (a1-a2) fingerprints in synthesized speech, and the next two (r1-r2) are conducted to verify the influence of the acoustic model and the vocoder on the fingerprints. finally, three experiments (n1-n3) are conducted to investigate the robustness of the fingerprints against perturbations in the input speech samples. the setups of the experiments are shown in ## conclusion in conclusion, our study centers on extracting fingerprints from acoustic models and vocoders to identify the sources of synthesized speech. our experiments confirm the existence of distinct, model-specific fingerprints for both vocoders and acoustic models. notably, we observe that vocoders tend to leave more prominent fingerprints, potentially overshadowing those of the acoustic models. this discovery holds promise, indicating the feasibility of employing fingerprint extraction for source attribution in synthesized speech. this insight is particularly relevant for researchers and developers in the field of computational linguistics and speech synthesis, as it highlights the importance of carefully selecting or designing vocoders to achieve the desired qualities in synthesized speech. nonetheless, our current research leaves room for further exploration. we have yet to consider the potential impact of different languages, which entail diverse phonological feature distributions, on the fingerprints of generated waveforms. it will also be interesting and important to devise methods for extracting fingerprints from end-to-end tts systems, which are gaining traction in recent years. lastly, even in the context of pipeline tts systems, we have yet to come up with a reliable method for extracting acoustic-model-specific fingerprints despite the masking presence of vocoder fingerprints. these unexplored dimensions c c l 2 0 2 4 will form the focal point of our future investigations, seeking to provide a more comprehensive understanding of the nuances in the fingerprints of synthesized speech.