Investigating Prosodic Signatures via Speech Pre-Trained Models for Audio Deepfake Source Attribution (2025)

Abstract: In this work, we investigate various state-of-the-art (SOTA) speech pre-trained models (PTMs) for their capability to capture prosodic sig-natures of the generative sources for audio deepfake source attribution (ADSD). These prosodic characteristics can be considered oneof major signatures for ADSD, which is unique to each source. So better is the PTM at capturing prosodic signs better the ADSD per-formance. We consider various SOTA PTMs that have shown top performance in different prosodic tasks for our experiments on benchmark datasets, ASVSpoof 2019 and CFAD. x-vector (speaker recognition PTM) attains the highest performance in comparison to allthe PTMs considered despite consisting lowest model parameters. This higher performance can be due to its speaker recognition pre-training that enables it for capturing unique prosodic characteristics of the sources in a better way. Further, motivated from tasks suchas audio deepfake detection and speech recognition, where fusion of PTMs representations lead to improved performance, we explorethe same and propose FINDER for effective fusion of such representations. With fusion of Whisper and x-vector representations through FINDER, we achieved the topmost performance in comparison to all the individual PTMs as well as baseline fusion techniques and attaining SOTA performance.

## introduction imagine waking up to find your voice used in a viral audio clip, falsely implicating you in a scandal. this increasingly plausible scenario highlights the growing threat of audio deepfakes. with advancements in text-to-speech (tts) and voice conversion (vc) technologies, malicious actors can now create synthetic audio that is nearly indistinguishable from authentic recordings. from high-profile * equal contribution frauds @xcite to the viral spread of falsified audio targeting political figures @xcite , the misuse of synthetic audio for financial scams and misinformation underscores the urgent need for reliable detection methods. as the authenticity of audio content becomes increasingly difficult to verify, the importance of audio deepfake detection (add) has never been more pressing. despite advancements in add, research has primarily focused on binary classification -distinguishing real from fake audio @xcite @xcite @xcite @xcite @xcite @xcite . this approach, while effective in its simplicity, lacks the granularity needed to address a crucial aspect of deepfake detection: source attribution. audio deepfake source attribution (adsd) goes beyond merely identifying whether audio is real or fake; it seeks to uncover the specific tool or model responsible for generating the synthetic audio @xcite @xcite . this capability is vital for improving the explainability of detection systems and enabling targeted countermeasures, especially in high-stakes contexts such as audio forensics and intellectual property protection. generative sources such as tts, vc, etc. systems embed their unique prosodic characteristics, such as pitch, tone, rhythm, and intonation, into their generated audios, reflecting the inherent design and processing patterns of the generative system. these prosodic signatures are vital for accurately identifying the source and can be considered one of the major fingerprint. in this study, we investigate various state-of-the-art (sota) speech pretrained models (ptms) for capturing these prosodic signatures of source for adsd. we specifically consider ptms that have shown efficacy in various prosodic tasks such as speech emotion recog-nition (ser), depression detection, etc. we evaluate ptms as they have shown efficacy in previous adsd and add works @xcite by providing performance benefit as well as prevention of training models from scratch. further, motivated by tasks like add @xcite , speech recognition @xcite , and emotion recognition @xcite , we also investigate the fusion of different ptm representations and propose, finder (fusion through renyi divergence) for effective fusion. we believe, we are the first work, to the best of our knowledge, for exploring fusion of ptms representations for adsd. to summarize, the main contributions: we will release the codes here foot_0 . ## related work early work on adsd introduced the problem of identifying attacker signatures, showing that representations from rnn can characterize both seen and unseen attackers with high accuracy @xcite . subsequent studies focused on detecting vocoder-specific fingerprints, revealing that vocoders leave identifiable traces in generated audio @xcite . building on this, methods such as t-sne visualization and resnet-based architectures further improved fingerprint detection accuracy @xcite . @xcite proposed vfd-net, a patch-wise supervised contrastive learning method, which achieved robust performance under cross-set and audio compression conditions. more recent work @xcite , ## pre-trained models wav2vec2 @xcite and wavlm @xcite are monolingual ptms. wav2vec2 trained on the librispeech dataset, masks the input in latent space, it has shown effectiveness in prosodic tasks such as ser @xcite . wavlm showed sota performance on superb including various prosodic tasks. xls-r @xcite and whisper @xcite are multilingual ptms. xls-r was pre-trained on 128 languages for 436k hours of unlabeled speech while whisper (radford et al., 2023) on 96 languages for 680k hours of labelled data. xls-r shows good performance in ml-superb @xcite that includes prosodic tasks while whisper shows potential for ser @xcite . in addition to these ptms, we consider, x-vector @xcite , trained for speaker recognition. it excels in various prosodic tasks such as ser @xcite , shout intensity prediction @xcite , depression detection (egas-lópez et al., 2022), and so on. we also consider, wav2vec2-emo foot_2 fine-tuned for ser, as ser is a prosodic task and we think its representations might be helpful for adsd. additional details regarding the above ptms are provided in appendix a.1. ## modeling we consider two downstream networks i.e. fully connected network (fcn) and cnn with individual ptm representations. the fcn model consists of 3 dense layers with 256, 128, and 64 neurons while cnn model has convolutional blocks followed by three dense layers and finally the output layer. for cnn, both the convolution blocks consists of 1d-cnn and max-pooling layer followed by flattening and fcn with similar configuration as fcn network above. hyperparameters detail is given in appendix a.3. ## conclusion in this work, we investigate various sota speech ptms across different prosodic tasks for their ability to capture prosodic signatures of generative sources for adsd. we evaluate monolingual, multilingual, and speaker recognition ptms on benchmark datasets (asv, cfad), finding that x-vector, a speaker recognition ptm, outperforms others due to its ability to capture better source-specific prosody. further, we explored fusion of ptm representations for adsd and propose finder for the same. with fusion of x-vector and whisper representations through finder, we achieve the topmost performance surpassing both individual ptms and baseline fusion techniques and attains sota performance.