Cross-Domain Audio Deepfake Detection: Dataset and Analysis (2024)

Abstract: Audio deepfake detection (ADD) is essential for preventing the misuse of synthetic voices that may infringe on personal rights and privacy. Recent zero-shot text-to-speech (TTS) models pose higher risks as they can clone voices with a single utterance. However, the existing ADD datasets are outdated, leading to suboptimal generalization of detection models. In this paper, we construct a new cross-domain ADD dataset comprising over 300 hours of speech data that is generated by five advanced zero-shot TTS models. To simulate real-world scenarios, we employ diverse attack methods and audio prompts from different datasets. Experiments show that, through novel attack-augmented training, the Wav2Vec2-large and Whisper-medium models achieve equal error rates of 4.1% and 6.5% respectively. Additionally, we demonstrate our models’ outstanding few-shot ADD ability by fine-tuning with just one minute of target-domain data. Nonetheless, neural codec compressors greatly affect the detection accuracy, necessitating further research. Our dataset is publicly available (https://github.com/leolya/CD-ADD).

## introduction audio deepfakes, created by text-to-speech (tts) and voice conversion (vc) models, pose severe risks to social stability by spreading misinformation, violating privacy, and undermining trust. for advanced tts models, the subjective score of the synthetic speech can surpass that of the authentic speech @xcite and humans are often unable to recognize deepfake audio @xcite . consequently, it is imperative to develop robust audio deepfake detection (add) models capable of identifying imperceptible anomalies. several datasets built upon various tts and vc models have been released to benchmark the add 1 https://github.com/leolya/cd-add task @xcite @xcite @xcite . however, these datasets mainly include the traditional tts models rather than the emerging zero-shot tts models. moreover, there is a lack of transparency regarding the specific types of models used within these datasets, hindering comprehensive analysis of cross-model performance. additionally, the range of attacks these datasets consider is confined to conventional methods, excluding attacks associated with deep neural networks (dnns), such as noise reduction and neural codec models. based on the aforementioned datasets, a multitude of detection models have been proposed. these models incorporate diverse features, such as the traditional linear frequency cepstral coefficient @xcite and features derived from self-supervised learning @xcite , emotion recognition @xcite , and speaker identification models @xcite . these studies mainly concentrate on a single benchmark dataset. to demonstrate generalization capabilities, several studies have implemented cross-dataset evaluation @xcite . furthermore, to enhance the models' generalizability, researchers have explored the combination of data from various sources @xcite and the integration of multiple features @xcite . in this paper, we present a novel cross-domain add (cd-add) dataset, which encompasses more than 300 hours of speech data generated by five cutting-edge, zero-shot tts models. we test nine different attacks, including those involving dnnbased codecs and noise reduction models. for cross-domain evaluation, rather than adopting the naive cross-dataset scenario, we formulate a unique task for zero-shot tts models by analyzing pairwise cross-model performance and utilizing audio prompts from different domains. experiments reveal: • the cross-domain task is challenging. ## experimental setups the training set for the cd-add dataset was generated using the train-clean-100 subset of lib-ritts @xcite , and the dev-clean and testclean subsets of libritts, along with the test set of tedlium3 @xcite , were utilized for the evaluation datasets. the transcriptions were used as the input text, and the real speech signals were used as the real samples and the speech prompts. for dataset construction, we used the five zero-shot tts models mentioned in section 2.1. we adopted a cer threshold of 10% and a maximum retry limit of five. for cross-model evaluation, the speech from seamless expressive served as the test set. appendix a provides comprehensive details on the tts model checkpoints and the models used for attacks, and appendix b presents the specific statistics of the cd-add dataset that is comprised of over 300 hours of training data and 50 hours of test data. for the add task, we combined our cd-add dataset with the asvspoof2019 @xcite training set and fine-tuned the base model, which includes wav2vec2 @xcite and the whisper encoder @xcite , for four epochs with a learning rate of 3e -5 and a batch size of 128. for attack-augmented training, we increased the number of epochs to eight, as the model converges more slowly due to attacks. the probability of each attack was 10% and only one attack type was used for each utterance. for the evaluation metric, we adopted the widely used equal error rate (eer). ## conclusion in conclusion, our study presents a cd-add dataset, addressing the urgent need for up-to-date resources to combat the evolving risks of zero-shot tts technologies. our dataset, comprising over 300 hours of data from advanced tts models, enhances model generalization and reflects realworld conditions. this paper highlights the risks of attacks and the potential of few-shot learning in add, facilitating future research.