Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfake (2024)

Abstract: In this work, we investigate multilingual speech Pre-Trained models (PTMs) for Audio deepfake detection (ADD). We hypothesize thatmultilingual PTMs trained on large-scale diverse multilingual data gain knowledge about diverse pitches, accents, and tones, during theirpre-training phase and making them more robust to variations. As a result, they will be more effective for detecting audio deepfakes. To validate our hypothesis, we extract representations from state-of-the-art (SOTA) PTMs including monolingual, multilingual as well as PTMs trained for speaker and emotion recognition, and evaluated them on ASVSpoof 2019 (ASV), In-the-Wild (ITW), and DECRO benchmark databases. We show that representations from multilingual PTMs, with simple downstream networks, attain the best performance for ADD compared to other PTM representations, which validates our hypothesis. We also explore the possibility of fusion of selected PTM representations for further improvements in ADD, and we propose a framework, MiO (Merge into One) for this purpose. With MiO, we achieve SOTA performance on ASV and ITW and comparable performance on DECRO with current SOTA works.

## introduction the popularity of audio deepfakes has raised multiple concerns in areas dealing with personal and public security due to its capability to impersonate and share false, often malicious information. scammers, for example, have utilized audio deepfake to mimic a german executive, successfully convincing a transfer of 220,000 euros to a hungarian supplier @xcite . thus, checking and evaluating authenticity of any audio content is important through robust and reliable measures. motivated by this, in this work, we focus on audio deepfake detection (add). to combat this progressing issue, various detection methods have been proposed @xcite @xcite . these works leveraged statistical attributes or the raw audio as input to the models. however, with the wide-scale accessibility of pre-trained models (ptms), add as a task has undergone exponential advancement. representations from ptms are used as input features to downstream add and it comes with a series of benefits, which include higher accuracy in add and saving time as well as resources, in building add systems from the ground up. ptms come in various architectures as well as varied pre-training schemes and are trained on large-scale datasets. they can be either trained in a supervised (eg. whisper @xcite ) or in a self-supervised manner (eg. wav2vec2 @xcite ) and also on either single or multiple languages. despite these ptms being pre-trained on only real speech data, representations from these ptms have shown exceptional performance in identifying real content from their fake counterparts @xcite . our work relies on the hypothesis that multilingual ptms trained on large-scale diverse multilingual data, acquire knowledge about diverse pitches, accents, tones, and are more robust to variations, hence will be more effective for identifying audio deepfakes than other ptms. so to validate our hypothesis, we extract representations from eight state-of-the-art (sota) ptms including multilingual (xls-r, whisper, mms), monolingual (unispeech-sat, wavlm, wav2vec2), speaker recognition (x-vector), and emotion recognition (xlsr_emo) and evaluate them with two simple probing networks (fully connected network (fcn), convolution neural network (cnn)) on three benchmark datasets asvspoof 2019 (asv), in-the-wild (itw), and decro. we also investigate by combining representations from different ptms as it has been seen in other speech processing tasks such as speech recognition @xcite that certain representations act as complementary to each other and we propose a framework, merge into one (mio) for the same. to the best of our knowledge, this is the first study, to explore fusion of ptm representations for add. our study makes the following contributions: ## related works in this section, we give an overview of various prolific add methods proposed. add as a task has caught the attention with the release of the asvspoof 2015 @xcite database. initially, researchers built gmm and svm-based modeling approaches with statistical audio features as input @xcite . previous works have also harnessed neural network-based models such as cnn, rnn, etc. for add @xcite @xcite . researchers have exploited self-supervised learning (ssl) modeling approaches for add @xcite @xcite . further, different types of ptms such as wav2vec, hubert, tera, mockingjay, etc also been explored for add @xcite . @xcite showed that generalization of add systems increases with combination of wav2vec, prosodic, and pronounciation information as input features. in this work, we evaluate eight ptms to validate our hypothesis that multilingual ptms trained on extensive and diverse multilingual datasets allow them to capture knowledge related to diverse pitch, accent, tone, and so on. this broad exposure enhances their robustness to different variations in audio signals. as a result, the representations learned by these ptms are particularly effective for discerning audio deepfakes when compared to representations from other ptms. ## pre-trained models we compile the top-performing ptms for our experiments. for multilingual ptms we choose, xls-r @xcite , whisper @xcite , and massively multilingual speech (mms) @xcite . xls-r was pretrained on 128 languages while whisper on 96 and mms over 1400 languages. whisper improves over xls-r in various downstream speech processing tasks while mms improves over whisper. we selected the monolingual ptms (wavlm, unispeech-sat, wav2vec2) based on the su-perb @xcite . wavlm and unispeech-sat have shown sota performance on superb so we choose them. wav2vec2 @xcite has not shown top performance like wavlm @xcite and unispeech-sat @xcite on superb, however, as previous works have shown its efficacy for add @xcite , so we selected it. additionally, models pre-trained for more specific tasks such as speaker recognition ptm @xcite and models trained for emotion recognition @xcite show exceptional performance for add, so we included them in our experiments. as speaker recognition ptm, we consider, x-vector @xcite and as emotion recognition ptm we use, xlsr_emo @xcite . additional details regarding these selected ptms are available in appendix 9.2. ## modeling as we are evaluating how representations of different ptms will behave for add, we keep the ptm layers frozen and keep the downstream modeling as simple as possible. we experimented with two modeling approaches (see figure 1a and 1b ). for the first approach (fcn), we employ an fcn on the extracted ptm representations and for the second (cnn), we use a 1d-cnn layer on top of representations followed by a maxpooling layer and fcn. softmax is used as the activation function in the output layer of the models which gives output as probabilities. merge into one: for fusing representations of ptm representations maxpooling 1d cnn ptm maxpooling 1d cnn bilinear pooling input fcn classifier output representations figure 2: proposed modeling framework for fusion of ptm representations, mio different ptms, we propose mio. the model architecture is shown in figure 2 . here, we follow the same modeling pattern for each incoming representation as the second approach mentioned above. then we apply linear projection to a dimension of size 120 followed by bilinear pooling (bp), which allows effective interaction between the features as shown by @xcite . bp is the outer product of two vectors p and q of dimension (d,1) and (d,1) such that the resultant will be a matrix of dimension (d, d) and it is given as: we keep the number of epochs at 20 and the batch size of 32 for all the modeling approaches. we use cross-entropy as the loss function and adam as the optimizer. we use tensorflow library for our experiments. for reproducing our experiments, we will make our codebase available here foot_0 . ## conclusion in this work, we validated our hypothesis that multilingual ptms pre-trained on large diverse multilingual data will be more effective for add as they learn diverse pitches, accents, and tones during their pretraining phase and are more robust to variations. we carried out a comprehensive empirical analysis by extracting representations from eight ptms and our findings show that representations from multilingual showed the lowest eer on three benchmark datasets asv, itw, and decro. also, we found that fusion of representations from ptms lead to a further drop in eer and for this, we proposed, mio. we report sota performance in asv, itw and competitive performance in de-cro in comparison to previous sota works with mio.