E mo F ake: An Initial Dataset for Emotion Fake Audio Detection (2024)

Abstract: “To enhance the effectiveness of fake audio detection techniques, researchers have developed mul-tiple datasets such as those for the ASVspoof and ADD challenges. These datasets typically focuson capturing non-emotional characteristics in speech, such as the identity of the speaker and theauthenticity of the content. However, they often overlook changes in the emotional state of theaudio, which is another crucial dimension affecting the authenticity of speech. Therefore, thisstudy reports our progress in developing such an emotion fake audio detection dataset involvingchanging emotion state of the origin audio named EmoFake. The audio samples in EmoFake aregenerated using open-source emotional voice conversion models, intended to simulate potentialemotional tampering scenarios in real-world settings. We conducted a series of benchmark ex-periments on this dataset, and the results show that even advanced fake audio detection modelstrained on the ASVspoof 2019 LA dataset and the ADD 2022 track 3.2 dataset face challengeswith EmoFake. The EmoFake is publicly available1 now.”

## introduction in the past few years, voice conversion (vc) technology has been able to generate very natural converted audio. but it is still not equipped with human-like emotions adequately. emotion, as a vital component of human communication, plays a crucial role in manifesting itself on the semantic and pragmatic levels of spoken language. therefore, the research of emotion voice conversion (evc) has emerged. generally speaking, vc refers to convert the speaker's identity while retaining linguistic information. as a special type of vc, evc converts the emotion state of the speech from the source emotion to the target emotion as shown in figure 1 . currently, the framework commonly used on evc task are: vaw-gan @xcite , sequence-to-sequence (seq2seq) @xcite , cyclegan @xcite @xcite and stargan @xcite . in personalized voice generation, such as singing voice conversion, and intelligent conversational systems like artificial intelligence voice assistants, evc plays an indispensable role. however, if people misuse this technology to attack the security system or interfere with the forensic process in a case, it can pose the immeasurable impact on people's lives and social stability. in a legal context, imagine a crucial piece of evidence is a recorded confession. using evc technology, someone could manipulate the recording by altering the suspect's emotional tone. this could lead to a misinterpretation of the confession, casting doubt on the reliability of the evidence and influencing the case outcome. therefore, enhancing the ability to discriminate emotion fake audio is urgently needed. there has been a lot of very interesting work on the fake audio detection. comprehensive datasets have been developed to improve the performance of the fake audio detection model. based on sas @xcite , the first automatic speaker verification spoofing and countermeasures challenge (asvspoof) @xcite have been organized in 2015. the dataset of asvspoof 2017 @xcite promoted the development of replay attack countermeasures. the dataset of asvspoof 2019 @xcite consists of text to speech (tts), vc, and replay attacks. compared to previous datasets, the dataset of asvspoof 2021 @xcite focus upon deepfake speech in addition. datasets of the first audio deep synthesis detection (add 2022) @xcite and the second audio deep fake detection (add 2023) @xcite add partially fake audio @xcite to further accelerate and foster research on detecting deepfake audio. yi et al. @xcite developed a dataset which consists of the scene fake audio. yi et al. @xcite highlight the key differences across various types of deepfake audio, then outline and analyse competitions, datasets, features, classifications. zang et al. @xcite proposed the singing voice deepfake detection task and presented a database named singfake. @xcite dataset, created using 54 tts models in 23 different languages. these datasets have been developed to further the development of research on fake audio detection. fake audio in previous dataset is mainly generated by changing timbre, linguistic content, channel noise of the original utterance or acoustic scene. however, none of previous datasets consider a scenario where the emotional state of speech is altered. therefore, to enhance the discrimination ability of the fake audio detection model against the emotion fake audio, this paper reports our progress in developing an emotion fake audio detection dataset involving changing emotional state in the speech. we developed a dataset named emofake consisting the emotion fake audio and real emotional speech, which is publicly available 1 . we also provided some benchmark results for detecting the emotion fake audio. the results show that our designed dataset poses a challenge to the fake audio detection model trained with the la dataset of asvspoof 2019. existing mainstream foundational fake audio detection systems experience varying degrees of performance degradation when faced with the emotion fake audio. the rest of this paper is organized as follows. section 2 describes the design policy of the emofake. section 3 presents the experiments and baselines. section 4 concludes this work. ## dataset design emofake is based on the dataset named emotional speech database (esd) @xcite which is designed for multi-speaker and cross-lingual evc studies 1 . esd includs five male english speakers, five female english speakers, five male chinese speakers, and five female chinese speakers. each speaker spoke 350 parallel utterances using five emotions: neutral (neu), happy (hap), angry (ang), sad and surprise (sur), and the average duration of each utterance was 2.9 seconds. each utterance is recorded in a standard room environment at a sampling rate of 16 khz and on a signal-to-noise ratio of 20 db or more. ## conclusion this work describes the design strategy of the emofake dataset, including the generation of fake data and the selection of real data. as far as we know, emofake is the first dataset designed for detecting the emotion fake audio. the emotion fake audio in emofake is generated using seven open source emotion voice conversion models. this work also evaluates the performance of existing fake audio detection models facing emotion fake audio. the results show that the fake audio detection model trained with the la dataset of asvspoof 2019 and the track 3.2 of add 2022 dataset does have a reduced discriminative ability for emotion fake audio. we believe that the emofake dataset will further accelerate research in fake audio detection and inspires broader advancements in the field. future work involves adding more emotion voice conversion models and expanding emofake to include multiple languages. furthermore, we will explore how varying emotional intensities impact fake audio detection while maintaining a consistent emotional category.