Gibberish is All You Need for Membership Inference Detection in Contrastive Language-Audio Pretraining (2025)

Abstract: Audio can disclose PII, particularly when combined with related text data. Therefore, it is essential to develop tools to detect privacy leakage in Contrastive Language-Audio Pretraining(CLAP). Existing MIAs need audio as input, risking exposure of voiceprint and requiring costly shadow models. We first propose PRMID, a membership inference detector based probability ranking given by CLAP, which does not require training shadow models but still requires both audio and text of the individual as input. To address these limitations, we then propose USMID, a textual unimodal speaker-level membership inference detector, querying the target model using only text data. We randomly generate textual gibberish that are clearly not in training dataset. Then we extract feature vectors from these texts using the CLAP model and train a set of anomaly detectors on them. During inference, the feature vector of each test text is input into the anomaly detector to determine if the speaker is in the training set (anomalous) or not (normal). If available, USMID can further enhance detection by integrating real audio of the tested speaker. Extensive experiments on various CLAP model architectures and datasets demonstrate that USMID outperforms baseline methods using only text data.

## introduction microphones in internet of things (iot) @xcite like phones can lead to unintended inferences from audio (shah et @xcite @xcite . vocal features and linguistic content can reveal personally identifiable information (pii) (schwartz and solove, 2011) like biometric identity and socioeconomic status. combining audio with text data increases susceptibility to inference attacks. thus, developing tools to detect privacy leakage in text-audio models like contrastive language-audio pre-training(clap) @xcite @xcite is essential. traditional methods like membership inference attacks (mias) @xcite focus on determining whether a specific data sample was used for model training. research on mias for multimodal contrastive learning (mcl) @xcite like contrastive language-image pretraining(clip) @xcite is extensive @xcite @xcite , but little attention is given to clap. traditional mias train shadow models to simulate target model's behavior @xcite @xcite , which requires high computational costs, particularly for multimodal models like clap. we first propose prmid, which uses the probability ranking provided by clap for membership inference detection, thereby avoiding the computational costs of shadow models. however, current mias for mcl as well as prmid often rely on dual-modal data inputs @xcite , which may lead to new leakage, as one modal of the pair might not have been exposed to the risky target model. therefore, a detector that does not query clap with explicitly matched audio-text pair of speaker (see an example in figure 1 ) is desirable. this concept is known as multimodal data protection @xcite . to address these limitations, we propose us-mid, a textual unimodal speaker-level membership inference @xcite detector for clap models, which queries the target model with only text data. specifically, we introduce a feature extractor that maps text data to feature vectors through clap-guided audio optimization. we then generate sufficient text gibberish that clearly does not match any text description in training dataset. as shown in figure 2 , we observe a distinct separation between the features of gibberish and members in training set. based on this observation, we train multiple anomaly detectors using the feature vectors of generated text gibberish, creating an anomaly detection voting system. during testing, usmid inputs the feature vectors of test text into the voting system to determine if the corresponding speaker is in(anomalous) or out(normal) of the training set. our contributions are summarized as follows: ## threat model consider a clap model m trained on a dataset d train . each sample s @xmath0, x i ) in d train contains the pii of a speaker, consisting of a textual description t i and its corresponding audio x i . for distinct indices i ̸ = j, it is possible for t @xmath1, indicating that multiple non-identical audio samples may exist for the same speaker. detector's goal. the detector aims to probe potential leakage of a speaker's pii through the target clap model m , seeking to determine whether any pii of the speaker were included in the training set d train . for a speaker with textual description t, the detector aims to determine whether there exists a pii sample (t i , x i ) ∈ d train such that t @xmath2. note that our goal is not to detect a specific text-audio pair (t, x), but rather to identify the existence of any pair with textual description t. this is because that multiple audio samples of the same speaker may be used for training, any of which could contribute to potential pii leakage. detector's knowledge and capability. the detector can query m and observe the output, including extracted audio and text embeddings as well as their matching score. for the target textual description t, depending on the application scenarios, the detector may or may not have actual audios corresponding to t. however, if the detector does have the corresponding audio samples, it cannot include them in its queries to m due to privacy concerns. additionally, the detector is unable to modify m or access its internal state. ## defense and covert gibberish generation in real-world scenarios, target models may implement defense mechanisms to detect anomalous inputs like gibberish, potentially leading to misleading outputs that cause usmid to misidentify the inclusion of pii. to address this, we prompted gpt-3.5-turbo to generate fictional character backgrounds rather than mere gibberish as shown in ## conclusion this paper presents the first focused study on membership inference detection in contrastive languageaudio pre-training models. we introduce prmid and usmid, both of which avoid the need for computationally expensive shadow models required in traditional mias. additionally, usmid is the first approach to conduct membership inference without exposing real audio samples to target clap models. evaluations across various clap model architectures and dataset demonstrate the consistent superiority of usmid across baseline methods.