TUNI : A Textual Unimodal Detector for Identity Inference in CLIP Models (2025)

Abstract: The widespread usage of large-scale multimodal models like CLIP has heightened concerns about the leakage of PII. Existing methods for identity inference in CLIP models require querying the model with full PII, including textual descriptions of the person and corresponding images (e.g., the name and the face photo of the person). However, applying images may risk exposing personal information to target models, as the image might not have been previously encountered by the target model.Additionally, previous MIAs train shadow models to mimic the behaviors of the target model, which incurs high computational costs, especially for large CLIP models. To address these challenges, we propose a textual unimodal detector (TUNI) in CLIP models, a novel technique for identity inference that: 1) only utilizes text data to query the target model; and 2) eliminates the need for training shadow models. Extensive experiments of TUNI across various CLIP model architectures and datasets demonstrate its superior performance over baselines, albeit with only text data.

## introduction recent years have witnessed a rapid development of large-scale multimodal models, such as contrastive language-image pre-training (clip) @xcite . these models synthesize information across different modalities, particularly text and images, facilitating applications from automated image generation to sophisticated visual question answering systems. despite their potential, these models pose significant privacy risks @xcite @xcite @xcite @xcite as the vast datasets used for training often contain personally identifiable information (pii) @xcite @xcite , raising concerns @xcite about pii leakage and misuse @xcite . therefore, it is extremely important to develop tools to detect potential pii leakage from clip models. specially, as the first step, we would like to address the identity inference problem, i.e., to determine if the pii of a particular person was used in training of a target clip model. traditional methods, like membership inference attacks (mias) @xcite , have focused on determining whether a specific data sample was used for model training. when applied to clip models, these approaches typically involve querying the model with both texts and images of the target individual @xcite , and exposing images of a person the clip model may have not seen in the training set brings new privacy leakage risk @xcite . hence, it is desirable to have a detection mechanism for id inference that does not query the clip model with real images of the person (see an example in figure 1 ). furthermore, traditional mias often rely on constructing shadow models that mimic the behaviors of the target model to obtain training data to construct attack models @xcite , which demands extensive computational resources and is less feasible in environments with limited computational capabilities @xcite @xcite . alternative methods for shadow models in mias, such as those based on cosine similarity @xcite and self-influence functions @xcite , exhibit either lower accuracy or still necessitate substantial computational resources @xcite . to address these limitations, we propose a textual unimodel detector (tuni) for identity inference in clip models, which queries the target model with only text information during inference. specifically, we first propose a feature extractor, which maps a textual description to a feature vector through image optimization guided by the clip model; then, we randomly generate a large amount of textual gibberish, which we know do not match any textual descriptions in the training dataset. as shown in figure 2 , we make the key observation that the feature distributions of textual gibberish and member samples in the training set are well distinguishable. leveraging this property, we use the feature vectors of the generated textual gibberish to train multiple anomaly detectors to form an anomaly detection voting system. at test time, tuni simply feeds the feature vector of the test text to the voting system, and determines that if the corresponding pii is included in the training set (abnormal) or not (normal). the training of the anomaly detector in tuni costs only several hours with four nvidia geforce rtx 3090 gpus, avoiding train-ing shadow models with the size of the clip model in traditional mias, which can cost over 18 days even with hundreds of advanced gpus @xcite @xcite . our contributions are summarized as follows: ## evaluations we evaluate the performance of tuni, for the task of id inference from the name of a person, with the corresponding image being the face photo of the person. ## defense and covert gibberish generation in real-world scenarios, target models being detected may deploy defense mechanisms to recognize anomalous inputs like gibberish and provide misleading outputs, causing tuni to misjudge inclusion of pii. to generate more covert gibberish data, we can create strings resembling normal text, with a few characters replaced by syllables from another language. for instance, the detector can craft query texts, by randomly combining english names with syllables from arabic medical terminology. one ## conclusion in this paper, we propose tuni, the first method to conduct identity inference without exposing acutal images to target clip models. tuni turns inference problem into anomaly detection, through randomly generating textual gibberish that are known to be out of training set, and exploting them to train anomaly detectors. furthermore, the incorporation of real images is shown to enhance detection performance. through evaluations across various clip model architectures and datasets, we demonstrate the consistent superiority of tuni over baselines.