Vision Language Model Helps Private Information De-Identification in Vision Data (2025)

Abstract: Visual Language Models (VLMs) have gained significant popularity due to their remarkable ability. While various methods exist to enhance privacy in text-based applications, privacy risks associated with visual inputs remain largely overlooked such as Protected Health Information (PHI) in medical images. To tackle this problem, two key tasks: accurately localizing sensitive text and processing it to ensure privacy protection should be performed. To address this issue, we introduce VisShield (Vision Privacy Shield), an end-to-end framework designed to enhance the privacy awareness of VLMs. Our framework consists of two key components: a specialized instruction-tuning dataset OPTIC (Optical Privacy Text Instruction Collection) and a tailored training methodology. The dataset provides diverse privacy-oriented prompts that guide VLMs to perform targeted Optical Character Recognition (OCR) for precise localization of sensitive text, while the training strategy ensures effective adaptation of VLMs to privacy-preserving tasks. Specifically, our approach ensures that VLMs recognize privacy-sensitive text and output precise bounding boxes for detected entities, allowing for effective masking of sensitive information. Extensive experiments demonstrate that our framework significantly outperforms existing approaches in handling private information, paving the way for privacy-preserving applications in vision-language models.

## introduction vision language models (vlms) @xcite @xcite , which are developed following the impressive success of llms, show a remarkable ability to solve imagerelated tasks. similar to text-only large language models (llms) @xcite ., 1 https://github.com/tiejin98/vlm_ deidentification 2024), which pose potential privacy risks by memorizing and outputting sensitive information from training data @xcite @xcite , vlms also suffer from privacy risks because vlms share the generation part with llms @xcite . to mitigate the privacy risks of text-only llms, several methods are proposed. for example, @xcite utilized knowledge editing to make llms forget the private information. moreover, @xcite proposed privacy restoration to remove the private information in the input and @xcite leveraged an auxiliary llm to remove the sensitive information in the training data. however, most of them focus on the text while neglecting the potentially sensitive information in visual input. for example, medical images often contain protected health information (phi), which is considered sensitive information. we also show an example of phi in fig. 1 . to tackle privacy issues arising from vision data, one promising solution is data deidentification @xcite . deidentification is the process of removing or masking personally identifiable information (pii) from datasets to ensure privacy. however, previous works on image de-identification mainly focus on faces, which aim at obscuring identifiable facial features using generative models @xcite . there is a lack of work focusing on textual private information in vision data. to the best of our knowledge, only presidio (microsoft, 2023) attempts to de-identify such information. however, presidio lacks the flexibility to define what constitutes private information and demonstrates suboptimal performance in our experiments. to address the lack of methods for de-identifying textual private information in vision data, two key tasks are required: accurately localizing sensitive text and processing it to ensure privacy protection. therefore, in this paper, we propose an end-toend framework named visshield (vision privacy shield), which leverages a vision language model to assist in the de-identification of vision data. our framework includes two components: 1) a specialized instruction-tuning dataset op-tic (optical privacy text instruction collection) designed to teach vlms how to handle privacysensitive textual elements. this dataset includes diverse, privacy-oriented instructions that guide vlms to perform ocr-based localization of private text. we generate synthetic image-text pairs with embedded fake private information, covering both natural and medical image scenarios, ensuring robust generalization. our dataset comprises 50m samples, providing a rich training resource for localizing sensitive text. our framework pipeline as shown in fig. 2 enables the vlm to understand customized definitions of private information and extract private information through ocr, which can then be masked to ensure privacy. extensive experiments demonstrate that our visshield achieves superior privacyaware ocr performance and leads to potential new applications of vlms. overall, we summarize our contribution below: ocr instruction definition of privacy few-shot example base image dataset private information text set input privacy definition image set llm generates examples text image (3) composing instruction tuning set append instructions for ocr sample images from a base dataset llm generates private information (1) diverse prompts generation (2) synthetic image dataset generation fine-tuning a pre-trained vlm to assist with de-identification. ## related work vinson language models with the help of llms' powerful reasoning abilities, vision language models (vlms) have achieved significant success in recent days. different models, including llava @xcite , blip2 @xcite , flamingo @xcite , qwen2-vl @xcite , mini-gpt4 @xcite have shown their impressive results among different vision-related tasks, which contains but not limited to visual question answering @xcite @xcite , image captioning @xcite or visual grounding @xcite . among all tasks, document ocr @xcite and its application, which outputs the bounding box for texts in the images and answers the question based on the texts, are the task most similar to ours, where our task is based on the bounding boxes for texts. however, none of the previous works have utilized vlms for de-identification to protect the privacy of vision data. our collected dataset and model not only address this gap but also expand the application scope of vlms. instruction tuning instruction tuning is used to make language models follow natural language instructions and complete more complex tasks @xcite @xcite . instruction tuning improves the zero-and few-shot generalization abilities of llms for both text-only llms, which include chatgpt @xcite , llama family @xcite and flan family @xcite , to vlms @xcite with diverse vision prompts as additional inputs. the quality of instruction tuning is highly dependent on the quality of the tuning dataset @xcite . therefore, previous works like llava @xcite leverage llms to expand the existing image dataset @xcite to various instruction-following datasets. in this work, we use a similar pipeline based on the flickr30k dataset @xcite and medical images @xcite . de-identification de-identification is the process of removing or obfuscating personal information from data to prevent the identification of individuals @xcite . for image deidentification, most current methods aim at face images, where replacing faces in images to protect privacy @xcite @xcite . however, to the best of our knowledge, there is no previous work focused on de-identifying burn-in pixels (texts in the images), especially with the help of vlms. therefore, our model fills the gap and extends the application range of vlms. ## experiments in this section, we provide our experimental results to show the robustness of fine-tuned models. we start with the experimental setting at first. ## conclusion in conclusion, this work presents a novel approach to de-identify textual information in visual data by leveraging the power of vlms. we generate a comprehensive instruction-tuning dataset with diverse images and instruction prompts. by fine-tuning kosmos-2.5 with this comprehensive instructiontuning dataset, we demonstrated that vlms can effectively identify and mask private information. our results show strong generalization and robustness across different datasets and real-world scenarios, laying a foundation for safer integration of vlms into privacy-sensitive applications.