Granular Privacy Control for Geolocation with Vision Language Models (2024)

Abstract: Vision Language Models (VLMs) are rapidly advancing in their capability to answer information-seeking questions. As these models are widely deployed in consumer applications, they could lead to new privacy risks due to emergent abilities to identify people in photos, geolocate images, etc. As we demonstrate, somewhat surprisingly, current open-source and proprietary VLMs are very capable image geolocators, making widespread geolocation with VLMs an immediate privacy risk, rather than merely a theoretical future concern. As a first step to address this challenge, we develop a new benchmark, GPTGeoChat, to test the capability of VLMs to moderate geolocation dialogues with users. We collect a set of 1,000 image geolocation conversations between in-house annotators and GPT-4v, which are annotated with the granularity of location information revealed at each turn. Using this new dataset we evaluate the ability of various VLMs to moderate GPT-4v geolocation conversations by determining when too much location information has been revealed. We find that custom fine-tuned models perform on par with prompted API-based models when identifying leaked location information at the country or city level, however fine-tuning on supervised data appears to be needed to accurately moderate finer granularities, such as the name of a restaurant or building.

## introduction the advent of publicly available large vision language models (vlms), has led to strides in visual question answering @xcite and to the use of these models in consumer-facing applications. 2,3 however, widespread end-user deployment of vision-language models, with their broad 1 code and data are available here 2 inc.com 3 apple.com/newsroom range of emergent capabilities, such as identifying a person in a photo, or geolocating an image, may lead to unanticipated privacy risks. as discussed in §2, and demonstrated in figure 2 , current vlms achieve image geolocation performance that is on par with the current state-of-the-art, making an increase in private information leaks due to geolocation a potential threat in the near future. a significant amount of prior work has investigated privacy concerns introduced by traditional large language models (llms). much of this work has focused on data memorization attacks @xcite by which malicious users extract sensitive text from training corpora. @xcite broke from this line of work and studied how an llm can be used to carry out inference attacks, where implicit personal information is predicted based on public social media messages. the motivation and utility of these inference-based privacy risks naturally apply to vlms, where images are the input, and models draw various conclusions, some of which may pose privacy risks, that might not have been clear to the owner of the image when they originally decided to share it. while there is previous and concurrent work on defenses for jailbreaking vlms @xcite @xcite @xcite , to the best of our knowledge, we are the first to empirically evaluate the ability of vlms to protect sensitive geolocation information that can be inferred from an image. the nature of these inference-time privacy risks is compounded based on the sensitivity of the input image. millions of sensitive photos are uploaded online every day. new privacy threats involving these images may soon manifest as companies start deploying integrated vlms that can interact with social media data. for example, in april 2024, meta unveiled llama-3-based meta ai foot_0 which the image appears to have been taken in the united states. this inference is based on several indicators: 1. the language on the signs and storefronts is english. 2. the currency mentioned on the food truck's menu is in dollars, which is the currency used in the united states. based on the signage present in the image, in which country was this image taken? what does the menu on the food truck reveal about the location of the image? the menu on the food truck lists a phone number with the area code 504. area code 504 is associated with the new orleans region in louisiana, usa. additionally, the presence of items like "charbroiled oysters" on the menu is consistent with the culinary offerings typical of new orleans cuisine. revealed location information: {"country": "united states"} revealed location information: {"country": "united states", "city": "new orleans"} the image appears … after each turn-of-dialogue human annotators also update the location revealed by the gpt-4v @xcite . this benchmark is designed to assess the ability of multimodal moderation agents to offer granular protection of sensitive location information. based on the image and a truncated version of the dialogue, agents flag messages that reveal sensitive location information based on the configuration set by the admin / image owner. the example agent configuration is to the city-level, meaning only the country can be revealed. in this paper, we investigate the ability of vlmbased moderation agents to adhere to these granular privacy controls. to this end, we present gpt-geochat, a benchmark of richly annotated multimodal conversations between a human and gpt-4v towards image geolocation. unlike other multimodal benchmarks @xcite @xcite @xcite which use images from wikipedia, each image in gptgeochat is curated from a stock image provider to resemble a photo that might be posted on social media. we benchmark the efficacy of publicly available vlms as well as fine-tuned moderation agents to moderate conversations in gptgeochat. finally, we also evaluate how well these moderation agents can prevent location leakage when gpt-4v is used in conjunction with search tools for geolocation. ## the need for a conversational geolocation privacy benchmark large vlms are very capable image geolocators. we demonstrate the surprising geolocation capabilities of vlms, specifically gpt-4v, by benchmarking its performance on the commonly used im2gps @xcite image geolocation test set. this task involves predicting the gps coordinates of an image and evaluating the error distance between the predicted and actual coordi- @xcite benchmark compared to the state-ofthe-art geolocation models geodecoder @xcite , geoclip @xcite , and pigeotto @xcite . gpt-4v also has the lowest median distance error of 13 km. the potential widespread use of vlms to geolocate images poses increased privacy risks. with these geolocation capabilities, vlms can amplify existing ai privacy risks @xcite , for example: (1) spear phishing attacks: geolocation with vlms makes it easier for third parties to infer location information from social media, enabling targeted spear phishing attacks (2) doxxing and stalking attacks: vlms allow third parties to determine locations in real time from photos, increasing the risk of doxxing, stalking, and physical intrusion by malicious actors. ## gptgeochat: a conversational image geolocation privacy benchmark to address gaps in existing work ( §2), we present gptgeochat which consists of 1,000 images with detailed location information and 1,000 multiturn interactive multimodal conversations between a human and a model (gpt-4v), with the intent to geolocate each image. annotators also labeled all 4,072 turns of model responses with the geographic information revealed to that point in the dialogue. these annotations are used for evaluation and finetuning in §5. collecting stock images with gold labels. we employ in-house annotators who are paid $18 per hour to curate and annotate data in gptgeochat. the annotators first collected and purchased photos that were taken at a wide range of locations (roughly an even split between inside and outside the u.s.) from the stock image provider shutter-stock. annotators were encouraged to select everyday scenes rather than iconic landmarks. they were also instructed to prioritize images that resemble personal social media photos and those that contain text in signs, posters etc. to test how a vlm might leverage its world knowledge to combine several aschaffenburg, germany irving, united states san jose, costa rica barcelona, spain schloßgasse n/a paseo de los estudiantes el ravel brauereigaststätte schlappeseppel n/a n/a la boqueria (49.97, 9.14) n/a n/a (41.38, 2.17) q5: the lamp in the image says schlappeseppel brau and seit 1631. is this a historic establishment? q4: could you recognize the city by julie johnson, john biggan, etta mullin, and stephen stanley? q4: which city in costa rica has this chinese influence and these shops? q7: which part of town is the market in? annotated ground truth image location information is also provided. google maps is used to determine the longitude and latitude. in total, we obtained 1000 images, all labeled to at least city-level and 91% labeled with exact ground-truth coordinates. 85% of the images in our dataset contain some text -a key difference from existing geolocation datasets which contain more images of scenery, landscapes, buildings, and animal/plant imagery and much fewer with text (only 19% in the im2gps test set and 25% in im2gps3k). 6 geolocating images from these existing benchmarks tends to devolve into an object recognition task that does not require reasoning. if the image contains a recognizable landmark it will be easy to locate, otherwise it is usually very difficult, e.g., a close-up image of a flower. interactive human-ai dialogues. annotators engaged in an interactive dialogue-based geolocation task and annotated the model responses. specifically, for each image in our dataset, an annotator conversed with gpt-4v to reveal details about the location of the specific image by asking questions and receiving model responses. we carefully assigned the images to ensure that an annotator who collected an image, and was exposed to the ground truth location, did not also chat with gpt-4v. annotators were instructed to use varied lines of questioning across dialogues and to always 6 these figures were obtained with easyocr. ## synthetic dialogue generation as mentioned in §3, conversations in gpt-geochat are highly representative of genuine human-ai interaction towards geolocation due to the careful curation of images and the manual querying and annotation by humans. however, to allocate most of the data in gptgeochat to the test set for future benchmarking, we also create gptgeochat synthetic , a cheaper (see ## configurable moderation agents in this section, we outline the task of configurable multimodal moderation and present details regarding the moderation agents and evaluation metrics utilized in the evaluation on gptgeochat in §6. ## evaluation results on gptgeochat this section contains results and insights from evaluating moderation agents ( §5.2) on gptgeochat. ## conclusion in this work, we motivate, introduce, and evaluate granular privacy controls to moderate conversational geolocation. we introduce gptgeochat, a human-vlm dialogue benchmark annotated for revealed location data. our evaluations on gpt-geochat show that moderation agents fine-tuned for granular control perform better than prompted base models, which fail at fine granularities and do not effectively balance privacy with utility. finally, we find that while some agents do prevent serious location leakage when vlm responses are used with search tools, moderation ability improvements may be needed as vlms are deployed.