Surfacing Privacy Settings Using Semantic Matching (2020)

Abstract: Online services utilize privacy settings to provide users with control over their data. However, these privacy settings are often hard to locate, causing the user to rely on provider-chosen default values. In this work, we train privacy-settings-centric encoders and leverage them to create an interface that allows users to search for privacy settings using free-form queries. In order to achieve this goal, we create a custom Semantic Similarity dataset, which consists of real user queries covering various privacy settings. We then use this dataset to fine-tune a state of the art encoder. Using this fine-tuned encoder, we perform semantic matching between the user queries and the privacy settings to retrieve the most relevant setting. Finally, we also use the encoder to generate embeddings of privacy settings from the top 100 websites and perform unsupervised clustering to learn about the online privacy settings types. We find that the most common type of privacy settings are ‘Personalization’ and ‘Notifications’, with coverage of 35.8% and 34.4%, respectively, in our dataset.

## introduction online services provide their users with privacy settings to control information access, collection, processing, and sharing. these settings include, but are not limited to, the option to opt-out of data collection, manage notifications, and marketing emails. however, recent work @xcite shows that privacy settings are often hard to locate for an average user. even when the privacy settings pages are easily accessible, they contain numerous settings that are usually distributed over several urls, making it challenging to find the settings of interest. for example, facebook contains 60 privacy settings located on several different urls. further, several service providers expose similar settings to the user. it is desirable for the user to apply the same preference to these settings at once, instead of locating and setting their preferences at each domain. the current privacy settings ecosystem prevents the users from making informed choices for the privacy settings; users unknowingly accept the default options for privacy settings, which tend to favor the interests of the web services over those of the users @xcite . due to their non-standard nature, privacy settings have received little attention in the privacy literature than the more standard opt-out options @xcite @xcite and privacy policies @xcite . there is a need to make privacy settings to be more accessible so that users can exercise more informed choices. achieving this objective requires crawling the websites for privacy control pages, parsing their html to extract the privacy settings, and presenting these settings in a more usable manner. in this paper, we tackle the latter problem of presenting privacy settings to the user by investigating the feasibility of an intra-and inter-domain search interface for privacy settings. in particular, we create a privacy setting-centric encoder and leverage it to answer the following questions: to answer these questions, we first create a custom semantic similarity dataset consisting of real user queries covering various privacy settings. we use this dataset to fine-tune the universal sentence encoder @xcite to match user queries with the settings text. next, we represent a privacy settings page as an abstract data structure consisting of privacy groups. we manually extract the text for these privacy groups for the top 100 websites. we demonstrate the feasibility of developing an accurate intraand inter-domain search interface for privacy settings. finally, we leverage the developed encoder to analyze the landscape of privacy settings across different websites. to summarize, we make the following contributions in this paper: ## related work while the existing literature on understanding the landscape of privacy settings is scarce, there has been some work done in the extraction and automation of privacy settings, particularly in mobile apps' context @xcite @xcite . chen et al. @xcite conducted a large-scale study on the usability of privacy settings for android applications. their methodology leverages the semantic relationship between the text descriptions of ui elements and the titles of application views to discover privacy menus hidden in apps. liu et al. @xcite studied the feasibility of generalized privacy profiles, predicting user permission decisions by modeling it as a classification problem. jialiu et al. @xcite similarly studied privacy profiles using unsupervised clustering of user preferences and app permissions. relatedly, a new privacy application called jumbo @xcite allows users to select one of three available privacy profiles and after that sets the user's privacy settings for their set of supported mobile apps. these works are specific to android and do not provide insights into the type of privacy settings employed by the services. in the web domain, nisal el al. @xcite created a browser extension built on top of a previously proposed automated-extraction methodology @xcite to capture opt-out choices from a website's privacy policy. the extension developed by nisal et al. displays the extracted information to users and helps the users to enforce their opt-out settings. in the privacy domain, prior works have used nlp techniques primarily to understand and analyze privacy policies @xcite @xcite . in particular, harkous et al. @xcite used a hierarchical text classification system to annotate segments from the privacy policy automatically. they also developed a free-form qa system for queries on privacy policies. similarly, oltramari et al. @xcite developed privonto, which analyzes privacy policies based on an ontology that uses a semantic framework to represent annotated policies; nejad et al. @xcite used text mining and rule-based semantic technologies for information extraction from contractual agreements. andow et al. @xcite proposed policylint which leverages sentence level nlp techniques to identify internal contradictions within policy by capturing both positive and negative statements of data collection and sharing. there has also been some research done in using nlp for verification of compliance @xcite of privacy practices by smartphone apps by comparing the permissions to the privacy policy text. all these works are essentially focused on privacy policies whereas, in this work, we fine-tune the state-of-the-art sentence encoders to create privacy-setting-centric embeddings. ## datasets in this section, we describe the two datasets that we created for fine-tuning the universal sentence encoder and evaluating semantic matching of the natural language interpreter present in the browser extension. ## fine tuning the encoder transfer learning has proven to be very effective in several nlp tasks @xcite @xcite . the key idea is to pre-train general embeddings on large text corpora using an unsupervised loss and then use transfer learning for downstream tasks like text classification, semantic matching, sentiment analysis, etc. the intuition behind this is inspired by the fact that humans do not learn everything from scratch, but extend learned knowledge to new domains. there are two standard techniques used for transfer learning in nlp: fine-tuning and feature-based transfer learning. in feature-based transfer learning, the main idea is to find good feature representation to minimize domain divergence and classification er-ror @xcite . in fine-tuning, the pre-trained embeddings are re-trained on downstream tasks (classification, etc.) using labeled data. further, prior work has shown that fine-tuning achieves better performance than feature-based transfer learning @xcite in this paper, our goal is to generate privacysettings-centric embeddings for semantic matching to enable users to search for privacy settings. further, using these embeddings, we can understand the landscape of privacy settings by using unsupervised clustering techniques. to achieve this, we utilize the semantic similarity dataset that we collected (sec. 3.1). in particular, we fine-tune universal sentence encoder (use) @xcite using the sentence similarity task with the dataset described in sec. 3.1. use is trained with a deep averaging network. @xcite and is considered as one of the state of the art sentence encoders @xcite . we further note that we did not perform any pre-processing on the text as use is designed to work with the raw textual data. finally, we set aside 100 query-setting pairs from the dataset for testing and we train on the remaining 496 query-setting pairs. the schematic of the training procedure is shown in fig. 2 . during training, we first pass the query and the setting text through the networks to get the embedding vectors; then we compute the cosine similarity score using the two vectors. finally, this score and the manually annotated score are used to compute the mean squared error (mse) loss. we used adam @xcite optimizer to optimize the loss function and update the weights. we used the test set loss to decide how many iterations the training should go on for. we found that the test set loss was minimized with 10 iterations. ## the browser extension this section describes the browser extension which leverages the fine-tuned encoder to allow the user to search for privacy settings across multiple domains. we, first, start with a brief introduction of our collection method for privacy settings. we, then, describe the browser extension. ## analysis of privacy settings to further understand the types of privacy settings employed by the web services, we take the privacy settings text that we extracted (sec. 3.1), encode them using the fine-tuned encoder described in sec. 4, and group them using unsupervised clustering. we use two types of clustering techniques; k-means @xcite and agglomerative hierarchical clustering @xcite to perform the clustering. we further compare the results from the two techniques and find that agglomerative hierarchical clustering performs significantly better than k-means clustering. ## limitations in this section, we describe the technical limitations of this project. ## conclusion in this paper, we present a sentence similarity dataset for privacy settings consisting of real-world user queries about privacy settings and their corresponding settings from several domains. we further use this dataset to fine-tune universal sentence encoder to generate privacy settings centric sentence encoder. to demonstrate the use case of these embeddings, we develop a browser extension that allows the users to search for their privacy settings for popular websites, including google, facebook, twitter, etc. the extension is backed by a natural language interpreter that takes in the user query and performs semantic matching with the privacy settings text. additionally, we also use the embeddings of privacy settings text to understand the type of privacy settings employed by the website by using unsupervised clustering to find that notification and personalization settings are the most common settings.