Automated Detection and Analysis of Data Practices Using A Real-World Corpus (2024)

Abstract: Privacy policies are crucial for informing users about data practices, yet their length and complexity often deter users from reading them. In this paper, we propose an automated approach to identify and visualize data practices within privacy policies at different levels of detail. Leveraging crowd-sourced annotations from the ToS;DR platform, we experiment with various methods to match policy excerpts with predefined data practice descriptions. We further conduct a case study to evaluate our approach on a real-world policy, demonstrating its effectiveness in simplifying complex policies. Experiments show that our approach accurately matches data practice descriptions with policy excerpts, facilitating the presentation of simplified privacy information to users.

## introduction do internet users care about their online privacy? while studies have shown that users care about their privacy online @xcite , they are also willing to give away their personal information @xcite . studies have found that this is often because, privacy policies, which are intended to inform users about what happens with their data online, are notoriously difficult to understand and time-consuming to read @xcite ). an average user needs to spend ∼200 hours to read all the policies that they come across each year @xcite . moreover, privacy policies are written at a college reading level @xcite , use complicated legal jargon and are hard to comprehend @xcite . some studies have attempted to ease the design of privacy policies @xcite to make them more comprehensible to users. @xcite found that presenting privacy information visually in the form of a privacy nutrition label significantly improved the ability of users to find and understand privacy information. while companies such as apple and google have incorporated a privacy label as a way to systematically present users with concise summaries of an app's data practices, they often fail to answer important privacy questions @xcite . moreover, most organizations do not provide access to easy-to-read labels. given that there are several issues related to conducting privacy policy research @xcite , scaling the creation of such labels using nlp approaches is a non-trivial task. we hence propose to automatically present users with information regarding their online privacy using data practice ratings (blocker/good/bad/neutral) and descriptions derived from the terms of service; didn't read (tos;dr) platform @xcite . we make the following contributions: ## related work various nlp methods have been employed to analyze privacy policies, yet none offer a holistic comprehension on data practices. approaches centered on question answering @xcite , summarization @xcite and classification @xcite @xcite @xcite often lack comprehensive coverage of all data practices outlined in a policy @xcite . moreover, these techniques place the onus on users to identify their specific information needs @xcite @xcite . users also face the challenge of investing substantial time in deciphering the nuances of each encountered policy @xcite . our work builds upon the prior research that has leveraged tos;dr data for various nlp tasks. while our primary focus is on document summarization and question answering, previous studies have explored the use of tos;dr data @xcite for tasks that involve risk assessment as an intermediate step. keymanesh et al. (2020) employed elmo embeddings for risk classification on tos;dr data. their findings provide valuable context for the risk-related aspects explored in our work. additionally, binns and matthews (2014) examined the community structure of tos;dr and highlighted the efficient flow of information. previous studies categorized internet users into five categories based on varying levels of privacy concern and online privacy-related behaviors @xcite . given that each user category necessitates a different level of privacy information, there is a clear need for an approach that addresses this diversity. our solution caters to this requirement. ## results the results of privbert trained on datasets created by random sampling (rs) and cluster-based sampling (cbs), for a 1:3 sampling ratio, is shown in for the rs model, we saw that the results on the contrasting set improved with larger negative sampling ratios while the cbs model remained the same. this could indicate that a larger number of contrasting case training samples were necessary for the rs model. on the other hand, the results for both the models on the standalone set remained constant across different sampling ratios. we achieve state-of-the-art results on this dataset, the performance of several other approaches is shown in under real-world conditions, identifying all data practices within a privacy policy would involve testing whether an excerpt matches any case in the full set of tos;dr cases. it is therefore important that the model be trained robustly on dissimilar pairs, supporting training on larger sampling ratios and providing high precision scores for negatives. furthermore, the erroneous classification of a pair as a 'match' can be readily discerned and flagged, 4 4570 thus facilitating the establishment of a feedback loop. conversely, the misclassification of a pair as dissimilar may evade detection, reinforcing our argument for the necessity of robustly performing models on dissimilar pairs. we present a case study evaluating our model's performance on a previously un-annotated policy. we chose airbnb, 3 a popular online marketplace, due to its popularity and its legitimate need to access sensitive personal user information. we split the policy based on new line characters, resulting in 150 segments encompassing paragraphs, section titles, and list items. two authors of the paper thoroughly reviewed each segment and tagged them with any associated tos;dr cases. subsequently, we employed the best-performing cbs model to identify matching case-excerpt pairs. ## conclusion our research introduces a novel solution to the challenge of simplifying complex privacy policies. we provide an automated approach to scale identification of data practices in privacy policies, providing users with information regarding their data privacy. by automating the process of aligning data practices with policy excerpts, we are taking a crucial step towards eliminating the need for users to laboriously parse through lengthy and intricate documents, ensuring that they receive precise and easily digestible information about their online privacy. this lowers the barrier for internet users to understand what is happening with their data online, thereby allowing them to make informed privacy decisions. while there is room for further refinements and research, our model presents a promising foundation for future advancements in the field of usable privacy, ultimately promoting greater user empowerment and security in the digital age.