C 3 PA : An Open Dataset of Expert-Annotated and Regulation-Aware Privacy Policies to Enable Scalable Regulatory Compliance Audits (2024)

Abstract: The development of tools and techniques to analyze and extract organizations’ data habits from privacy policies are critical for scalable regulatory compliance audits. Unfortunately, these tools are becoming increasingly limited in their ability to identify compliance issues and fixes. After all, most were developed using regulation-agnostic datasets of annotated privacy policies obtained from a time before the introduction of landmark privacy regulations such as EU’s GDPR and California’s CCPA. In this paper, we describe the first open regulation-aware dataset of expert-annotated privacy policies, C3PA (CCPA Privacy Policy Provision Annotations), aimed to address this challenge. C3PA contains over 48K expert-labeled privacy policy text segments associated with responses to CCPA-specific disclosure mandates from 411 unique organizations. We demonstrate that the C3PA dataset is uniquely suited for aiding automated audits of compliance with CCPA-related disclosure mandates.

## introduction privacy policies are a crucial mechanism for organizations to communicate their data habits with external entities, including consumers and regulators. unfortunately, they have been known to fall short of meaningfully achieving these goals due to their inaccessibility or incomprehensibility @xcite . in response, privacy advocates and researchers have focused on developing natural language processing tools to make privacy policies more usable for consumers and regulators @xcite . unfortunately, the usefulness of tools to automatically process privacy policies are hampered by the dynamicity of the digital privacy landscape. new data regulations and technologies provoke (or mandate) significant changes in the language and types of disclosures within privacy policies. these changes can limit the usefulness of tools which are developed using datasets that are unaware of these concepts. the california consumer privacy act ("ccpa," n.d.) is a hallmark privacy legislation in the united states which caused significant changes in organizations' privacy policies. introduced in 2018 and amended in 2020, it provides comprehensive privacy protections for the data generated by californians. the law offers californians several rights which facilitate more control over the data that they generate. in addition, the ccpa mandates that organizations which are subject to the ccpa make 12 specific disclosures in the privacy policies. importantly, the ccpa serves as the base framework for comprehensive privacy regulations recently introduced in other us states (colorado, connecticut, delaware, indiana, iowa, montana, new jersey, oregon, utah, and virginia). consequently, the privacy regulations in these states share significant similarities with the ccpa. in fact, the 12 disclosures mandated by the ccpa are a superset of the disclosures mandated by each of them. therefore, the ability to automatically recognize or verify the presence of ccpa-specific disclosures will serve auditors of many state regulations. unfortunately, as we will show in this paper, existing tools are largely incapable of recognizing or verifying the presence of ccpamandated disclosures in privacy policies. this is because of the absence of ccpa-specific annotations during their training and fine-tuning. this paper aims to address this gap by developing an expert-annotated dataset of ccpa-specific mandated disclosures extracted from privacy policies. to develop this dataset, we first devise a process to identify organizations that are highly likely to be subject to the ccpa and extract their privacy policies. next, we employ a team of native-english speaking legal professionals to extract segments of text that are responsive to each of the 12 ccpa-mandated disclosures, from our dataset of privacy policies. these expert-annotations form the publicly available ccpa privacy policy provision annotations (c3pa foot_0 ) dataset. c3pa consists of a total of 48,947 annotations extracted from 411 organizations' privacy policies. finally, we demonstrate that tools built with c3pa data will be better suited for automatically assessing regulatory compliance with the ccpa and other similar regulations. ## related work datasets. @xcite published a dataset of 1000 privacy policies and applied hidden markov models (hmm) on them to essentially create unlabeled clusters of text segments. @xcite created opp-115, a dataset of 115 privacy policies annotated using a comprehensive taxonomy of privacy concepts. the opp-115 dataset was the first expert-annotated dataset of data practices from privacy policies and continues to be the most frequently used dataset for training models to examine privacy policies. @xcite captured general data practices of 350 mobile app privacy policies in his dataset. @xcite mapped opp-115's taxonomy to the ccpa and created a dataset of 64 privacy policies annotated with ccpa related data practices. unfortunately, none of these datasets is specifically created using a ccpa-driven or regulation-aware taxonomy, including @xcite . instead, they were created to identify generic data habits and operations of organizations. consequently, they are lacking direct connections to regulatory mandates and in fact do not contain data related to many ccpa mandates. in contrast, c3pa is the first and only fully regulation-aware expert-annotated dataset. besides these datasets of segments from privacy policies, other work @xcite has focused on developing tools to crawl, scrape, and parse the web to create large datasets of privacy policies. our method for curating privacy policies borrows from these prior approaches. privacy policy analysis. @xcite used a naïve bayes classifier to create a web extension that summarizes data practices in privacy policies. @xcite used their opp-115 dataset, to train a machine learning model to predict data practices in segments of privacy policies. as a follow up, many works on privacy policy analysis leveraged opp-115 to develop related tools. @xcite used opp-115 to train ml models to analyze general data practices in mobile app privacy policies of 9k apps. @xcite and @xcite trained ml models to detect opt-out clauses and consent provisions in privacy policies. since these tools are powered by regulation-unaware datasets, they are unable assess privacy policies compliance with the ccpa and other related regulations. the c3pa dataset will introduce these capabilities to existing and upcoming tools. ## annotation scheme to analyze privacy policies that are responsive to the ccpa, it is essential to understand ccpa disclosure mandates. the last update to their privacy policy. the categories labels (l2-l4) are responsive to sections 1798.130(a)(5)(b) and (c) which require firms to disclose types of pii collected, how they are used, and with whom they are shared and sold. the descriptions labels (l5-l11) are responsive to section 1798.130(a)(5)(a) which requires businesses to disclose the rights granted to consumers. finally, the methods label (l12) is responsive to section 1798.130(a)(1)(a) which requires businesses to describe how a consumer may exercise their ccpa-granted rights. it is to be noted that l6 and l10 were amendments to the ccpa made effective in 2023. ## dataset creation in this section we provide an overview of how we created the c3pa dataset. we outline our sourcing for privacy policies and our annotation process. ## dataset evaluation in this section we evaluate the quality and relevance of our annotations. ## utility of the c3pa dataset we now perform a comparative analysis between a classification model trained on opp-115 (the most popular annotated privacy policy dataset) with a model trained on our dataset for the task of extracting ccpa-specific segments. we will show that the same model, trained on a ccpa-specific dataset (such as c3pa) will outperform one trained on a dataset which maps generic data practices to the ccpa (such as the mapping of opp-115 to the ccpa produced by @xcite ). ## concluding remarks limitations. our analysis in this work has established the importance of utilizing regulationaware datasets for assessing compliance (as opposed to general-purpose datasets with post-hoc mapping to regulations). however, it is unclear how representative this dataset is of privacy policies in general, or how well it will perform for general privacy policy analysis, as we focused on gathering ccpa specific policies. finally, a systematic analysis of sources, model selection and document segmentation for building an automation tool for extracting ccpa related text from privacy policies is out of the scope of this work. however, such future work can increase the privacy policies that can be analyzed efficiently and further our understanding of ccpa rights and requirements.