A Fine-grained C hinese Software Privacy Policy Dataset for Sequence Labeling and Regulation Compliant Identification (2022)

Abstract: Privacy protection raises great attention on both legal levels and user awareness. To protect user privacy, countries enact laws and regulations requiring software privacy policies to regulate their behavior. However, privacy policies are written in professional languages with many legal terms and software jargon that prevent users from understanding and even reading them. It is necessary and urgent to use NLP techniques to analyze privacy policies. However, existing datasets ignore law requirements and are limited to English. In this paper, we construct the first Chinese privacy policy dataset, namely CA4P-483, to facilitate the sequence labeling tasks and regulation compliance identification between privacy policies and software. Our dataset includes 483 Chinese Android application privacy policies, over 11K sentences, and 52K fine-grained annotations. We evaluate families of robust and representative baseline models on our dataset. Based on baseline performance, we provide findings and potential research directions on our dataset. Finally, we investigate the potential applications of CA4P-483 combing regulation requirements and program analysis.

## introduction a privacy policy is a legal document written in natural language that discloses how and why a controller collects, shares, uses, and stores user data @xcite @xcite . privacy policies help users understand whether their privacy will be abused and decide whether to use the product. however, privacy policies are tedious, making it hard for users to read and understand them @xcite . natural language processing techniques achieve big success on understanding document semantics @xcite @xcite . thus, it is neces1 our dataset and code are publicly available in https:// github.com/zacharykzhao/ca4p-483 sary to apply natural language processing to analyze the privacy policies @xcite @xcite and help users be aware of apps' privacy access behavior @xcite . chinese software privacy policy processing (csp 3 ) task is a sequence labeling problem that recognizes privacy-related components in the sentences. csp 3 has two main unique features. first, privacy policies contain an amount of information inside @xcite , such as how the app stores user data, and how to contact app developer. in our dataset, we concentrate on data access-related sentences as the sentences directly related to user privacy. second, privacy policies are written in a legally binding professional language and contain software jargon. thus, it requires strong background @xcite to understand the statements inside. both characteristics prevent users from understanding the privacy policies. a well-annotated dataset can facilitate building automatic privacy policy analysis tool and further help users protect their privacy. although privacy policy datasets have been proposed recently @xcite , labels in existing datasets are coarse-grained (i.e., sentence-level annotations @xcite ) and limited to few privacy practices @xcite . besides, existing datasets only include english privacy policies, which limits the application of these datasets in regions with other languages. we construct a fine-grained chinese dataset for software privacy policy analysis. in this work, we focus on android application privacy policies as android possesses the largest share of mobile operating systems @xcite and a large number of android privacy data leaks have been revealed @xcite . unlike previous work @xcite , we deal with the problem using sequence label-ing methods, and pay special attention to chinese privacy policies. the motivations come from the following four aspects: first, worldwide regulation departments enact laws @xcite @xcite @xcite to regulate the software's behaviors and protect users' privacy. the laws require the software to clarify how and why they need to access user data. analyzing privacy policies can help users understand how app process their data and identify whether apps comply with laws. second, for sequence labeling tasks, csp 3 aims to identify how and why the software collects, shares, and manages users' data according to regulations. csp 3 can be abstracted as identifying components in the privacy policy documents, such as data type and the purpose of using user data. nlp techniques can help automatically analyze privacy policies. third, existing privacy policy analysis research is limited to english and totally omits other languages. with over 98.38 billion app downloads @xcite and privacy-related regulations enacted in china, it is necessary and urgent to research csp 3 . last but not least, recent research in other communities, such as software engineering @xcite and cyber security @xcite , demonstrates requirements for analyzing privacy policies to help the analyst identify whether the apps' behavior is consistent with privacy policies. in this work, we make the following efforts to advance csp 3 : first, we construct a novel large-scale humanannotated chinese android application privacy policy dataset, namely ca4p-483. specifically, we manually visit the software markets, such as google play (google, 2022a) and app-gallery @xcite , check the provided privacy policy website, and download the chinese version if available. we finally collect 483 documents. to determine the labels in the privacy policy analysis scenario, we read through chinese privacy-related regulations and summarize seven components ( §2.2). we annotate all occurrences of components in 11,565 sentences from 483 documents. unlike paragraph-level annotations in existing privacy policy datasets @xcite , ca4p-483 annotates character-level corpus. second, based on ca4p-483, we summarize families of representative baselines for chinese sequence labeling. in detail, we first evaluate the per-formance of several classic sequence labeling models on our dataset, including conditional random forest (crf) @xcite , hidden markov model (hmm) @xcite , bilstm @xcite , bilstm-crf @xcite , and bert-bilstm-crf @xcite . recent work shows lattice knowledge improves the performance of chinese sequence labeling tasks. we involve lexicon-based models, such as lattice-lstm @xcite . third, we investigate potential applications of ca4p-483. combining law knowledge, we first identify whether the privacy policy violates regulation requirements based on ca4p-483. we also identify whether the app behaves consistently with privacy policy statements combing software analysis @xcite . the contributions of this work are three-fold: ## evaluation in this section, we evaluate baseline methods on all 18,579 sentences that are divided into training, development, and testing sets as detailed in we analyze the identification performance of each component to investigate the challenges and limitations of ca4p-483. next, we analyze the confusion matrix of bilstm-crf results that performs the best on ca4p-483. in fig. 2 , the depth of background color denotes the proportion of classification, the darker the color the higher the proportion, and the digit denotes the number of classification results. to have a deep understanding of divergences between ground truth and predictions, we inspect the misclassifications. we find that the algorithm may fail to identify conditions, which are in the adverbial clause as shown in fig. 3(a) where the highlighting for chinese is ground truth and highlighting for english is prediction results. besides, when the data controller is the user, as is shown in fig. 3(b) , the algorithms fail to distinguish purpose and condition. more illustrations in appendix 9.3 also reveal that models need to be well designed to learn deep semantic information, such as distinguishing overlapping among components, and distinguishing purpose in modifiers. ## case study in this section, we will present cases of potential applications of ca4p-483, such as whether privacy policies comply with regulatory requirements and whether privacy policies is consistent with the apps' functionalities. regulation compliance identification. chinese privacy-related laws (piss, 2020; nisstc, 2020; clprc, 2016) ask developers to clearly claim purpose conditions for processing user privacy data. we first investigate the distribution of annotations in ca4p-483. fig. 4 sketches the box plot of the frequency of components in each privacy policy. fig. 4 indicates that some privacy policies claim data processing without clarifying the purpose and condition, i.e., the minimum frequency of data is positive while that of purpose is zero. we manually inspect privacy policies. we find that the privacy policies, whose package name is com.yitong.weather, claim the app collects users' data while omitting to give the purposes or conditions of data access, which violates regulation requirements. thus, ca4p-483 can facilitate the research in the area of privacy compliance identification @xcite . app behavior consistency identification. to improve the security of the android community, researchers design systems @xcite to identify the consistency between privacy policies and app behaviors to prevent apps from abusing user data or conducting malicious behavior. one popular method to check the app's behavior is dynamic analysis (yan and yin, 2012), i.e., running the app on the device and checking the log information. to investigate the application of ca4p-483 in security community, we first identify the privacy policies without purpose or condition components. then, we install the app on one smartphone, manually interact with the app and try our best to trigger all possible functions in the app by clicking every visible buttons. we use logcat to capture the app's running information. we find that the app (id: com.chengmi.signin) requests device storage to use the app's functionalities while no condition-related statements are claimed in its privacy policy. with more intelligent automatic software engineering techniques, ca4p-483 can facilitate the research in this area, and more vulnerabilities in the consistency between app behavior and privacy policy could be investigated. ## related work prior privacy policy datasets are all english and omit other languages. opp-115 @xcite collects 115 english websites' privacy policies and makes annotations at the sentence level. opp-115 designs labels based on previous works @xcite . app-350 @xcite gathers android apps' privacy policies written in english. app-350 only conducts limited annotations, including two types of data controllers, namely first party and third party, thirteen types of specific data, and two types of modifiers, i.e., do and do not. existing chinese sequence labeling datasets are generally gathered from news @xcite @xcite and social media @xcite @xcite . the datasets include abundant corpus, but their annotations are limited to location, person name, and organization. even though cluener2020 @xcite expands the labels, such as game, gvoerment, book, the datasets are still hard to be applied in specific downstream tasks. cnerta @xcite includes another media data, i.e., voice data, to improves the sequence labeling performance. ## conclusion this paper introduces the first chinese android application privacy policy dataset, ca4p-483. ca4p-483 contains fine-grained annotations based on requirements of privacy-related laws and regulations. the dataset can help promote natural language processing research on practical downstream tasks. we also conduct experimental evaluations of popular baselines on our dataset and propose potential research directions based on the result analysis. we also conduct case studies to investigate potential applications of our dataset and potential application of our dataset to help software engineering and cyber security protect user privacy. in the future, we hope that we can build new models for ca4p-483 to counter the challenges. sharing 收集 (collect), 获取 (obtain), 接 受 (get), 接 收 (receive), 保 存 (save), 使用 (use), 采集 (gather), 记录 (record), 存储 (store), 储存 (store) collection 披 露 (reveal), 分 享 (share), 共 享 (share), 交换 (exchange), 报 告 (report), 公 布 (public), 发 送 (send), 交 换 (exchange), 转 移(transfer), 迁移 (migrate), 转 让 (make over), 公开 (public), 透 露 (disclose), 提供 (provide)