Privacy Pitfalls of Online Service Terms and Conditions: a Hybrid Approach for Classification and Summarization (2022)

Abstract: Verbose and complicated legal terminology in online service terms and conditions (T&C) means that users typically don’t read these documents before accepting the terms of such unilateral service contracts. With such services becoming part of mainstream digital life, highlighting Terms of Service (ToS) clauses that impact on the collection and use of user data and privacy are important concerns. Advances in text summarization can help to create informative and concise summaries of the terms, but existing approaches geared towards news and microblogging corpora are not directly applicable to the ToS domain, which is hindered by a lack of T&C-relevant resources for training and evaluation. This paper presents a ToS model, developing a hybrid extractive-classifier-abstractive pipeline that highlights the privacy and data collection/use-related sections in a ToS document and paraphrases these into concise and informative sentences. Relying on significantly less training data (4313 training pairs) than previous representative works (287,226 pairs), our model outperforms extractive baselines by at least 50% in ROUGE-1 score and 54% in METEOR score. The paper also contributes to existing community efforts by curating a dataset of online service T&C, through a developed web scraping tool.

## introduction despite legislative advances such as the european union's general data protection regulation (gdpr) 1 regarding specific, informed and unambiguous consent for the collection and use of personal data on the internet @xcite , understanding how online services can read, edit, distribute and sell user data, as documented in their terms of service (tos), remains out of reach for the typical user, with most (98%) consenting to the terms without reading the documents in their 1 https://eugdpr.org/ entirety @xcite . two major factors contributing to this are the length of the documents and the ambiguous and complicated terminology used @xcite , with users unable to interpret the implications of the terms of such a legally-binding unilateral contract. in addition to the implication for users' rights, the distribution and use of user data is also important for companies looking to use third-party services in their product. this can be exemplified with the case of the global science research (gsr) company tasking cambridge analytica to build psychological profiles of users through a quiz app, which also collected information from the users' facebook friends, allowing the company to acquire data from millions of unwitting facebook users foot_0 . this data was then matched with existing voter datasets, enabling aggressive voter-targeting operations in the 2016 us presidential election foot_1 . delving into the app's tos reveals that it states: "we collect any information that you choose to share with us ...this may include, inter alia, the name, demographics, [. . . ] of your profile and of your network." in addition to this, they permit gsr to "edit and sell" user data by accepting the conditions @xcite . when queried in 2018 if it had read and evaluated the terms and conditions for the app, facebook responded: "we did not read all the terms and conditions" 4 . with a lack of regulations around standard terms in which consumer contracts should be drafted @xcite , a condensed equivalent of the salient points of a tos document can em-power users to understand their rights and avoid privacy invasion and legal disagreements. summarization, which condenses text into a shorter form whilst keeping the most crucial and informational parts intact, is an intuitive approach for replacing unnecessary (and in some cases, intentionally convoluted) long text with a digestible summary. building on recent community approaches to annotate and curate tos sentence/summary pairs @xcite , we propose a hybrid extractive-classifier-abstractive model that can extract tos sentences related to privacy and data collection/use and paraphrases these into concise and informative tos highlights. the hybrid model forms part of a web application and browser plugin that enables users to view an at-a-glance summary of any online service (specified through its url) t&c. we also contribute to community efforts for curating a tos dataset by developing a web scraping engine to build a novel tos dataset from 163 different online services. the proposed hybrid model addresses limitations of existing works as it relies on significantly less training data (4313 training pairs) than previous representative works in hybrid extractive-abstractive methods @xcite . the summarization results are compared against the baseline unsupervised, extractive techniques, achieving significant improvements in performance (50% improvement in rouge-1 score versus the best performing baseline, and 54% in meteor score). ## related work this section explores the state-of-the-art community efforts and research within the domain of t&c and text summarization. the "terms of service; didn't read" (tos;dr) foot_3 community-driven project highlights alarming statements in tos. services are given grades ranging from a-e based on the severity of the terms listed; e being very serious concerns. summaries are manually submitted by the tos;dr community, which limits the scope of summarization to only those that already exist in the database. moreover, manually reading and analysing long documents of terms is a laborious and time-consuming task. the tl;drlegal foot_4 website hosts community-submitted software license summaries that are peer-reviewed by the website managers, with the same manual-process limita-tions as tos;dr. a notable work in tos data curation is that by @xcite with 446 sets of contract sections and corresponding reference summaries from tos;dr and tl;drlegal, thus presenting the first dataset in this genre. automated summarization techniques have been applied successfully to curated news (e.g. cnn/dailymail corpus @xcite ), scientific articles @xcite or microblogging (e.g. large scale chinese short text summarization (lctcs) @xcite ) corpora. categorized either as extractive @xcite or abstractive @xcite methods, existing summarization approaches are however, not directly applicable to the t&c domain. extractive approaches work by selecting the most salient sentences for the summary @xcite and deciding on their order of presentation. they rely on the structural features of documents, i.e. typically news, scientific articles, where the title and abstract/first few lines of the document, contain a snapshot of the key content. these heuristics do not translate well for tos documents, which have differing structures for different jurisdictions and where the terminological nuances in legal language are difficult to capture @xcite . moreover, as the resultant extractive summary matches source sentences word-for-word, complex legal terms in the summary may still confuse the reader @xcite . existing works for privacy policies and tos include the extractive approach of a supervised convolutional neural network (cnn) model @xcite to predict which content has the most risk of unsafe data practices, that is followed by extracting a calculated amount of sentences with the highest risk score. the model did not perform well when compared to the tos;dr summaries, as a fully-extractive approach cannot mimic the humanlike qualities in the tos;dr summaries, and also suffers as it generates "legalese" rather than plain english, making it less accessible. abstractive methods, on the other hand, generate concise summaries by compressing and paraphrasing, but are weak at content selection and prone to information loss @xcite . these supervised approaches also require a large corpus of parallel document/summary pairs for training neural models and their evaluation. unlike the news/microblogging genres, where such large curated datasets are available for training, resources for tos documents are currently not large enough, being "intended for evaluation, rather than training" @xcite . other abstractive models include pointer-generator models with coverage mechanism @xcite , which use pointing @xcite , and a hybrid extractiveabstractive approach to improve accuracy and handle unknown words. ## data this section describes the datasets (shown in figure 1 ) created and compiled for training and evaluation of the hybrid tos model. ## methodology the methodology for the tos hybrid model proposes to automate the summarization and grading (sentence extraction) process, allowing a broader scope of companies and websites to be analyzed while also removing the manual step of summary review. ## conclusions in this paper, we proposed a domain-aware hybrid extractive-abstractive model that highlights privacy and data collection sections in a tos document and paraphrases these into concise and informative sentences. a novel dataset is also created using a developed web-scraping tool, with the purpose of automatically fetching tos documents from any online service. the dataset used for classification training was found to be highly imbalanced; despite this, the hybrid model performed well in rouge and meteor scores when compared against unsupervised, extractive baselines. to resolve the imbalance problem, the data was resampled before being used in the classifiers for training, which reduced the false negative rate by 64%. however, this did increase the false positive rate, which implies that the extractive classifier is more inclined to incorrectly label sentences as important. given that the abstractive model is only trained on important sentences, this can lead to incorrect warnings by the tos model. in the context of this paper, to maintain the integrity of legal concepts, this can still be seen as the preferable outcome since users can verify statements made by the model by reviewing the original tos if necessary. more training data for both the classifier and abstractive model can improve performance; this can be obtained by the developed web-scraping tool, in addition to future tos;dr community contributions. this would result in more data for the classifier post-resampling, which in turn would help the imbalance issue and false positive rate. another direction for future work would be testing the generated summaries for comprehension through qualitative user studies, from participants recruited through platforms such as mturk or prolific.