A Corpus for Multilingual Analysis of Online Terms of Service (2021)

Abstract: We present the first annotated corpus for multilingual analysis of potentially unfair clauses in online Terms of Service. The data set comprises a total of 100 contracts, obtained from 25 documents annotated in four different languages: English, German, Italian, and Polish. For each contract, potentially unfair clauses for the consumer are annotated, for nine different unfairness categories. We show how a simple yet efficient annotation projection technique based on sentence embeddings could be used to automatically transfer annotations across languages.

## introduction cultural and linguistic diversity is a guiding principle of the european union. within the context of an ever expanding eu, the concept of multilingualism stands out as one of the most prominent symbols of european historical, political, social and cultural diversity. @xcite from a legal perspective, the eu commitment to multilingualism is significant not only as a guarantee of legal certainty, clarity, and transparency, but also democratic accountability. accordingly, for example, the eu legislation is generally published in all official languages. @xcite while multilingualism certainly has important benefits, it also comes with costs and challenges. @xcite a recent indication of the importance of multilingualism for the eu is found in the european commission communication of 22 november 2005 "a new framework strategy for multilingualism" (com(2005) 596 final). to illustrate, the existence of 24 official versions of every legal act, all of which are equally authentic, 3 may inevitably create interpretative difficulties. in this regard, the court of justice has repeatedly found that the wording used in one language version of a union provision cannot serve as the only basis for its reading, and has pointed to the role of cross-language comparisons as well as teleological and systematic methods of interpretation. @xcite ethnic languages, and related requirements, are also an important issue in the eu internal market and consumer protection policies. the eu has so far mostly left this matter to national authorities, recognizing the associated tensions and cultural contexts. @xcite for example, the eu has refrained, so far, from regulating languages in which standard terms in consumer contracts should be drafted, allowing for differing approaches to emerge in various jurisdictions (loos, 2017). @xcite in view of the above, and considering that consumer protection authorities and non-governmental organisations in europe tend to operate in their respective languages, ef-3 judgment of the court of 6 october 1982, c-283/81 -cilfit, ecli:eu:c:1982:335, para. 18. @xcite see, e.g., judgment of the court of 19 december 2013, c-281/12 -trento sviluppo, ecli:eu:c:2013:859, para. 25 and the case law cited. in this work, we focus on the automated detection of unfair clauses in terms of service (tos), which are usually available in multiple languages. in particular, we investigate methods for automatically transferring the annotations made on tos in the context of the claudette project foot_0 onto the corresponding versions of the same documents in a target language, where such resources and expertise may be lacking. our ultimate goal is to use automatically generated annotations for training linguistic tools, usually available only in english, like claudette, for the target language without resorting to expert annotators in that language. this would leverage the creation of classifiers that can leverage the linguistic resources available in the target language, to analyze documents in that language. from a machine learning point of view, the question is whether we shall necessarily train independent models for each and every language. this paper only partially answers the question, by extending previous work (galassi et al., 2020) towards three different languages and by making a novel multilingual corpus available to the community. future research will consider training and comparing the classifiers. ## corpus we built a multilingual parallel corpus consisting of 25 terms of service annotated in english, italian, german and polish. these languages are spoken in large eu countries as well as in different regions, and they have been selected based on the availability of mother tongue legal experts for the annotation task. the analysed tos were retrieved from the claudette pre-existing corpus, covering 100 english tos (lippi et al., 2019; ruggeri et al., 2021). such terms mainly concern popular digital services provided to consumers, including leading online platforms (such as search engines and social media). the predominant language of drafting of these tos is english, with differing availability of corresponding tos in other languages. to carry out the present study, the ultimate 25 tos foot_1 were selected on the basis of three main cri-teria: a) their availability in the four selected languages; b) the possibility of identifying a correspondence between the different versions, given their publication date; and c) the similarity of their structure (e.g. number of clauses, sections, etc.). to illustrate, while tos in both german and italian were identified for 63 out of the 100 tos contained in the pre-existing claudette training corpus, polish versions were found for only 42 of these 63 tos. out of the 42 tos available in the four languages, we selected those with the more closely corresponding versions based on criteria b) and c) above. perfect correspondence across the 4 languages, however, could not be achieved for all 25 tos. as further discussed in section 3, some discrepancies may persist. we hypothesise that a particularly strong similarity between the english tos and other language versions exists where the latter are translations of the former. although we could not assess this comprehensively in the present study, we infer from the wording of the tos that at least in 9 out of 25 cases, german, italian and polish documents were indeed translations of the english originals. more specifically, in the tos of 9 companies 9 we identified explicit references to the language of the terms. consider, for example, the following clause taken from mozilla tos (updated june 23, 2016) and its corresponding german version: in the event of a conflict between a translated version of these terms and the english language version, the english language version shall control. (line 71). ## discrepancies our analysis revealed some discrepancies across the different language versions of the selected documents. this makes the projection task particularly challenging, also affecting the performance evaluation whenever a perfect match between documents is missing. a general observation concerns the asymmetric length of documents across the different language versions. in some cases, this is the product of missing/extra clauses, for example because of country-specific terms. this is the case, among others, for the terravision and spotify tos (see discussion infra in this section). in other cases, asymmetric length is a feature of linguistic differences. take for example evernote tos that, while showing a remarkable similarity of structure across the 4 languages (no extra/missing clauses), shows a different word and sentence count across the 4 languages: 6,798 words for the polish version (283 sentences), 7,580 for the german (291 sentences), @xcite as an example consider the following clause taken from the evernote tos: <@xmath0, you acknowledge, accept and agree with all provisions of the privacy policy, including, without limitation, the use and treatment of the text, images, and other data you choose to input, upload, or store in evernote (collectively, "content") and your personal information in accordance with such privacy policy.</pinc2>. 7,902 for the italian (270 sentences) and 7,795 for the english version (279 sentences). moving from the structure of documents to the analysis of sentences, we identified four types of discrepancy across languages. the first one concerns sentence structure. this type of discrepancy is mostly illustrated by situations in which the same information, contained in a single clause in the english version, is split into more than one clause in the target languages (or vice versa). in some cases such situations do not appear very problematic as the same tag of the original english sentence is identically reproduced across the two sentences in the target language. @xcite in other cases the implications for annotation are more problematic, as nested tags may need to be manually split. consider for instance the following sentence from the tripadvisor tos (updated october 1, 2018) in its english and italian version. other problematic cases may be those in which only one of the two sentences in the target language is relevant and should be marked as unfair. consider the following clause from the google tos (updated april 30, 2014) and their polish version: <@xmath1, and we may remove or refuse to display content that we reasonably believe violates our policies or the law.</@xmath2, czy s ą one nielegalne lub czy naruszaj ą nasze zasady. <@xmath3, że dany materiał jest niezgodny z tymi zasadami lub obowi ązuj ącym prawem, możemy go usun ąć lub odmówić jego wyświetlenia.</@xmath4 in this case, the different segmentation results in only one of the sentences in the target language being annotated as a potentially unfair content removal clause. the preceding part, related to the reviewing of the content, remains irrelevant for the envisaged task. a second type of discrepancy consists in errors or inaccuracies in translation from the source english tos into the target languages. consider the following example from the english and polish versions of the yelp tos (updated november 27, 2012). la registrazione al servizio o il suo uso richiedono di confermare la propria accettazione dei termini, cosa che rende di conseguenza l'utente un soggetto vincolato al presente contratto. (line 4). here the english sentence is a clear example of a contract by using clause, thus potentially unfair (by registering, the consumer consents to the tos). by contrast, the italian translation simply states that in order to register for the service, the consumer needs to accept the terms. thus, the italian version has not been annotated. a similar discrepancy is not observed in the german and polish versions of this sentence, which again suggests a translation error. the third type of discrepancy we identify is missing/extra clauses, meaning the absence of certain clauses in the different versions of the tos. consider, for instance, the following clause taken from the terravision tos (date of update not provided; tos accessed on 31 january 2021), which is completely missing in the italian tos: missing/extra clauses may be the product of an omission or error, but they also may be product of deliberate choices to draft contracts differently for different national markets. such different choices may be made necessary by country-specific services offered only at certain locations, by the need to comply with country-specific regulation, or, more generally, as product of the company's intention to regulate the contractual relationship differently in different countries. for example, the need to comply with country specific regulation may result in substantive differences in clauses providing for age requirements to use certain services. @xcite in a few cases, especially in the german versions of some tos, an entirely different approach is adopted to the drafting of clauses related to certain matters, like provider's liability. among others such discrepancies were found in the electronic arts tos. @xcite the fourth type of discrepancy derives from the choice of legal terminology, which may be more or less deliberate. indeed, legal languages remain profoundly culture-bound and terminological nuances in different languages may be difficult to capture regardless of the applied translation process. in our corpus this type of discrepancy emerges in relation to the use of particular terms, for instance terms related to responsibility and liability. unlike in english, in the other languages of our corpus such a distinction is hard to capture. consider, for instance, the following clause taken from the pinterest tos (updated may 1, 2018) and its polish and italian counterparts that employ the expressions "odpowiedzialność" and "responsabilità", without differentiation between responsibility and liability: while linguistic variations related to responsibility (obligations) and liability do not always entail differing annotations, they sometimes may. consider, for example, the following situation from the terravision tos (as accessed on 31 january 2021): <@xmath5.</@xmath6. here, a translation of the english term "to be liable" into the more generic italian "avere obbligo" results in different annotation choices in the different languages. finally, as a general remark, it is important to note that deviations from the english source tos are uneven across languages, being largest in german tos. this may suggest that the drafting of such tos is done by human agents, who may pay more attention to the national legal context and specific terminologies than automated translators. for example, note the markedly different take on the matter of privacy and data protection in these clauses of spotify, where the german drafters refrained from packaging data protection consent with the agreement to the tos: <@xmath7.</@xmath8, privacy policy, and any additional terms that you agree to, as discussed in the entire agreement section, are referred to together as the "agreements".)</@xmath9, der sie zustimmen, wie im abschnitt vollständiger vertrag beschrieben (gemeinsam als die "vereinbarungen" bezeichnet). (line 37) the hypothesis of a more careful drafting being applied in the drafting of german language documents seems confirmed by the fact that retrieving identical corresponding versions of tos was most hard for german. conversely, we observed a lower mismatch in both polish and italian tos, where significant structural differences can be retrieved only in limited cases. ## experiments we employ the same methodology proposed in galassi et al. (2020), by extending the study to two additional languages, namely italian and polish. the general idea of the methodology is to transfer the annotations of a given english document onto the same document, given in a different language. as a first step, we exploit an automatic translation system to obtain english sentences from the target language (i.e., german, polish, or italian). then, we find correspondences between the original (tagged) english sentences and the ones (untagged) obtained via translation. in this way, the annotations of the first document can be associated to the sentences in the second document. for this step, any distance or metric that measures the (dis-)similarity between sentences can be exploited. in this work, we use the distance between elmo (peters et al., 2018) sentence embeddings coupled with dynamic time warping (berndt and clifford, 1994), which was the approach achieving the best performance in galassi et al. (2020). finally, the projected annotations are then easily associated back to the original sentences in the target language. to segment our documents into sentences we exploit the spacy python package, @xcite whereas for translation into english we use the deep_translator python library, in particular the google translator. @xcite we measure the performance of our system by considering the result of the projection as a multi-label classification task on the sentences of the target documents. we report precision as the percentage of projected tags that are correct and recall as the percentage of original tags that have been correctly matched to the target document, in both cases macro-averaged over the documents. f1-macro is the harmonic mean between the first two metrics. we also report the f1-micro, which sums up all the true positives, false positives, and false negatives, without distinguishing among the classes, i.e., tags. finally, the f1-weighted is the weighted sum of the f1 scores computed per each class (tag), where the weight is given by the support (i.e., number of tagged sentences) of each tag. as shown in ## discussion discrepancies across language versions may significantly affect the performance of the projection task. in particular, we found that the system produces some erroneous results where a) there is no correspondence between clauses (e.g., some of them are missing in one or more language versions); b) sentences are split differently; c) there is a mismatch in the tos structure; d) incorrect translations or specific linguistic choices can be identified. as an example of no correspondence between clauses, consider the electronic arts tos (updated may 17, 2018), in which liability clauses are built in an entirely different manner between english and german. in this case the projection missed the two following clauses, which have no equivalent in the english version: division of longer passages into shorter sentences as well as discrepancies in the documents' structure have been identified as recurrent causes of false positives and false negatives also in relation to examples reported in section 3. this is the case, among others, for the previously described term on content removal in the polish version of the google tos, where only one of the resulting sentences in the target language merited annotation. erroneous projection outcomes could also be identified when a longer english sentence, marked under two categories using a nested tag, was divided into two shorter sentences in the target language, each annotated under one category. the previously mentioned clause in the tripadvisor tos on unilateral change and contract by using illustrates such a case. moreover, a comparably high number of false positives and negatives was observed in relation to terravision, where the document structure varied in different language versions, among others due to country-specific terms. finally, as an example of incorrect projection due to translation errors and linguistic choices, consider the case of a clause in grindr tos (updated 13 march 2020), where the phrase "bez zobowi ązań" (without commitments) was used instead of a reference to liability. while this clause was not marked in the target language, the system nonetheless incorrectly recognized it as a potentially unfair limitation of liability. the same was true for the previously mentioned clause in the italian terravision tos, referring to a generic term "obbligo". erroneous projection results were further identified in relation to the previously mentioned clause in the yelp tos, which in the polish version lacked reference to liability. similarly, a clause on contract by using, which was marked in the english tos of evernote, but not in their italian version, was found among false positives. ## conclusions this paper presents an extension of the annotation projection technique presented in galassi et al. (2020) for the automated detection of potentially unfair clauses in online tos by applying it to multiple languages. the presented approach directly addresses the call for automatically transferring the annotations made on legal documents in a language with significant linguistic resources and domain experts, such as english, onto the corresponding ver-sions of the same documents in a target language, where such resources and expertise may be lacking. this approach does not apply to online tos only, but to a wide category of legal documents, and in particular within the european union, with a great variety of different languages. for these reasons, we presented the first multilingual corpus for the detection of potentially unfair clauses for the consumers in online tos. the data set consists of 25 contracts tagged in four different languages (english, german, italian, and polish) for a total of 100 documents. we illustrated the discrepancies across the different versions of the same contract, and we analyzed the major causes of errors in the projection process. in the future, we will further study correspondence between the english and other language versions, and consider whether computational methods can be deployed to confirm existence of translations. our final aim is to compare machine learning predictors trained on original and projected annotations, to assess whether the projection procedure can substitute the time-consuming task of document annotation performed by experts.