Clause Topic Classification in G erman and E nglish Standard Form Contracts (2022)

Abstract: So-called standard form contracts, i.e. contracts that are drafted unilaterally by one party, like terms and conditions of online shops or terms of services of social networks, are cornerstones of our modern economy. Their processing is, therefore, of significant practical value. Often, the sheer size of these contracts allows the drafting party to hide unfavourable terms from the other party. In this paper, we compare different approaches for automatically classifying the topics of clauses in standard form contracts, based on a data-set of more than 6,000 clauses from more than 170 contracts, which we collected from German and English online shops and annotated based on a taxonomy of clause topics, that we developed together with legal experts. We will show that, in our comparison of seven approaches, from simple keyword matching to transformer language models, BERT performed best with an F1-score of up to 0.91, however much simpler and computationally cheaper models like logistic regression also achieved similarly good results of up to 0.87.

## introduction so-called standard form contracts, i.e. contracts that are drafted unilaterally by one party of the contract, usually a company, like terms and conditions of online shops or terms of services of social networks, are cornerstones of our modern economy. while the concept of a contract that is completely decided upon by one party might seem unfair and inherently flawed, its existence is a necessity in our modern economy. it would simply not be possible for companies like amazon, facebook or google, to negotiate individual contract terms with each of their customers. it is largely acknowledged, that most consumers do not read such contracts before buying something online or registering for a service. the actual share of consumers that regularly read such contracts ranges from as little as 3.5% @xcite to 9% @xcite in the literature. in acknowledgement of this fact, lawmakers around the world have tightly restricted the provisions that can be made by standard form contracts in a bid to protect consumers. this makes them an interesting subject for different natural language processing (nlp) tasks, because they have a high economical and therefore practical relevance, but are still somewhat restricted with regard to their content. in this work, we use contracts that have been drafted under the jurisdiction of the european union (eu). many of the regulations applying to standard form contracts in this jurisdiction originate from the council directive 93/13/eec of 5 april 1993 on unfair terms in consumer contracts. while standard form contracts are relevant in almost all kinds of business-to-consumer transactions, like banking, insurances, and data processing, we here focus on terms and conditions (t&c) from online shops for three reasons: they have high economical relevance, they are publicly available in large quantities on the internet in a machinereadable format, and they are among the contracts which are least likely to be read, compared to a contract for life insurance, for example. being able to automatically classify the topics of clauses in t&c could help consumers to find relevant regulations faster and make more informed decisions, but it could also support legal professionals, like lawyers specialising in consumer protection law, in their work. ## related work contract review is one of the main commercial applications of nlp in the legal domain @xcite . unlike standard form contracts, "normal" contracts, i.e. contracts that have been negotiated by all contracting parties, allow for more variation and are less regulated, especially in business-to-business contexts. therefore, building a taxonomy of clause topics and performing topic classification (i.e. a supervised approach), would be less suitable for such contracts. for reasons of data availability, most research projects use publicly available contracts, which happen to be standard form contracts, like t&c from online shops, terms of services (tos) from online platforms, and, since the introduction of the general data protection regulation (gdpr) in the eu, one particular focus has been on privacy policies. although, legally, it is not settled whether they have a contractual status or not @xcite , from an nlp perspective, they can be treated as a special variety of standard form contracts. most of the existing research on the analysis of standard form contracts is focusing on automatically finding void clauses. the claudette project @xcite , for example, focuses on finding so-called "unfair clauses", which are void under eu legislation, in tos from large online platforms like facebook or netflix. later, they also applied their approach to privacy policies @xcite . similarly, @xcite focus on finding void clauses in t&c from online shops by using a fine-tuned bert model, in earlier work, they summarised specific aspects of t&c using an abstractive summarisation approach @xcite . in comparison to these works, which try to make a fully automated decision on the validity of clauses, the classification of clause topics could be used to support humans in the decisionmaking process, by helping them to find relevant clauses faster. in the area of privacy policies, approaches are more diverse. @xcite , for example, presented a q&a system that can answer user questions about privacy policies. binary assessments in classes like valid or void are less desirable in the domain of privacy policies where, especially before the introduction of the gdpr, much of what was legally allowed was still undesired by users. ## taxonomy for a topic classification approach, i.e., a supervised approach, rather than an unsupervised topic modelling approach, a taxonomy of clause topics is needed. at first, this might seem like a limitation of the approach, because, in theory, a contract can regulate arbitrary aspects. in practice, however, standard form contracts underly strong lim-itations, because, under eu jurisdiction, clauses that are "unexpected" are automatically void under the unfair contract terms directive (93/13/eec). therefore, if a taxonomy is extensive enough, the information that a clause is not covered by one of the topics in the taxonomy is already important information in itself, because it means that the clause is very likely void. to build such an extensive taxonomy for t&c from online shops, we used contract templates from legal literature @xcite and industry associations (ihk munich and upper bavaria, 2020; @xcite , as well as a commercial t&c generator foot_0 and analysed which topics are present in these templates, because they are used by many online shops. for each of these sources, two legal experts with experience in consumer protection law went through the templates and annotated each clause with one or more fitting topic and zero or more fitting subtopic label(s). in the end, the topic labels from the different annotators were aligned by the authors. by the combination of the abovedescribed sources, we derived a taxonomy of 22 classes (topics) and 36 sub-classes (subtopics). the differentiation between topics and subtopics was mainly based on the structure of the sources, i.e. the organisation in sections and subsections in the templates. a topic, for example, could be "delivery" and subtopics could be the delivery time or the delivery costs. to get an estimate of how extensive our taxonomy is, we used it to manually annotate more than 6.000 clauses from real t&c (see section 4). of these more than 6,000 clauses, the taxonomy was able to cover 90.08%. the remaining clauses (336) fell into only two classes: 285 clauses contained information regarding vouchers and gift cards, and 51 clauses contained information about codes of conduct. we added both classes to the taxonomy. the final taxonomy, therefore, consists of 23 labels for topics and 37 labels for subtopics. all labels in the taxonomy are shown in ## corpus since no corpus of topic-annotated clauses from t&c existed, we had to build our own corpus. for this, we parsed the list of merchants from two german price comparison websites ("idealo" 2 and "geizhals" foot_2 ) that also offer a localised version of their respective websites in english, targeted to the british market foot_3 . on these websites, shop operators manually report the urls to their t&c, which we extracted with a web-crawler. we randomly selected 142 german t&c and 30 english t&c from these pages. each clause from these contracts was subsequently copied into an excel file, in which each row contains one clause. in addition to the text of the clause itself, each row contains a unique id, an id for the contract the clause belongs to, (if existing) the title of the superordinate paragraph and (if existing) the title of the clause. ## approaches we compared seven different approaches to the classification of clause topics in standard form contracts from german and english online shops: rulebased keyword matching, logistic regression, random forest, multilayer perceptron (mlp), long short-term memory (lstm), and bidirectional encoder representations from transformers (bert). in the following sections, we will shortly introduce how we used the different approaches. for each of the approaches, we trained two classifiers, one which only classifies topics and one which only classifies the subtopics. experiments we conducted with joint classification models, i.e., models that classify topics and subtopics at the same time turned out to decrease the classification quality for both, topics and subtopics. we split both corpora into a training (80%) and a test (20%) set, using scikit-multilearn @xcite to make sure the representation of labels is balanced between the training and the test set and reflects the original distribution. for the stochastic approaches, we performed a grid search with a k-fold cross-validation on the training data to find the optimal parameters for each approach. our initial hypothesis, based on similar research, was, that with increasing complexity of the models, the performance would also increase, i.e. we expected bert to perform best, followed by lstm, mlp, and the "classic" ml approaches. ## evaluation our baseline approach of using keywords achieved an f1-score of 0.78 in german for topic classification and 0.64 for subtopic classification. the performance in english was worse with an f1score of 0.72 for topics and 0.46 for subtopics. we noticed that, in german, the list of keywords in english, the picture was less clear, with logistic regression still performing best for topic classification (f1-score 0.80) but being surpassed by the neural network approaches for subtopic classification. however, the clear overall winner, with the best performance on both languages and classification levels was bert, which scored up to 0.91. for the approaches we evaluated with different inputs, i.e. the mlp and lstm, the values in ## transferability due to their practical relevance and availability, we focused on t&c from online shops in this paper. however, there is no reason why the same technology could not be applied to other types of standard form contracts, e.g. from banks and insurances. at the same time, the taxonomy we developed and the models we trained have a component that is specific to online shopping, broken packaging, for example, is a topic, that is not relevant for banking. to get an idea of how domain-specific the taxonomy and the models are, we annotated the t&c of three of the largest german banks (commerzbank, deutsche bank, and sparkassen) in the same way described in section 4.2: first, a student, then authors annotated the all clauses of the contracts independently with their topics and subtopics, then conflicting labels were resolved by the team of experts. we used the taxonomy described in section 3 for the classification, however, we added a class "n.a." to mark clauses that cover a topic that is not represented by any of the classes in the taxonomy. the three contracts consist of 214 clauses, about 71 clauses per contract, and 13,681 words, an aver- the annotation process confirmed this assumption. of the 214 clauses, only 71 (or 33%) are concerned with a topic that is covered by our taxonomy (see since the taxonomy is still able to cover onethird of the banking contracts, we wanted to test how well the classifiers we trained would perform on this data set. therefore, we took the best performing topic classifier, i.e., the bert model, and applied it to the new corpus. the evaluation of the results is shown in ## conclusion in this paper, we compared different approaches to classify the topics of clauses in standard form contracts from online shops, based on a taxonomy while bert did indeed perform best for both languages, the much simpler logistic regression approach showed the second-best performance. considering the computational time and power that is needed to not only train the more complex model but also during inference of the labels, the simple logistic regression approach might for some practical application be the better choice. we were surprised to find that multilingual approaches, using the german and english data together, did not seem to bring any improvements for this task, even though earlier work in the legal domain, e.g. by @xcite and @xcite , have shown that multilingual models can improve performance. this aspect needs further investigation. while the taxonomy we developed and the models we trained are domain-specific for ecommerce, first tests suggest that the approaches can be transferred to other types of standard form contracts and that event the models can partially be transferred to other domains, at least for more "technical" clauses concerning the contract itself. this could apply to all types of consumer standard form contract within a highly regulated domain, like insurances, housing, and employment, and is something we would like to investigate further in the future.