AGB - DE : A Corpus for the Automated Legal Assessment of Clauses in G erman Consumer Contracts (2024)

Abstract: Legal tasks and datasets are often used as benchmarks for the capabilities of language models. However, openly available annotated datasets are rare. In this paper, we introduce AGB-DE, a corpus of 3,764 clauses from German consumer contracts that have been annotated and legally assessed by legal experts. Together with the data, we present a first baseline for the task of detecting potentially void clauses, comparing the performance of an SVM baseline with three fine-tuned open language models and the performance of GPT-3.5. Our results show the challenging nature of the task, with no approach exceeding an F1-score of 0.54. While the fine-tuned models often performed better with regard to precision, GPT-3.5 outperformed the other approaches with regard to recall. An analysis of the errors indicates that one of the main challenges could be the correct interpretation of complex clauses, rather than the decision boundaries of what is permissible and what is not.

## introduction standard form consumer contracts, i.e. consumer contracts that are drafted unilaterally by a company, have huge significance for our economy, but also for consumer protection. their review is a laborious task, performed by companies, law firms, governmental organizations, and nongovernmental organizations (ngos). in recent years, researchers have investigated different computational approaches to automate parts of the contract reviewing process (see section 2). one of the big challenges for such research is the scarcity of contract data in general and annotated data in particular. even more so for languages other than english. annotating contracts with legal assessments is a laborious process that can only be performed by highly qualified experts and is therefore very expensive. commercial providers of large language models (llms), like openai, but also the scientific community, increasingly use legal tasks to assess (or promote) the capabilities of llms. for many of the existing datasets and tasks, it is questionable whether llms like gpt-3.5 have not been trained on them, which would question the validity of evaluations performed on them @xcite . in this paper, we present a new corpus consisting of 3,764 clauses, 11,387 sentences, and 250,859 tokens from german consumer contracts. each clause has been annotated by legal experts with a clause topic and whether the clause is valid or potentially void (see section 3.2). a contract clause is a section in a contract that is usually separated explicitly through formatting from other clauses and deals with a specific provision. the corpus contains a total of 8,582 labels and is available on github foot_0 and as hugging face dataset foot_1 . in addition, we present a baseline for the task of identifying potentially void clauses, comparing an svm, three open language models in different sizes, and gpt-3.5. our results show the challenging nature of the task, with no approach exceeding an f1-score of 0.54. the best-performing model, agbert, is also available for download 3 . while the fine-tuned models often performed better with regard to precision, gpt-3.5 outperformed the other approaches with regard to recall. an analysis of the errors indicates that one of the main challenges could be the correct interpretation of complex clauses, rather than the decision boundaries of what is permissible and what is not. the code that was used to prepare the datasets and train the models is also available on github. we hope that the corpus will contribute to enabling future open and reproducible nlp research. ## related work legal tasks and datasets have become increasingly relevant for the evaluation of language models over the past years. from domain-specific models, like legal-bert @xcite , to benchmark datasets, like lexglue @xcite , culminating in the presentation of gpt-4 by openai, where the alleged "legal skills" of the model have been one of the main communication points to show its improvement from the previous version @xcite . additionally, the question of whether a gpt model would be able to get through law school @xcite or pass the bar exam @xcite has been raised repeatedly in the last few years. ## corpus the corpus consists of 3,764 clauses from 93 standard form consumer contracts. in this section, the construction of the corpus is described. a detailed datasheet @xcite for the corpus can be found in appendix f. ## corpus analysis the corpus consists of 93 contracts with 3,764 clauses (an average of 40 clauses per contract), which contain 11,387 sentences (avg. of 3 sentences per clause) and 250,859 tokens (avg. of 22 per sentence). out of the 3,764 clauses present in the corpus, 179 (or 4.8%) have been annotated as potentially void. that is comparable, although slightly lower, than the 6% reported by @xcite on a much smaller dataset of 24 contracts. while that results in a corpus that is imbalanced, we believe it to be a realistic reflection of reality, where void clauses are also significantly less frequent than void clauses. an analysis of the distribution of void clauses in relation to the topic shows that there are some classes which are particularly prone to be seen as potentially void by the experts. a deeper analysis, together with the experts, revealed that these are very often types of clauses that are particularly strictly regulated. the class changes, for example, captures clauses that relate to changes that are made to the contract after it came into force. given that standard form contracts are already considered to "reflect an imbalance of contracting power" @xcite , it is not surprising that the possibilities for a company to change them after the customer agreed are very strictly regulated. therefore, it makes sense that such clauses are exceedingly considered void by the annotators. similar reasoning can be applied to other topics, like liability or severability clauses. ## automated legal assessment in order to present a baseline for the main task for which the corpus was designed and evaluate the difficulty of the task, we compared different language models and an svm for the classification of clauses into (potentially) void and valid. ## evaluation the results of the evaluation are shown in for the agb-de dataset, none of the models was able to handle the very imbalanced dataset well. the best-performing model on the dataset was the fine-tuned bert model. while roberta is usually better than bert in handling imbalanced data @xcite , the roberta model that we fine-tuned is a multilingual model, while the bert model that we fine-tuned was specifically pre-trained on german legal data. the second best performance was achieved by the svm with an f1-score of 0.31. for gpt-3.5, there was not much difference in the performance independent of the dataset: for both datasets, gpt-3.5 achieved the highest recall, at the cost of a very low precision. i.e., gpt-3.5 falsely classified the majority of clauses as potentially void (see also figure 2d ). all models performed better on the agb-de-under dataset, showing that undersampling is a suitable strategy for the corpus. bert again performed best with an f1-score of 0.54. to test whether the improvement performance on the undersampled dataset was caused by a better model and not just by the fact that, by chance, difficult items got removed from the test set, we also tested all models trained on the undersampled dataset on the original test set from the agb-de dataset. the results of this evaluation are shown in appendix d. except for the svm, all models performed better when being trained on the undersampled dataset and evaluated on the full dataset compared to being trained and tested on the full dataset, indicating that removing excessive imbalanced training data can indeed lead to better models on this dataset. ## error analysis figure 1 shows precision and recall for each model on the undersampled dataset for the largest topics in the corpus. the numbers show that the models perform very differently for the individual topics. for liability, for example, no model was able to achieve a recall higher than 0.25. with 11% of void clauses, liability clauses are most frequently void among the large topics. for the expert annotators, liability clauses were relatively easy to annotate, because of very explicit regulations. § 309 no. 7 of the german civil code, for example, explicitly states that "an exclusion or limitation of liability for damage from injury to life, limb or health due to negligent breach of duty by the user" is void. linguistically, however, an analysis of the liability clauses shows that they are often complicated, with multiple explicit inclusions and exclusions in one sentence, which could be a reason for the poor performance of most models. while clauses of a certain topic can be void for different reasons, for some topics there are predominant patterns. warranty clauses, in the corpus, for example, are predominantly void because they restrict the warranty in cases where defects are not reported within a specified (too short) time frame. similarly, payment clauses are predominantly void because they introduce excessive fees for late payments. the bert model was better at picking up these more repetitive patterns compared to the more nuanced other topics. the gpt-2 model achieved the highest precision for payment clauses, which have the largest share in the corpus, and therefore heavily influence the overall result. overall, the performance differences between classes could indicate that using the topic labels as an additional feature could improve classification performance. the performance of gpt-3.5 differs from the other models as it was more prone to generating false positives, i.e. flagging clauses as void that are actually valid. the prompt we used for gpt-3.5 not only asked it to perform the classification into potentially valid and void but also asked the model to provide an "explanation" foot_6 . while the texts provided are not an explanation in the sense that they make the reasons for the assessment transparent, it is still interesting to see that the issues described are most of the time correct, however, the assessment is still wrong, in both the text and the annotation. for a clause about withdrawals (corpus id 5), gpt-3.5 dataset model precision recall f1-score agb-de svm 0.37 0.27 0.31 bert-base-german-cased 0.50 0.27 0.35 xlm-roberta-base 0.00 0.00 0.00 gerpt2 0.71 0.14 0.23 gpt-3.5-turbo-0125 0.06 0.92 0.11 agb-de-under svm 0.40 0.32 0.36 bert-base-german-cased 0.51 0.57 0.54 xlm-roberta-base 0.75 0.08 0.15 gerpt2 0.64 0.43 0.52 gpt-3.5-turbo-0125 0.13 0.92 0.22 figure 2 : confusion matrices for the evaluation on the agb-de-under dataset for example "explains": "the clause is potentially invalid as it could unfairly disadvantage the consumer. according to the law, the consumer must be able to exercise his right of withdrawal clearly and conspicuously, without any additional hurdles or conditions being imposed. by specifying here that registering a return under my account is considered a revocation, this could limit the consumer's ability to exercise their right of revocation. it is important that the consumer can exercise his right of withdrawal without additional obligations." (see appendix e.1 for the original german text) while the text by itself would be factually correct, it is not applicable to the clause in question. the clause clearly describes other ways to exercise the right to revoke a contract and explicitly states that it is not mandatory to use the "my account" feature. in line with the text, the clause was falsely labeled as potentially void by gpt-3.5. similarly, for the clause "if the customer is a merchant [...] the place of jurisdiction for all disputes arising from this contract is the court responsible for the seller's place of business [...]." (corpus id 3212) gpt-3.5 concludes the clause is potentially void, because "the clause puts the consumer at an unreasonable disadvantage because this clause binds the consumer to the company's place of jurisdiction''. the legal reasoning of that is again correct, however, not applicable to the clause, because the clause only applies to merchants in which case it is valid. to test whether gpt-3.5 is aware that the clause is valid for merchants, we adapted the prompt to say that the system should imagine being a lawyer for a merchant (instead of a consumer) and indeed the response was "the clause is unlikely to be potentially invalid if the customer is an entrepreneur." more often than not, the main challenge for gpt-3.5 seems not to be missing legal "knowledge", but the incorrect interpretation of the clause, especially in cases where the clause contains elements that are optional or not applicable to all customers. another typical problem of llms in general can be seen in the explanation generated for the clause with the corpus id 10. here, the model generates the following text: "according to section 357 of the german civil code (bgb), a consumer may not be charged the costs of return shipping in the event of a revocation. therefore, a clause requiring the consumer to bear the costs of return shipping is potentially invalid." (see appendix e.2). until 2014, it was indeed the case that "a consumer may not be charged the costs of return shipping in the event of a revocation" (at least for purchases over 40 eur). however, today, §357 bgb explicitly states the opposite, i.e. that customers have to bear the costs of return shipping. however, texts that relate to the old legislation are probably more frequent in the data gpt-3.5 was trained on, which is data up to 2021. ## conclusion in this paper, we have introduced the agb-de corpus, consisting of 3,764 clauses from german consumer contracts that have been annotated by legal experts. in addition to the corpus, we presented two datasets that have been derived from the corpus and have been split into training and test sets. the datasets can easily be used to train or fine-tune machine learning models. we have used these datasets to provide a benchmark on the corpus for the task of classifying whether a clause is valid or potentially void. for the dataset that is representative of the distribution in the corpus, we showed that language models struggle with the imbalanced data and could not achieve an f1-score above 0.35. for the second dataset, which is more balanced through undersampling, we showed that open models like bert and gpt2 were able to better identify void clauses achieving an f-1 score of 0.54 and 0.52. on both datasets, open models outperformed gpt-3.5 with regard to f1-score, which generated a huge amount of false positives leading to an f1-score of 0.11 on the more imbalanced and 0.22 on the less imbalanced dataset. false legal statements always have to potential to cause harm. while we worked with experienced experts who carefully decided on each annotation, it is always possible that the dataset contains errors. claiming falsely that a clause is void could potentially have negative impacts on a business, claiming falsely that a clause is valid could potentially have a negative impact on consumers. one measurement we took to avoid such impacts is anonymising the clauses, in order to make it harder to connect them with a specific business. many clauses, like severability or liability clauses, are highly standardised and often directly drawn from boilerplate contracts. in such cases removing the explicit identifiers is sufficient to make them completely anonymous. in other cases, particularly for example in clauses about bonus and rewards programs, even after removing the explicit identifiers, clauses can be so specific that they can be traced back to individual businesses, if they still use the same clause. we believe that the practical impact of that is rather limited, because retrieving the information about a specific clause from the dataset is not something that a significant number of potential customers is likely going to do. we added an explanation text to the readme of the published version of the corpus to highlight that the annotations are based on the perspective of individual experts and not based on a concrete court ruling and therefore not legally binding. overall we believe that the publication of the dataset can help to address an existing imbalance of power between customers and companies and thereby, together with the anonymisation strategy, warrants the small but existing risk. at the same time, being aware that a company uses potentially void clauses that disadvantage consumers and not doing anything about it could be seen as unethical. therefore, the ngos we worked with did take legal steps if they encountered clauses during the annotation that they deemed so disadvantageous for consumers that legal steps were necessary. in general, the dataset consists of publicly accessible data that is usually carefully drafted by lawyers. we did not encounter any instances within the dataset that we deem problematic from an ethical perspective.