What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions (2023)

Abstract: Reviewing and comprehending key obligations, entitlements, and prohibitions in legal contracts can be a tedious task due to their length and domain-specificity. Furthermore, the key rights and duties requiring review vary for each contracting party. In this work, we propose a new task of party-specific extractive summarization for legal contracts to facilitate faster reviewing and improved comprehension of rights and duties. To facilitate this, we curate a dataset comprising of party-specific pairwise importance comparisons annotated by legal experts, covering ~293K sentence pairs that include obligations, entitlements, and prohibitions extracted from lease agreements. Using this dataset, we train a pairwise importance ranker and propose a pipeline-based extractive summarization system that generates a party-specific contract summary. We establish the need for incorporating domain-specific notions of importance during summarization by comparing our system against various baselines using both automatic and human evaluation methods.

## introduction a contract is a legally binding agreement that defines and governs the rights, duties, and responsibilities of all parties involved in it. to sign a contract (e.g., lease agreements, terms of services, and privacy policies), it is important for these parties to precisely understand their rights and duties as described in the contract. however, understanding and reviewing contracts can be difficult and tedious due to their length and the complexity of legalese. having an automated system that can provide an "at a glance" summary of rights and duties can be useful not only to the parties but also to legal professionals for reviewing contracts. while existing works generate section-wise summaries of unilateral contracts, such as terms of services @xcite or employment agreements (balachandar 1 data available at http://bit.ly/legal-importance figure 1 : an illustration of input contract with a party (tenant) and output from contrasum. it first identifies sentences in a contract that contains any of the obligations , entitlements , and prohibitions specific to the party. then, ranks the identified sentences based on their importance level (darker the shade, higher the importance) to produce a summary. bt: bradley-terry. @xcite , as well as risk-focused summaries of privacy policies @xcite , they aim to generate a single summary for a contract and do not focus on key rights and duties. however, we argue that a single summary may not serve all the parties as they may have different rights and duties @xcite (e.g., a tenant is obligated to timely pay the rent to the landlord whereas, the landlord must ensure the safety and maintenance of rented property). further, each right or duty may not be equally important (e.g., an obligation to pay rent on time (otherwise pay a penalty) is more important for tenant than maintaining a clean and organized premise) and this importance also varies for each party. to address this, we introduce a new task of party-specific extractive summarization of important rights and duties in a contract. motivated by the different categories in the software license summaries available at tl;drlegal foot_0 describing what users must, can, and cannot do under a license, we define rights and duties in terms of deontic modalities @xcite @xcite : obligations ("must"), entitlements ("can"), and prohibitions ("cannot"). in this work, we refer to these as (modal) categories. existing summarization systems perform poorly on legal contracts due to large compression ratios and the unavailability of public datasets @xcite that are large enough to train neural summarization systems @xcite . while previous studies have either devised unsupervised methods to identify obligations, entitlements, and prohibitions for a party in a contract @xcite or collected annotations for the same @xcite , they do not consider the relative importance of instances within each category. although existing unsupervised summarization methods rely on sentence centralitybased importance measures @xcite , capturing sentential meaning @xcite , or analyzing the relationship between sentences to capture the structure of document @xcite , these methods fail to capture the legal-domain-specific notion of importance among sentences. to address these issues, we construct a legal-expert annotated dataset of party-specific pairwise importance comparisons between sentences from lease agreements. we focus on lease agreements, which have clearly defined parties with asymmetric roles (tenant, landlord), and for which modal category labels have previously been collected @xcite . we break down the contract-level task of partyspecific summarization into two sentence-level subtasks: (1) content categorization -identifying the modal categories expressed in the sentences of a contract for a specified party, and (2) importance ranking -ranking the sentences based on their importance for a specified party. this approach has three benefits: (a) enabling us to use an existing corpus for identifying deontic modalities in contract, (b) cognitively simplifying the annotation task for experts who only need to compare a few sentences (resulting in higher agreement) at a time rather than read and summarize a full contract (spanning 10 -100 pages), and (c) reducing the cost of annotation as a contract-level end-to-end summarization system requires more data to train than does a sentence-level categorizer and ranker. this work makes the following contributions: (a) we introduce a new task of party-specific extractive summarization of important obligations, entitlements, and prohibitions in legal contracts; (b) we are the first to curate a novel legal expert annotated dataset ( §4) (using best-worst scaling) consisting of party-specific pairwise importance comparisons for sentence pairs (that include obligations, entitlements, or prohibitions) from lease agreements; and (c) we train a pairwise importance ranker using the curated dataset to build a pipeline-based extractive summarization system ( §5) for the proposed task, and show the effectiveness of the system as compared to several unsupervised ranking-based summarization baselines, under both automatic and human evaluations ( §8); underscoring the domainsensitive nature of "importance" in legal settings. ## task definition we formally define the new task as: given a contract c consisting of a sequence of sentences (c 1 , c 2 , . . . , c n ) and a party p , the task is to generate an extractive summary s consisting of the most important m o obligations, m e entitlements, and m p prohibitions (where m @xmath0, e, p}) for the specified party. as previously mentioned, we break this task into two sub-tasks: (1) content categorization to identify the categories expressed in a sentence of a contract for a given party, and (2) importance ranking to rank the sentences based on their importance to a given party. ## dataset curation the lexdemod dataset @xcite ) contains sentence-level, party-specific annotations for the categories of obligations, entitlements, prohibitions, permissions, non-obligations, and nonentitlements in lease agreements. however, the data does not contain any importance annotations. to facilitate importance ranking, we collect partyspecific pairwise importance comparison annotations for sentences in this dataset. we only consider the three major categories: obligations, entitlements foot_1 , and prohibitions. ## contrasum: methodology we build a pipeline-based extractive summarization system, contrasum (figure 2 ), for the proposed task. it consists of two modules for the two sub-tasks: content categorizer ( §5.1) and importance ranker ( §5.2). the content categorizer first identifies sentences from a contract that mention any of the obligations, entitlements, or prohibitions for a given party. then, the identified sentences per category for a given party are pairwise ranked for importance using the importance ranker. from all the pairwise comparisons, a ranked list of sentences is obtained using the bradley-terry model (bradley and @xcite to produce a final party-specific summary consisting of most important m o obligations, m e entitlements, and m p prohibitions. ## experimental setup contrasum is a pipeline-based system consisting of two modules. the two modules are trained (and evaluated) separately and pipelined for generating end-to-end summaries as described in §5.3. ## evaluation we perform an extrinsic evaluation of contra-sum to assess the quality of the generated summaries as well as an intrinsic evaluation of the two modules separately to assess the quality of categorization and ranking. intrinsic evaluation of the categorizer and the ranker is done using the test sets of the datasets ( §6) used to train the models. although the size of datasets used to evaluate the two modules at a sentence-(∼1.5k for categorizer) or sentence pair-level (∼130k for ranker) is large, it amounts to 5 contracts for the extrinsic evaluation of contrasum for the end-to-end summarization task at a contract-level. since the number of contracts is small, we perform 3-fold validation of contrasum. each fold contains 5 contracts (in the test set) sampled from 11 contracts (remaining contracts are used for training the modules for each fold) for which both category and importance annotations are available to ensure the availability of reference summaries. contrasum uses the best categorizer and the best ranker (roberta-l models) trained on each fold (see evaluation measures we report macroaveraged precision, recall, and f1 scores for predicting the correct categories or importance order for the content categorizer and importance ranker. we also report the accuracy of the predicted labels. as both the reference and predicted summaries are extractive in nature, we use metrics from information retrieval and ranking literature for the end-to-end evaluation of contrasum. we report precision@k, recall@k, f1@k (following @xcite , mean average precision (map), and normalized discounted cumulative gain (ndcg) (details on formulae in a.3) computed at a category-level for each party; these scores are averaged across categories for each party and then across parties for a contract. we believe these metrics give a more direct measure of whether any sentence is correctly selected and ranked in the final summary. for completeness, we also report rouge-1/2/l scores in ## results and analysis automatic evaluation of the categorizer and the ranker we report the evaluation results for the ranker in ## conclusions we introduced a new task to extract party-specific summaries of important obligations, entitlements, and prohibitions in legal contracts. obtaining absolute importance scores for contract sentences can be particularly challenging, as we noted in pilot studies, thus indicating the difficulty of this task. instead, we collected a novel dataset of legal expert-annotated pairwise importance comparisons for 293k sentence pairs from lease agreements to guide the importance ranker of con-trasum built for the task. automatic and human evaluations showed that our system that models domain-specific notion of "importance" produces good-quality summaries as compared to several baselines. however, there is a large gap between the performance of contrasum and the upperbound, leaving scope for future work including the generation of abstractive summaries.