Development of an Enterprise-Grade Contract Understanding System (2021)

Abstract: Contracts are arguably the most important type of business documents. Despite their significance in business, legal contract review largely remains an arduous, expensive and manual process. In this paper, we describe TECUS: a commercial system designed and deployed for contract understanding and used by a wide range of enterprise users for the past few years. We reflect on the challenges and design decisions when building TECUS. We also summarize the data science life cycle of TECUS and share lessons learned.

## introduction a contract is an agreement between businesses and/or individuals to create mutual obligations enforceable by law @xcite . written contracts are also used by companies to safeguard their resources and as such, legal advice is sought prior to participating in a binding contract. currently, legal review remains an arduous and expensive process. for instance, a procurement contract requires 5 hours of legal review on average, contributing to thousands of dollars in total cost @xcite . while contract reviewing is a well-established legal process, building an enterprise-grade system for contract understanding (cu) to facilitate this process poses three major challenges: c1: model cu as an nlp problem. cu does not have a corresponding standard nlp definition. c2: lack of representative data. contracts, while often proprietary, also vary significantly across domains and businesses. thus, one cannot assume the presence of representative contracts towards building models. moreover, subject matter experts (smes) qualified to label ground truth are expensive foot_0 . as such, nlp models may need to be developed with limited non-representative labeled data but still be able to generalize well over previously unseen data. c3: need for model stability. cu models are integrated into existing business processes to drive decisions. as the models evolve over time (e.g. due to availability of new data, updates to existing labeled data, etc.), users expect the models to behave in a stable manner and produce no surprising results @xcite . problem example task concepts sample supported question "where will disputes regarding this agreement be settled?" (location) "is there an additional charge for services and utilities?" (pricing & taxes) "what is the effective & termination date of this agreement?" (effective date, termination date) "is tenant allowed to install additional fixtures?" (right -tenant) to overcome the above challenges, we designed and developed the transparent and expert contract understanding system (tecus), a commercial system that enables legal professionals to review contracts with minimal effort. tecus first models cu as a series of text classification and extraction tasks, defined collaboratively with smes, to capture the information that legal experts seek when reviewing contracts. second, it leverages systemt, a state-of-the-art declarative text understanding engine for the enterprise @xcite , towards developing transparent models on top of syntactic and semantic linguistic features, to mitigate a possible lack of representative labeled data and to satisfy model stability requirements. this approach enables (1) the development of stable models that explicitly capture domain knowledge without requiring large amounts of labeled data or representative samples; and (2) a data science workflow that supports systematic error analysis and incorporation of user feedback towards continuous model improvement (section 3). tecus is available as part of multiple commercial products including ibm watson ® discovery (ibm, a) and ibm watson ® compare and comply foot_1 . as part of these products, it has been in use by enterprise customers since 2017 to support a variety of contract understanding use-cases ( in addition to assisting in the understanding of a single contract, as described in this paper, tecus also allows legal professionals to compare two contracts, identifying similarities and differences along multiple dimensions; another critical task in the contract reviewing process. tecus models this problem as a clause-level comparison problem, identifying (i) clauses that are identical between two contracts, (ii) clauses that are on the same topic but have changed, and (iii) clauses that appear in one contract but not in the other. the comparison component, similar to the contract understanding component, leverages syntactic and semantic linguistic features provided by systemt and an associated data science workflow tuned towards a systematic and stable model development. however, for space reasons, this work focuses on single contract analysis, enabled by tecus' contract understanding (cu) component. ## modeling cu as an nlp problem working with legal experts, we first define the cu problem as a combination of multi-class multi-label foot_2 classification and entity extraction tasks, as depicted in figure 1 . fying, classifying and reviewing individual clauses to spot potential risks. for example, the sentence "purchaser will purchase the assets by a cash payment of four hundred fifteen thou-sand us dollars." a consistent ontology (shown in figure 1 ) was defined in the early stages of the project, in collaboration with smes via an iterative process, and reflecting the prevailing views of legal experts. however, to also accommodate users who adopt slight variations of the definitions (which we discovered can be common due to the subtle nature of legal terms), users can also customize tecus through user feedback, as described in section 3.3. attribute and metadata extraction. in addition to classifying elements, legal teams are also interested in extracting entities of particular importance to corporate law. these fall into two categories: ## system overview as can be seen in figure 2 , tecus consists of three main components, which we subsequently describe in detail: runtime, model building, and feedback incorporation. ## results & discussion we next present evaluation results, showing how tecus addresses the challenges discussed above. precision recall jan 59.79 25.33 … july 64.7 59.96 aug 71.53 67.41 sep 73.5 68.09 jan … july aug sep precision recall human performance improvements observed b a by company x leveraging global model improvements improvements specific to company x based on their specific feedback ## conclusion we have presented tecus, a commercial system that effectively assists and supplements legal experts in understanding and reviewing contracts. tecus' effectiveness is based on (a) the transparent nature of the cu model, comprised of linguistic logical expressions on top of srl, which in turn enables @xcite a systematic data science workflow towards swift yet stable model development. this leads to models that can be developed with limited, non-representative labeled data and remain stable and predictable over time; traits that are essential not just in the contract understanding domain but the wider legal domain as well. finally, while the system was developed for the cu problem, we believe that its design and associated insights could inform efforts in other areas that pose similar requirements of generalizability and stability.