Supervised Identification of Participant Slots in Contracts (2021)

Abstract: This paper presents a technique for the identification of participant slots in English language contracts. Taking inspiration from unsupervised slot extraction techniques, the system presented here uses a supervised approach to identify terms used to refer to a genre-specific slot in novel contracts. We evaluate the system in multiple feature configurations to demonstrate that the best performing system in both genres of contracts omits the exact mention form from consideration—even though such mention forms are often the name of the slot under consideration—and is instead based solely on the dependency label and parent; in other words, a more reliable quantification of a party’s role in a contract is found in what they do rather than what they are named.

## introduction contracts create a relationship between two parties, generally centered around some kind of stance object. for example, a non-disclosure agreement (nda) often features a disclosing party, a receiving party, and some sort of confidential information; similarly, a services agreement may feature a service provider, a service recipient, and some sort of services to be provided. in both types of contracts, the agreement will refer to itself as well. contracts contain language that is more repetitive than those typically studied in nlp research @xcite , but this is not necessarily consistent language. a disclosing party may be referred to as a "disclosing party," "discloser," "providing party," "company," "owner," "client," or any of a limitless set of possible referring expressions, including the actual company name of the disclosing party. further, some of these expressions have no particular relation to either party and may be used ambiguously throughout the corpusi.e. "company" may refer to the disclosing party in one contract and the receiving party in another. thus, while enumeration of such expressions may seem to present a viable heuristic for slot identification in contract language, a more robust technique is required. we avoid neural network methods for this task for a few reasons. results from @xcite suggest that pre-trained models trained on more general language struggle on the idiosyncrasies of contract language-at least those pretrained on non-legal data. further, a resourceintensive slot detection system is not scalable in the desired environment. this is a subprocess of a larger system @xcite , and thus the footprint in terms of memory and hardware expense need to remain low. we also want to build a system that would provide some transparency into its decisions so users may have an opportunity to customize the system to their needs and preferences. given that such an understanding of neural network methods remains an on-going research task, a method derived from older techniques promises better opportunities for inspection than techniques that may be more popular. for this, we look toward more linguisticallyinformed methods for identifying participant slots in language generally. @xcite 's model of script learning from episodic knowledge is a sensible theoretical fit for learning participant slots from contracts. just as scripts are presumably learned from exposure to repetitive sequences of events, contracts themselves of particular types are also repetitive series of propositions; an individual reading many contracts would readily observe patterns that would generalize into interchangeable participant slots and relationships among them. @xcite present a model for doing exactly that; however, their script learning method is unsupervised, while the sort of slots identified in this work have been prescribed in advance by subject matter experts. thus, in this model, we turn the techniques pre-sented by @xcite on their head to learn participant slots from supervised data. section (2) discusses prior work related to contract language and slot identification, particularly how this work exists at the boundary between two threads of thought on slot identification. section (3) outlines our annotation efforts, their results, and other nlp pre-processing necessary to extract slots. section (4) describes the mathematical underpinnings of our slot identification technique, and section (5) enumerates how those mathematical relations are turned into slot selections. section (6) describes the results of these experiments, and section (7) goes into more qualitative detail in understanding the causes of the quantitative results. ## prior work prior nlp work on contract language is extensive, including summarization @xcite information extraction and understanding @xcite @xcite , as well as corpus studies looking at intrinsic properties of contracts @xcite or providing new annotations over contract language @xcite . we were not able to identify any prior work specifically on participant slots in contracts. @xcite devised a system that identifies the obligations of parties in contracts, leveraging similar features as in this paper but with some key differences. participants are determined based entirely on their mention form-either overtly, or through a dictionary of synonyms. mention forms that fall outside of these are ignored, and this technique was not overtly evaluated. as their primary goal was to identify obligations and to look at large volumes of statistics about those obligations within a corpus of employment contracts, the mention form approximation is sensible to make. on the other hand, the goal of the system presented in this paper is to identify mention forms specifically. since this is part of a user-facing contract review system, ignoring forms that could not be determined in advance would fail to meet the needs of that system; as will be discussed in section (3.1), such unpredictable mention forms occur frequently. the theoretical foundation for slots comes from @xcite 's seminal work on scripts-generalizations of events from prior exposure to them. the classic example is that of a "restaurant script"-an understanding of the general series of events and participants involved in visiting a restaurant. while such a conception of knowledge is intuitive, the precise nature of how this knowledge should be realized in nlp systems falls into roughly two threads of thought: supervisedwhere such knowledge is explicitly enumeratedand unsupervised-where such knowledge is inferred from a corpus of text. while not exactly the same as scripts, some approaches attempt to model frames in a supervised manner-that is, rather than inferring them from natural language descriptions in corpora, projects such as framenet overtly name and describe structured frames and their participant slots @xcite . however, frames are not scripts or schemas; rather, frames are the building blocks scripts and schemas, which can be thought of as networks of frames with shared participants. even after years of annotation, the exhaustive enumeration of all frames seems unlikely. in attempting to understand language generally, a system may need to "possess many schema[s], perhaps hundreds of thousands" @xcite , which motivates an unsupervised approach to extracting such structures. @xcite re-ignited interest in related tasks, though most community efforts have centered around performance improvements on or variants of the narrative cloze task @xcite @xcite . other work has continued to focus on the extraction of schemas or scripts as discrete knowledge structures @xcite @xcite . latently, in the unsupervised learning of such schemas, participant slots are learned through the identification of sequences of events; in supervised techniques, slots are identified overtly and must be mapped on to text by some means. however, the unique properties of contracts allow us to execute at a hybrid approach. the system described in this paper has mathematical underpinnings that are very similar to those in unsupervised work, particularly @xcite . however, the overt annotation of a specific genre of contract is much more similar to a frame-based supervised approach, with a subject matter expert deciding in advance what the frame is and what the slots to be labelled are. a specific frame defines each genre of contract, composed of the relationship between the parties specified therein. table (1) enumerates this explicitly, juxtaposing the terminology used in our own annotation and that used in framenet. in an nda, that frame is framenet's exchange while a services agreement is a commercial_transaction. these differ because framenet requires as a core part of a commercial_transaction a money slot which is not often the case in an nda. ## data this section describes two components of our data preparation for slot extraction: the annotation which produced the system's training data (section 3.1) and and the other nlp pre-processing steps needed to learn and extract participant slots (section 3.2). ## slot scoring to decide which slot a particular chain might represent requires a technique for scoring each chain c with respect to a particular slot type k. this section describes the components of a function s which does exactly that. rather than considering all co-referring argument pairs in a chain, as @xcite did to assemble schemas, the system considers each link in a chain individually against each possible slot type. @xcite were using an unsupervised technique, and thus did not have the extra information of a particular identified slot type; in other words, the slot type was implied by a strong signal between co-referring event arguments. instead, this system associates the supervised slot class and features derived from each chain link. during training, each chain is then represented by tuples composed of the slot type k, the mention surface form m, the dependency relation r between the mention and its parent p, which is also represented by its surface form-i.e. k, m, r, p . in the case of a mention's immediate parent being a preposition-in a manner similar to the stanford typed dependencies' "collapsed dependencies" (de @xcite )-the relation is derived from the lemma of the preposition (e.g. "prep_in"); the parent is then the parent of the preposition. for example, from a document where "recipient" was annotated as receiving party and "sensitive documents" was annotated as confidential information, "the recipient shall destroy all copies of the sensitive documents" renders three tuples: the "none" class retains features to estimate independent probabilities. deviating from @xcite 's goals for slot extraction, the system skips the constraint of looking strictly at verbs to extract such relationships. while contracts may tell stories, they are not themselves intended as narrative texts. rather, it extracts k, m, r, p -tuples for every mention of an entity, regardless of what r is. in both training and slot identification, the system excludes any chains whose mention form is either "party" or "third party." this is because the meanings of these nouns are often conditioned on determiners and adjuncts with qualities that require more sophisticated handling of the coreference information entailed within them-e.g. "each party" vs "the party." these terms rarely point specifically and unambiguously to one party or the other, and thus do not help to determine the way a slot is referred to in a given text. from there, the system estimates probabilities as: pointwise mutual information (pmi) of a tuple t is defined as: the exact contents of t and, consequentially, the probabilities multiplied in the denominator will depend on the specific probability we are using. for example, for the p (k, r, p) model, pmi is realized as: to prevent rare items from being weighted highlyoften a flaw of pointwise mutual information-the system forces pmi to be 0.0 for joint items with counts less than six. when observing new items in a new document, for a given chain c, the system measures how well it fits with a particular class k using s: ## slot selection a score alone is not enough to decide which chains correlate with which slot. to decide which chains actually represent each slot, the system employs algorithm (1). this is composed of roughly two parts: a score for every chain as every possible slot and a selection procedure. the first loop scores every chain's association with every class; if a score is 0.0 or less, it is omitted from further consideration. these are sorted from highest score to lowest. the second loop, in the specified reverse score order, assigns chains to particular slots-only if the chain has not yet been assigned to a slot or the slot has not yet been assigned a chain. if a slot has no assignments made to it, it is left blank. ## results evaluation employs a holdout of 19 ndas and 18 services agreements against a total of 14 different models-7 for ndas and 7 for servicesrepresenting different ablations of the features con-sidered. 2 the first ablation predicts class based on only one feature: either the mention p (k, m), dependency relation label p (k, r), or dependency parent p (k, p). the second ablation predicts class based on pairs of features: mention and relation p (k, m, r), mention and parent p (k, m, p), and relation and parent p (k, r, p). the third ablation considers all features jointly, yielding one model p (k, m, r, p). at the first level of ablation, the mention-only model (p (k, m)) performs best on both ndas and services agreements. when other features are added to the mentions, however, the mention-2 a reviewer pointed out the limitations of this experimental configuration, as the absence of a third dev split lowers the rigorousness of these results. the absence of this further split is, in part, a consequence of the success of the system in production. there have been too few complaints from users about the slot system to justify further improvement. related models (p (k, m, p) and p (k, m, r)) generally decline in performance (nda recall, services precision and recall) or experience only small improvement (nda precision) in performance. however, when the two features from the first ablation that performed the most poorly are combined, they result in the highest performance over all, across both genres of contract. ## analysis the results in both genres show an interesting trend-when the values are considered independently, mention is the strongest feature in terms of performance. however, this changes in the second level, where mention being excluded and only the dependency-parent pairs of mentions (i.e. p (k, r, p)) results in the strongest performance. this outperforms the results in the third ablation, leading to a decline in precision and recall. these trends hold in both the nda and services genres of contracts, albeit to different degrees. the relationship between a candidate chain and the rest of the contract-as embodied through its dependency labels and parent-best embodies the slots, even more than the explicit mention of the label. though this was somewhat predictable from prior literature, it has not been explicitly tested. it is actually striking that, in a genre of text where the slot itself is so often explicitly named that performance is higher when that explicit naming is omitted, and the decision itself is only directly connected to how a particular entity participates in a contract rather than what it is called. among the top performing models, errors were generally concentrated in the terms used to refer to either one of the parties, especially in the nda results. examining the output more closely, mention form's harm to performance can be more readily teased apart. a particularly interesting nda, "software group inc. nda" actually expresses a four-part relationship between the state of minnesota, the state court administrator's office, a contractor, and another company tyler technologies. these last three are defined as "court," "recipient," and "tyler" respectively. the agreement seeks to protect, primarily, "tyler trade secret information," which our annotators marked as the term for confidential information, and since it revolves around tyler's trade secrets, "tyler" is the disclosing party; "recipient," in this case, is actually the receiving party. both the p (k, r, p) and p (k, m, r, p) models made errors on this document, but different errors that reveal some of their flaws. the p (k, m, r, p) model missed slots that included "tyler" in the name, as that explicit mention form was unpredictable in advance. instead, the model guessed nothing for the disclosing party and "information" for confidential information. on the other hand, the p (k, r, p) model was able to get both of those slots correct; however, it incorrectly guessed the other defined term "court" to be the receiving party. this is due to a use of "court" in a very similar manner to "recipient" and is expressed in relationships very similar to other chains in prior texts labelled receiving party. for example, in defining exclusions to the confidential information, the contract states (emphasis added): "tyler trade secret information" shall not include (i) any information which was known to or readily ascertainable by proper means by the court or recipient before being disclosed to the court or recipient by tyler; (ii) any information which is or becomes available to the general public without fault or action of the court or recipient... the relations expressed here for "court" are court, prep_by, ascertainable , court, prep_to, disclosed , and court, prep_of, action , all relations indicative of a receiving party while "recipient", in our model, only obtains three instances of recipient, conj, court . a more sophisticated handling of the dependency tree may rectify thisretrieving the proper semantic relationship for "recipient." above all, while these are errors, none of them are embarrassing errors; their causes are highly transparent given the design of the model and suggest opportunities for further improvement. in the services agreements, the error rate was higher for all models. this may, in part, be due to data sparsity, since far fewer services agreements were available for annotation. however, there were differences between the slots in each genre. ndas preferred more generic mention forms-e.g. "discloser"-while services agreements usually have at least one company name as one of the parties, so this hurt the p (k, m, r, p) model more than in the ndas where it labelled at least one slot wrong in all but one evaluation contract (17 of 18) because the mention contained a proper noun. the p (k, r, p) model also made some similar errors, but only in 6 of the 18 contracts. additionally, services agreements were more likely to have multiple mention forms for the same slot-for example, the "apple -developer agreement" uses both "you" and "apple developer" for service receiver. our algorithm is incapable of determining that both of those chains are part of the same slota limitation, but an intentional one, since opening up the possibility for slots to be multiple chains opens up tremendous room for error. ## conclusions this paper demonstrates a technique for identifying participant slots in ndas and services agreements. the mention form alone obtains high performance, but performance declines as other features are added. this is likely because the specific mention form makes it harder to capture the highly common phenomenon of referring to a slot by a party name rather than the name of the role. those features, dependency relation and dependency parent, on their own-excluding mention form-outperform mention form alone. this shows that what a participant does better reflects their role in a contract rather than the specific string used to refer to them. there are a number of improvements that could be made to future systems. it is possible that a model that considers the disjunction of the mention form and relation-parent pairs could perform even better than those attempted here, which are strictly joint models. this would more tightly conform to @xcite 's typed schemas system design. further annotation of services agreements would likely result in performance improvements, as well as better traversal of the parse tree, improved parsers for contract language, and architectures that allow for multiple mention form chains to be considered for a single slot to capture ambiguous cases.