Questions about Contracts: Prompt Templates for Structured Answer Generation (2023)

Abstract: Finding the answers to legal questions about specific clauses in contracts is an important analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. In this paper, we present an examination of using large language models to produce (partially) structured answers to legal questions; primarily in the form of multiple choice and multiple select. We first show that traditional semantic matching is unable to perform this task at acceptable accuracy and then show how question specific prompts can achieve reasonable accuracy across a range of generative models. Finally, we show that much of this effectiveness can be maintained when generalized prompt templates are used rather than question specific ones.

## introduction contracts and other legal agreements are complex documents that specify in a myriad of ways the obligations, restrictions, and other covenants between two or more parties. understanding such requirements for a single contract is a task that any lawyer could do if the need were to arise (e.g., whether a tenant can sublet an apartment), especially if there exist methods to automatically extract such clauses from documents @xcite . however, such a task becomes hard to scale when reviewing hundreds to thousands of documents in an efficient manner for different needs (e.g., determining market trends, due diligence) especially when there is a desire for granular answers rather than clauses (e.g., amounts of liability). granular answers to questions allow lawyers and other legal professionals the ability to triage the documents they need to read in order of risk, importance, or other criteria (e.g., first reviewing agreements that their client has provided unlimited liability coverage or most favoured na-tions guarantees). finding a solution to allow rapid triage and review of documents for such granular answers have yet to be solved to our knowledge and we show progress towards that goal in this work. the advent of and the increasing popularity of generative ai and applications to "chat" with one's documents (e.g., chatpdf foot_0 , docgpt foot_1 , casetext foot_2 ), which allows individuals to interact in a "natural" way with their documents and may facilitate a better understanding of legal obligations and restrictions, has complicated matters further due to the appeal of natural language interaction. this interaction can be convenient on a handful of documents but we believe that this approach also does not scale since "chatting" with a document means getting a human-like response that may be difficult to use in automated workflows without building complex post-processing systems (e.g., rules on how to respond to the various outputs). but added complexity means that further work is required either on the part of the end user or system builder to ensure that post processing can and does occur for outputs to be used as part of other pipelines. with these ideas in mind, this work investigates how generative ai can help generate structured and partially structured answers to questions about clauses in legal documents in a manner that is usable in automated workflows and data pipelines. to facilitate this process, we assume that we have a set (or sets) of clauses (potentially using ml to extract them) that a user would like to ask questions about in bulk with either predefined fixed options (i.e., multiple choice or multiple select) or text with a specific structure (e.g., lists of items, entities such as duration). such outputs can facilitate the creation of automated workflows that can easily take action on the results of the questions without resorting to the complex understanding of a generated answer or manual human review except at predefined points in the workflow (e.g., after documents have been ranked according to the risk associated with different questions and their answers). while such outputs are perhaps less compelling than having a "conversation" with a document, we believe that this approach scales better to large document collections, workflow automation, and general analysis of trends in contracts. moreover, we show that more natural prompts can very easily exhibit consistency and reliability issues when examining the responses generated in a "chat" interaction paradigm (section 3). it is tempting to not use llms to directly generate answers since there persist issues around hallucination @xcite and a lack of clarity around whether such models truly reason @xcite and instead use more traditional unsupervised techniques (e.g., embedding similarity). we show that such techniques are not well suited to this task, especially in light of how straightforward it is to get much better results with generative techniques (section 5.1). the downside to the generative approach is that it requires prompting engineering, a new and popular area of research @xcite @xcite and book publication foot_3 , but it is not a task that we expect lawyers or other legal professionals actually want to do on a regular basis. that is, we might expect that a lawyer would want to "chat" with a set of documents but they do not want to go to the extreme efforts to constrain outputs to a particularly useful format via prompt engineering. our end goal is to find and use prompts that provide consistent answers for the same clause and reliable answers across clauses (section 4). in particular, we are interested in finding reusable templates that allow end users to "fill in answer options" or "specify the structure of outputs" but not need to worry about overall prompt engineering (section 5.3). the ultimate benefit to this approach is that users of such a system would only need to worry about their area of domain expertise (i.e., determining appropriate answers) and that system builders would not need to foresee all possible different combinations of options or build bespoke prompts for every lawyer-posed question. ## open-ended response generation with the popularity of llm powered chat bots and conversational agents, it is easy to think that "chatting" with one's documents would be advantageous for lawyers and legal professionals. such an approach can work when dealing with one or two documents but when scaling to hundreds or thousands of documents this becomes untenable. this results in needing to triage and prioritize work to be done; that is, some documents are invariably more important than others. for example, a commercial real estate company may be concerned with environmental indemnification foot_8 present in their leases due to new legislation, their lawyer may seek to find which documents do not have such a clause, then documents where the tenant indemnifies the company, then where the real estate company indemnifies the tenant, and finally where this indemnification is mutual. if one can identify the presence (or absence) of such a clause then the lawyer simply needs to make a determination but, again, this does not scale effectively. we might reasonably wonder what happens if we were to ask a llm model about the direction of mutual indemnification in a sample clause (figure 1 ). if one prompts openai's gpt-3.5-turbo model (with temperature set to 0) with the following: in the clause below, who indemnifies whom? and then append the example clause, we get the following responses after regenerating a handful of times: while the first and third are close, they aren't identical and the second is more of an overall summary. moreover, these still require some amount of manual work to read and potentially process the (simplified) outputs. that is, a lawyer still reads a non-trivial amount for every clause found when they might prefer to direct efforts towards leases that do not indemnify their client. while something like nugget matching @xcite could help automate this process, this is also not something that we might reasonably expect a lawyer to do. to further illustrate this problem, we present the responses from the same prompt with a different omitted environmental indemnity clause: we see responses that are correct but are also quite different from the responses for the previous clause. as should be apparent by now, this inconsistency and unreliability as well as response verbosity is not ideal for simplifying work process and may just exacerbate things in the long run. the following sections detail our investigation into how well popular large language models can be prompted to consistently generate (partially) structured answers to questions based on contract clauses. ## structured answer generation to generate structured answers, one might reasonably turn toward a more traditional solution by training a discriminative multi-class (or multi-label) classifier to produce clear answers to the questions presented in this work. such solutions (e.g., svms, logistic regression) are well understood and highly optimized but come with a flaw, they require sufficient examples of each class to be effective (i.e., 9 https://www.sec.gov/edgar 10 https://www.sedarplus.ca/ low prevalence classes can be hard to overcome). for example, landlord indemnification is a relatively rare occurrence in our data and one for which data augmentation is not guaranteed to work. indeed, we have attempted to address this in prior work @xcite but the solution did not scale well for extremely low prevalence or nuanced differences between options. it has been our experience that such cases arise naturally in the legal domain where rare outcomes tend to be the most problematic and equally hard to identify at scale. accordingly, models that require little to no training data (or other labelled) examples become compelling solutions to this problem and we spend the rest of this section discussing our experiments. ## limitations and future work in this work, we focused on optimizing prompts for gpt-3.5-turbo for the sake of simplicity and cost. moreover, as there are increasing numbers of models being made available for commercial use @xcite @xcite , we felt that overall gpt-3.5-turbo provided a reasonable baseline due to it being an obvious "initial" starting point that one might use before jumping into fine-tuning an "open" model. the benefit to this was the somewhat surprising result from tables 2 and 3 is that prompts optimized for one llm can produce relatively competitive results (or exceed the base llm) with the same prompt. further exploration remains to determine how much benefit would be gained fine-tuning a smaller model, like llama2, given that the much larger models are "by default" apparently much better at our desired tasks. subsequent investigation into this is planned as it is far less cost efficient to fine-tune larger llms and more generally reducing model size has other social benefits (e.g., less carbon footprint). we have also largely operated under the assumption that an end user has the appropriate clause easily available about which they can ask questions. as retrieval augmented generation has become a popular topic of research @xcite , combining our approach with existing approach will invariably require examining how well these prompts and models deal with false positives. our prompts do have escape mechanisms and guard rails to hopefully prevent obvious errors from occurring but we leave investigation of how errors propagate in endto-end systems with our prompts to future work. ## conclusions we have presented a structured answer generation task based on identifying the correct answer for a legal question given an associated clause from a document, detailed some issues with relying solely on natural language question answering, and explore solutions using unsupervised and generative methods. we have shown that the prompt-based generative methods exceed semantic similarity-based approaches and that despite question specific prompts performing better, that generalized prompts can be used with little decrease in overall effectiveness. finally, partially structured answer generation is shown to be possible with bespoke prompts but generalized approaches do fare well due to the large amount of variance in the output formats that are required to accommodate the types of information desired.