Validity Assessment of Legal Will Statements as Natural Language Inference (2022)

Abstract: This work introduces a natural language inference (NLI) dataset that focuses on the validity of statements in legal wills. This dataset is unique because: (a) each entailment decision requires three inputs: the statement from the will, the law, and the conditions that hold at the time of the testator’s death; and (b) the included texts are longer than the ones in current NLI datasets. We trained eight neural NLI models in this dataset. All the models achieve more than 80% macro F1 and accuracy, which indicates that neural approaches can handle this task reasonably well. However, group accuracy, a stricter evaluation measure that is calculated with a group of positive and negative examples generated from the same statement as a unit, is in mid 80s at best, which suggests that the models’ understanding of the task remains superficial. Further ablative analyses and explanation experiments indicate that all three text segments are used for prediction, but some decisions rely on semantically irrelevant tokens. This indicates that overfitting on these longer texts likely happens, and that additional research is required for this task to be solved.

## introduction natural language inference (nli) in the legal domain has not been widely investigated, despite its importance and potential. one such important nli application is validity assessment of legal documents such as wills. legal procedures for creating and executing wills are evolving rapidly. processing a will via probate is a costly, time-consuming process that can be exacerbated by errors or by challenges to validity. these problems will likely increase as people increasingly employ electronic wills. developing natural language techniques that can determine a will's validity at creation and execution can increase validity, conserve legal resources, and effectuate the author's intent. to this end, this work explores how nli models can be employed to evaluate the validity of will statements. nli in general deals with determining whether a premise entails, contradicts, or is neutral to the hypothesis given. our project made two important changes to this approach to fit legal documents. first, our dataset contains three input types: statements from wills (as hypotheses), laws (as premises), and conditions, which are circumstances at the time of probate (e.g., eligibility of the testator or witnesses). this adaptation is necessary as will statements' validity cannot be evaluated without considering both circumstances at the time of probate ("conditions") and relevant legal rules ("laws"). also, we switched the labels from entailment, contradict, neutral to support, refute, unrelated to better represent the relation between will statements and laws. the major contributions of this work are: ## related work natural language inference (nli), also known as recognizing textual entailment (rte), determines entailment relations between a pair of sentences: a premise and a hypothesis. their relationship is either: a) entailment, if a premise entails a hypothesis; b) contradiction, if a premise contradicts a hypothesis; or c) neutral, if a premise neither entails nor contradicts a hypothesis. nli has been a key framework in natural language processing since @xcite proposed the rte challenge. numerous datasets exist for general nli tasks. for example, the stanford natural language inference (snli) corpus @xcite dataset contains about 570,000 pairs of sentences (premises and hypotheses) generated from flickr image captions. more recently, datasets for domainspecific nli tasks are being introduced. scinli @xcite is drawn from scientific texts. it contains 107,412 sentence pairs extracted from papers in natural language processing and computational linguistics. there have been a few nli-related resources for the legal domain, including iflylegal @xcite , statutory reasoning assessment (sara) dataset @xcite , graphbased causal inference (gci) framework @xcite , and contractnli @xcite . one iflylegal module was developed for natural language article inference; it provides relevant legal articles when asked legal questions. sara contains rules extracted from the us internal revenue code and natural language questions. gci generates causal graphs based on fact descriptions. though the framework was not specifically designed for law, @xcite demonstrated that it can be utilized for legal text analysis. con-tractnli is a dataset consisting of 607 legal contracts designed to handle document-level natural language inference. our work differs from these existing nli works in two ways: (a) it contains three types of input information; and (b) it operates in a pragmatic middle space with respect to text length: larger than datasets with sentence-level texts, which are insufficient to capture legal details, but shorter than document-level inference datasets. ## dataset our dataset includes three types of entries (will statements, applicable law, and facts representing external state) rather than the usual two types. validity often depends upon such external circumstances. thus, a will statement can be legally valid under some facts, but not others. hence, a will can change from valid to invalid or vice versa; wills must, at minimum, be evaluated when executed (drafted and signed) and when probated (where a court determines whether to implement its provisions). for example, if a will contains a statement appointing a specific person as executor, that person's eligibility must be verified. this requires information about both the law and the person's circumstances at probate time. for example, according to tennessee code section 40-20-115, any person who has been imprisoned cannot be an executor. thus, to assess eligibility of a tennessee executor in tennessee, one must know whether they have been in prison at probate time. ## future work this work can be expanded in several directions. first, it can be extended to cover multiple states by adapting the models or adding more data. future work can investigate if models trained on a single state's data can be adapted to evaluate data from other states. also, wills from multiple states could be added to the dataset. we expect the augmented dataset would enhance the models' performance on evaluating wills from other states. this work can be expanded to other legal domains. models trained on our data can be adapted to similar tasks in other legal domains. lastly, this work can be extended by investigating novel transformer models. given the uniqueness of our dataset with regard to the number of inputs and text length, it is likely that further experiment and modification is needed to handle such characteristics. ## conclusion this work presented an annotated dataset for natural language inference (nli) in the legal domain, consisting of 1,014 data points generated from 23 publicly available wills. the dataset is novel for two reasons: it included texts with three input types (statement, law, and condition) rather than two (premise and hypothesis) in the traditional nli, and included texts are longer than in general nli datasets. the nli models trained on our dataset showed reasonable performance in assessing the validity of will statements. ablative experiments demonstrated that the models' performances worsen if any input type (condition, law, or both) is omitted. this suggests that the models utilize all three input types. the lime implementation reveals that even the best performing model makes errors in some cases by using semantically irrelevant tokens. our open-access dataset is publicly available at: https://github.com/ml4ai/ nli4wills-corpus