Transferring Legal Natural Language Inference Model from a US State to Another: What Makes It So Hard? (2023)

Abstract: This study investigates whether a legal natural language inference (NLI) model trained on the data from one US state can be transferred to another state. We fine-tuned a pre-trained model on the task of evaluating the validity of legal will statements, once with the dataset containing the Tennessee wills and once with the dataset containing the Idaho wills. Each model’s performance on the in-domain setting and the out-of-domain setting are compared to see if the models can across the states. We found that the model trained on one US state can be mostly transferred to another state. However, it is clear that the model’s performance drops in the out-of-domain setting. The F1 scores of the Tennessee model and the Idaho model are 96.41 and 92.03 when predicting the data from the same state, but they drop to 66.32 and 81.60 when predicting the data from another state. Subsequent error analysis revealed that there are two major sources of errors. First, the model fails to recognize equivalent laws across states when there are stylistic differences between laws. Second, difference in statutory section numbering system between the states makes it difficult for the model to locate laws relevant to the cases being predicted on. This analysis provides insights on how the future NLI system can be improved. Also, our findings offer empirical support to legal experts advocating the standardization of legal documents.

## introduction this study investigates whether a legal natural language inference model trained on the data from one us state can be transferred to different us states. natural language inference (nli) is a textual reasoning task determining whether a premise entails, contradicts, or is neutral to a hypothesis. kwak et al. (2022) suggests that the validity assessment of legal documents (e.g, wills) can be framed into a nli task. it adapts the traditional nli approach to fit in the legal domain. unlike the general setting where a model gets two inputs (i.e., premise and hypothesis), the legal nli model requires three inputs: a legal document of which validity would be assessed, a condition (a circumstance relevant to the validity assessment of the legal document), and a law. this is an intriguing adaptation, but it also introduces the potential of overfitting on the law texts. developing legal nli models that can evaluate the validity of legal documents provides much benefit to legal professionals and anyone involved in writing such legal documents. assessing the legal documents' validity through legal nli models can reduce the time and resource required for review. it can also increase the validity of legal documents by preventing any errors at creation time. in addition, legal nli models can serve as foundations for downstream tasks such as legal document review automation, electronic will system, and smart contract. as us states have different legal systems, there is no guarantee that a legal nli model trained on one state would work in another state. however, training a model with data from all us states is a time and labor intensive task. further, it is often the case that we have access to data from only a few states. given these practical difficulties, transferring a model trained on one us state to other states is a great alternative to training a model on every state from scratch. to this end, we explore whether a legal nli model trained on one us state is transferable to another state. we also conduct error analysis to identify any pitfalls in transferring a model trained on one state to another. we focus on a specific task in this study to evaluate a validity of a legal will statement. we chose to focus on this task for two reasons. first, probate code is one of the fields where us states are divided into two groups: states which have adopted uniform probate code (upc) in full vs. states which have not adopted the full upc. this clear division makes a good testing case for our study. we tested if a model trained on the data from a state belonging to the former group (i.e., idaho) works for the data from a state belonging to the latter group (i.e., tennessee), and vice versa. another reason is that this task is highly practical. wills are important legal documents that allow people to maintain control over their assets. unlike most legal documents, wills are commonly written by/for people without legal training. however, will execution/probate is not always straightforward, and there is a risk that mistakes in will writing or execution procedure invalidate parts of the will. developing a model that can evaluate the validity of a will statements can help preventing such mistakes. in addition, as previously mentioned, the model can serve as a foundation for developing electronic will system. given all the benefits that the model could provide, we decided to conduct our study on this task. the main contribution of our study is as follows: ## methods to figure out if a legal nli model trained on a single us state can be transferred to other states, we compared a model's performance on the indomain data (i.e., the data which is from the same state as the model is trained on) and its performance on the out-of-domain data (i.e., the data which is from a state other than the one the model is trained on). we fine-tuned a pre-trained model on a task of evaluating the validity of legal will statements, once with the dataset containing the tennessee wills and once with the dataset containing the idaho wills. the format of the datasets is distinct from the traditional nli datasets in that it contains three types of inputs (statement, condition, and law) rather than two (premise and hypothesis). statement is a sentence or a short paragraph excerpted from a legal will. its legal validity is assessed based on the other two input types. condition is a circumstance relevant to the evaluation of validity of a will statement. lastly, law is a state provision that the will statement's validity is assessed on. these inputs are labeled as either support, refute, or unrelated, depending on whether the condition and the law support, refute, or are unrelated to the will statement given. the for further details on the datasets, see the section 3.1. the model we fine-tuned with our dataset is roberta-large-mnli (liu et al., 2019). for the implementation of the fine-tuning, see the section 3.2. we used the fine-tuned models to predict both the in-domain testing data (i.e., tennessee model predicts tennessee data and idaho model predicts idaho data) and the out-of-domain data (i.e., tennessee model predicts idaho data and idaho model predicts tennessee data). to better understand the models' behaviors, we generated confusion matrices with the models' predictions and conducted error analysis by using local interpretable modelagnostic explanations (lime; ribeiro et al. 2016). ## results to figure out the cause of the performance drop, we generated confusion matrices with both models' predictions in the out-of-domain setting. figure 1 shows the confusion matrix generated with tennessee models' predictions on idaho data. it is noticeable that the majority of the model's errors (207/247) is from incorrectly predicting support (116/247) or refute cases (91/247) as unrelated cases. figure 2 presents the confusion matrix generated with idaho models' predictions on tennessee data. this time, more than half of the errors (103/173) originated from incorrectly predicting unrelated cases as support (69/173) or refute cases (34/173). ## error analysis we identified two major error types by observing the confusion matrices. the first is that the tennessee model wrongly predicts support or refute cases as unrelated when predicting idaho data (207/247). the second is that the idaho model wrongly predicts unrelated cases as support or refute cases (103/173). we focus on analyzing these two error types in this section. ## conclusion this study found that legal natural language inference model trained on the data from one us state can be transferred to another state, but the performance deteriorates considerably. the f1 scores of the tennessee model and the idaho model are 96.41 and 92.03 when predicting the data from the same state, but they drop to 66.32 and 81.60 when predicting the data from another state (i.e., tennessee model predict idaho data and vice versa). our error analysis found two major causes for the performance drop. first, we found that the model struggles with identifying the equivalent laws across states when there are stylistic differences (e.g., terms, formats, capitalization, and etc.) between laws. we also found that difference in statutory section numbering system between the states makes it harder for the model to locate laws relevant to the cases that they are predicting on. this analysis gives hints for the design of the future legal nli systems. for instance, it is expected that including a knowledge base with equivalent laws between states would enhance the model's performance on predicting the cross-state data. the knowledge base should help with the normalization of statutory section numbers across the states. normalization of the law text formats (e.g., capitalization) would also improve the model's performance on the data from across the states. the analysis also provides empirical support to legal experts for the argument that legal documents such as wills should be standardized. as illustrated by our analysis, differences in legal documents' texts or formats between the states can easily interfere with the model's performance. standardization of legal documents would facilitate the development of legal nli system that can work across the states. our open-access dataset and source code are publicly available at: https: //github.com/ml4ai/nli4wills-corpus