Information Extraction from Legal Wills: How Well Does GPT -4 Do? (2023)

Abstract: This work presents a manually annotated dataset for Information Extraction (IE) from legal wills, and relevant in-context learning experiments on the dataset. The dataset consists of entities, binary relations between the entities (e.g., relations between testator and beneficiary), and n-ary events (e.g., bequest) extracted from 45 legal wills from two US states. This dataset can serve as a foundation for downstream tasks in the legal domain. Another use case of this dataset is evaluating the performance of large language models (LLMs) on this IE task. We evaluated GPT-4 with our dataset to investigate its ability to extract information from legal wills. Our evaluation result demonstrates that the model is capable of handling the task reasonably well. When given instructions and examples as a prompt, GPT-4 shows decent performance for both entity extraction and relation extraction tasks. Nevertheless, the evaluation result also reveals that the model is not perfect. We observed inconsistent outputs (given a prompt) as well as prompt over-generalization.

## introduction wills are crucial legal documents that enable individuals to maintain control over their assets and ensure that their wishes are carried over after their death. unlike most legal documents, wills are commonly written by/for lay people. given the importance and ubiquity of wills, formalizing and developing legal procedures for understading, evaluating and executing wills is crucial. extracting key information from legal wills is a prerequisite for understanding, evaluating and executing wills. information extraction (ie) from legal wills serves as a foundation for many downstream tasks such as automatic will review and electronic will creation and execution. beyond social impact, will understanding is an important application of artificial intelligence (ai) for several reasons. first, will execution is an excellent use case for the flourishing smart contract platforms @xcite . moreover, closer to natural language processing (nlp), information extraction from wills can serve as an important evaluation platform for large language models (llms). lastly, given that most wills are not publicly available, foot_0 evaluating llms on ie from wills is less likely to suffer from contamination @xcite than other nlp tasks. to enable such use cases, we introduce a dataset for ie from legal wills, and pertinent in-context learning experiments on this dataset. in particular, the main contributions of this work are as follows: ## evaluation our dataset can be used to evaluate llms' performance on ie task. to this end, we used gpt-4 foot_5 with in-context learning. when evaluating the model, we restricted the range of the task to extracting the four most-common entities (testator, beneficiary, asset, will) and the four mostcommon relations (testator-beneficiary, testatorasset, beneficiary-asset, testator-will) to reduce the complexity. our dataset was randomly split into four sets: a training set (30 tennessee wills), a development set (5 tennessee wills), a test set (5 tennessee wills), and an out-of-domain (ood) set (5 idaho wills). the training set served as an example pool for developing a prompt, and the development set was used to optimize the prompt. the evaluation was conducted on the test set and the ood set. we treated the wills from idaho as the ood set given the differences between the wills from tennessee and the wills from idaho. we evaluated the model in an in-context learning setting where only a prompt with a few examples is given without fine-tuning the llm. the prompt utilized in the evaluation was manually crafted using the training set and development set. the prompt consists of two parts: instruction and example. in the instruction part, a detailed instruction regarding the ie task is provided. in the example part, examples of human annotations are given to offer further guidance. after the instruction and the examples are provided, a will statement on which the model should perform the ie task is given with a brief instruction. for the relation extraction task, a set of entities extracted from the given will statement (by human annotators) are also provided. see appendix d for the actual prompts used in our study. the llm outputs were compared with the human annotator's annotations. for the entity extraction task, we used exact text match as an evaluation metric, i.e., if the entire span of the extracted text does not exactly match the human annotation, it is considered incorrect. for the relation extraction task, we considered an extracted relation correct when any two mentions of the entity arguments match exactly and the correct type is produced. in many cases, gpt-4 extracted multiple instances of a single relation, creating a large set of duplicates. these duplicates were ignored and removed before the evaluation. ## results it is also noticeable that gpt-4 performed better on the ood data than on the in-domain test data. this result suggests that the prompts created from our training partition can be transferred and used to extract information from ood data (e.g., legal wills from the us states other than tennessee). ## discussion the evaluation result shows that gpt-4 is capable of handling the legal ie tasks without given training nor fine-tuning. however, the error analysis revealed two major issues of gpt-4's performance on ie tasks in a few-shot setting, which are 1) inconsistency and 2) overgeneralization of the prompt. ## conclusion we present an ie dataset consisting of 16,018 annotations from 45 legal wills. this dataset is valuable to both legal domain and ai domain, as it can serve as a foundation for several downstream tasks concerning legal wills. the dataset can also serve as an important evaluation platform for llms. we evaluated gpt-4 with in-context learning on our dataset, and demonstrated that the model can handle legal ie tasks reasonably well. however, the error analysis revealed two major issues of the gpt-4 in in-context learning setting, which are inconsistency and prompt overgeneralization. further investigation is needed to mitigate these issues and enhance the performance of gpt-4 in in-context learning setting. our open-access dataset is publicly available at: https://github.com/ml4ai/ie4wills/