Data-efficient end-to-end Information Extraction for Statistical Legal Analysis (2022)

Abstract: Legal practitioners often face a vast amount of documents. Lawyers, for instance, search for appropriate precedents favorable to their clients, while the number of legal precedents is ever-growing. Although legal search engines can assist finding individual target documents and narrowing down the number of candidates, retrieved information is often presented as unstructured text and users have to examine each document thoroughly which could lead to information overloading. This also makes their statistical analysis challenging. Here, we present an end-to-end information extraction (IE) system for legal documents. By formulating IE as a generation task, our system can be easily applied to various tasks without domain-specific engineering effort. The experimental results of four IE tasks on Korean precedents shows that our IE system can achieve competent scores (-2.3 on average) compared to the rule-based baseline with as few as 50 training examples per task and higher score (+5.4 on average) with 200 examples. Finally, our statistical analysis on two case categories — drunk driving and fraud — with 35k precedents reveals the resulting structured information from our IE system faithfully reflects the macroscopic features of Korean legal system.

## introduction legal practitioners often need to analyze a vast number of documents while preparing legal cases. for instance, finding appropriate precedents from ever-growing court decisions on previous cases can be challenging due to its number, while they are critical for decision making on subsequent legal actions. although legal search engines based on both lexical and semantic similarity can dramatically decrease the burden, retrieved texts are still unstructured and require further reading. furthermore, statistical analysis of legal documents is impossible without additional structuralization. such statistical analysis from a vast amount of documents, if possible, may help decreasing implicit bias on judicial decision @xcite . however, structuralizng legal documents is very challenging due to their diversity as they reflect virtually all social phenomena. this makes building a comprehensive ontology and ie system very demanding. instead of building a single perfect complex ontology and corresponding ie system, one can resort to building a task-specific ie system focusing on small number of relevant target information at each task. however, the number of required ie systems will quickly grow together with their development and maintenance cost due to the need of task-specific engineering. the cost from the taskand domain-specific engineering can be reduced if end-to-end systems based on generative models are employed but such systems are often unstable and require a large amount of training data. in this study, we try to answer the following questions. (1) is it possible to build an end-to-end (generative) neural system for legal information extraction showing high precision? if so, (2) how many training examples will be required? (3) what would be the best model architectures for this? (4) would prompt-tuning be more efficient than fine-tuning? to answer these questions, we perform experiments using various language models with different model sizes and architectures on four ie tasks on korean precedents. we show that the resulting end-to-end system (isla 1 ) can achieve competent or better performance compared to the rule-based baseline with only 50 training examples per task. with 200 training examples, isla achieves up to +27% f 1 compared to the baseline. using isla, we, for the first time, perform a large scale statistical analysis of "drunk driving" (24k samples) and "fraud" (11k samples) cases from korean criminal trials. the results show that the structured information by our ie system faithfully reflects the macroscopic features of the korean legal system. our contribution can be summarized as below. ## related works many previous methods on legal ie tasks are based on tagging (classification) approach @xcite @xcite @xcite @xcite . the brief description of individual works are presented in appendix a.1. compared to these studies, we map all ie tasks into a text-to-text format @xcite . this end-to-end approach removes the burden of task-and domainspecific engineering and is known to show competent or better accuracy compared to tagging based method with enough amount of training examples @xcite . @xcite also develop end-to-end ie system to extract information from four types of portuguese legal documents. compared to this study, we focus on building data-efficient end-to-end system combining both prompt-and fine-tuning methods. in this context, we investigate the effect of scaling model size, pre-training corpus, and training examples. domain-wise, we focus on korean precedents exclusively. lastly, we show how end-to-end ie system can be applied for statistical legal analysis by analyzing 35k korean precedents. ## tasks we formulate all ie tasks as text generation where a model needs to generate the values of target fields. we prepare four ie tasks over korean precedents, (1) three tasks from the facts and (2) one task from the rulings. only the precedents from criminal trials of 1st level courts are used. in the facts ie tasks, a model needs to extract a subset of legally important information from the factual description of cases. we consider three case categories, (1) drunk driving, (2) embezzlement, and (3) fraud. tasks were selected to be composed of various difficulty levels. in drunk driving, a model extracts "blood alcohol level", "travel distance", "type of the vechicle", and "previous criminal record on drunk driving". in embezzlement, a model needs to extract the loss from embezzlement. finally, in fraud, a model needs to extract the loss from fraud, or from aiding and abetting fraud. although similar to embezzlement, the facts from fraud cases show more diverse patterns which is reflected in their number of unique words ( ## models we use language models combined with taskspecific prompts. for a given source text (facts or ruling) and corresponding task-specific prompt, a model generate the values of individual fields autoregressively. see section a.3 for the details. for the comparison, we develop a rule-based baseline using regular expression. for each of the categories (drunk driving, embezzlement, fraudand ruling-criminal), we read through diverse cases and manually identified suitable identification rules/patterns. see section a.4. ## experiments all experiments are performed on nvidia a6000 gpu or rtx6000 using transformers @xcite and openprompt @xcite libraries. we use precedent corpus (150k korean precedents) from lbox open @xcite for the pre-training/domain-adaptation. see section a.5 in appendix for the details. ## results in this section, we summarize our experimental result on four legal ie tasks with varying difficulties (section 3). we examine 12 models while changing "model architecture", "model size", "the size of legal corpus for pre-training", or "the size of training dataset" ( pre-training with legal corpus is critical. the use of legal corpus greatly improves the accuracy across all tasks for both gpt2-base (+5.6 f 1 on average, 1st vs 4th rows, first column) and mt5-small (+7.3 f 1 on average, 2nd vs 8th rows). notably, on loss field of fraud task, gpt2-base shows +20.2 f 1 and mt5-small shows +21.9 f 1 (11th column) results highlighting the importance of pre-training with domain-specific corpus for ie tasks. ## analysis in this section, we report the result of statistical legal analysis on two case categories, drunk driv-ing and fraud. using isla, we first extract information from facts and rulings. for high precision, we control recall rate based on the model confidences that are computed by averaging the probabilities of the generated tokens (appendix a.7). we analyze 24,230 confident cases out of 33,554 in drunk driving task, and 10,898 confident cases from 15,106 in fraud task. with the structured data in hand, we first analyze drunk driving cases (fig. 1 ). two things are noticeable; (1) the average imprisonment period increases since 2019 (a); (2) the people with previous drunk driving record are sentenced longer imprisonment (b, c) regardless of their bacs (d). first result may be related to the fact that the korean government has strengthened punishment on drunk driving since jun 25, 2019 2 . the second result may reflect the article 148-2 (1) of the road traffic act which subjects repeat offenders to aggravated punishment 3 ( ## conclusion we develop a data-efficient end-to-end ie system for legal statistical analysis. we show that the system can achieve competent performance with small number of examples and exceed the rule-based baseline by large margin upon scaling model size, pre-training corpus, and training examples. the statistical analysis on 35k precedents reveals the resulting structured data faithfully reflect the macroscopic features of the korean legal system. stitutional (constitutional court of korea 2019hun-ba446, 2020hun-ka17, 2021hun-ba77 (consolidated) ruled on nov 25, 2021; https://law.go.kr/lsw/detcinfop.do?@xmath1://sc.scourt.go.kr/sc/krsc/criterion/ criterion_10/fraud_01.jsp ).