NESTLE : a No-Code Tool for Statistical Analysis of Legal Corpus (2024)

Abstract: The statistical analysis of large scale legal corpus can provide valuable legal insights. For such analysis one needs to (1) select a subset of the corpus using document retrieval tools, (2) structure text using information extraction (IE) systems, and (3) visualize the data for the statistical analysis. Each process demands either specialized tools or programming skills whereas no comprehensive unified “no-code” tools have been available. Here we provide NESTLE, a no-code tool for large-scale statistical analysis of legal corpus. Powered by a Large Language Model (LLM) and the internal custom end-to-end IE system, NESTLE can extract any type of information that has not been predefined in the IE system opening up the possibility of unlimited customizable statistical analysis of the corpus without writing a single line of code. We validate our system on 15 Korean precedent IE tasks and 3 legal text classification tasks from LexGLUE. The comprehensive experiments reveal NESTLE can achieve GPT-4 comparable performance by training the internal IE module with 4 human-labeled, and 192 LLM-labeled examples.

## introduction legal documents include a variety of semistructured information stemming from diverse social disputes. for instance, precedents include factual information (such as blood alcohol level in a driving under the influence (dui) case or loss in an indemnification case) as well as a decision from the court (fine, imprisonment period, money claimed by the plaintiff, money approved by the court, etc). while each document contains detailed information about specific legal events among a few individuals, community-level insights can be derived only by analyzing a substantial collection of these documents. for instance, the consequence of the subtle modification to the statute might only become evident through a comprehensive statistical analysis of the related legal corpus. indeed a recent study shows that how the revision of the road traffic act has changed the average imprisonment period in drunk driving cases by analyzing 24k korean precedents @xcite . conducting a comprehensive statistical analysis on a legal corpus on a large scale may entail following three key steps: (1) choosing a subset of the corpus using retrieval tools, (2) structuralizing the documents using information extraction (ie) systems, and (3) visualizing the data for the statistical analysis. each step requires either specialized tools or programming knowledge, impeding analysis for the majority of legal practitioners. particularly during text structuration, if the target information is not predefined in the ontology of the ie system, one needs to build their own system. to overcome such limitation, we developed nes-tle 1 , a no-code tool for statistical analysis of legal corpus. with nestle, users can search target documents, extract information, and visualize statistical information of the structured data via the chat interface, accompanied by an auxiliary gui for the finelevel controls, such as hyperparameter selection, ontology modification, data labeling, etc. a unique design choice of nestle is the combination of llm and an custom end-to-end ie system @xcite ) that brings the following merits. first, nestle can handle custom ontology provided by users thanks to the end-to-end (generative) property of the ie module. second, nestle can extract target information from the corpus with as few as 4 examples powered by the llm. for given few examples, llm builds the training dataset for the ie module under few-shot setting. finally, the overall cost can be reduced by 200 times, and the inference time can be accelerated by 6 times compared to ie systems that rely exclusively on llm, like chatgpt, when analyzing 1 million documents. we validate nestle on three legal ai tasks: (1) 4 korean legal ie tasks @xcite , (2) 1 no-code tool for statistical analysis of le-gal corpus 11 new korean legal ie tasks derived from lbox-open dataset @xcite , and (3) 3 english legal text classification tasks from lexglue @xcite @xcite . the comprehensive experiments reveal nestle can achieves gpt-4 comparable performance with just 4 humanlabeled, and 192 llm-labeled examples. in summary, our contributions are as below. there are few studies focusing on the capability of llm as a data analysis agent. zhang et al. develop data-copilot that can help users to interact with various data sources via chat interface. ma et al. examines the capability of gpt-3 (codex, code-davinci-002) as few-shot information extractor on eight ner and relation extraction tasks and propose using llm to rerank outputs from small language models. ding et al. evaluate the capability of gpt-3 as a data annotators on sst2 text classification task and crossner tasks reporting that gpt-3 shows good performance on sst2. he et al. propose 'explain-then-annotate' framework to enhance llm's annotation capability. under their approach, gpt-3.5 achieves either super-human or human-comparable scores on three binary classification tasks. our work is different from these previous works in that we focus on building a no-code tool for "statistical analysis" of "corpus" where efficient, accurate, yet customizable methods of structuralization of large-scale documents are necessary. our work is also different in that we focus on information extraction tasks from legal texts. finally, rather than performing all ie via llm, we focus on hybridization between commercial llm and open-sourced small language model (slm) by distilling knowledge of llm to slm. in this way, the api cost of using llm does not increase linearly with the size of corpus enabling nestle to be applied to industrial scale corpus. viswanathan et al. recently proposes prompt2model allowing users to construct an nlp system by providing a few examples. compared to prompt2model, nestle is specialized in large-scale ie task in legal domain and provides additional features like chat-based statistical analysis and gui for fine-level control. also nestle is rigorously validated on a variety of legal ie tasks. information extraction from legal texts previous studies build ie systems for legal texts using tagging-based methods @xcite @xcite @xcite @xcite or generative methods @xcite . our system is similar to @xcite in that we use an end-to-end ie system and focus on statistical analysis of legal information. however our work is unique in that we present a no-code tool and explore hybridization of commercial llm and open-sourced slm to expand the scope of analysis to a large-scale corpus while focusing on three realworld metrics: accuracy, time, and cost. ## demo in this section, we provide the explanation for our demo. the video is also available at https: //youtu.be/twkpjyjrvi8 labeling interface users can upload their data (unstructured corpus) using an upload button. althernatively, they can test the system with examples prepared from 7 legal domains by selecting them through the chat interface. each dataset comes with approximately 1500 documents and 20 manually labeled examples. after loading the dataset, users can view and perform manual labeling on documents using the dropdown menu where the values of individual fields (such as blood alcohol level, fine amount, etc.) can be labeled or the new fields can be introduced. the changes are automatically saved to the database. ## experiments all experiments are performed on nvidia a6000 gpu except the experiments with mt5-xxl where eight a100 gpus are used. the ie module of nes-tle is fine-tuned with batch size 12 with learning rate 4e-4 using adamw optimizer. under this condition, the training sometimes becomes unstable. in this case, we decrease the learning rate to 3e-4. the high learning rate is purposely chosen for the fast training. the training are stopped after 60 epochs (nestle-s), or after 80 epochs (nestle-l, nes-tle-l+). in case of nestle-xxl, the learning rate is set to 2e-4 and the model is trained for 20 epochs with batch size 8 using deepspeed stage 3 offload @xcite . for efficient training, lora is employed in all experiments (hu et al., 2022) using peft library from hugging face @xcite . in all evaluations, the checkpoint from the last epoch is used. for the data labeling, we use chatgpt version gpt-3.5-turbo-16k-0613 and gpt-4 version gpt-4-0613. in all other operations with llm, we use the same version of chatgpt except during normalization of numeric strings such as imprisonment period and fines where gpt-3.5-turbo-0613 is used. we set temperature 0 to minimize the randomness as ie tasks do not require versatile outputs. the default values are used for the other hyperparameters. during the few shot learning, we feed llm with the examples half of which include all fields defined in the ontology while the remaining half are selected randomly. ## results we validate nestle on 15 korean precedent ie tasks and 3 english legal text classification tasks. three classification tasks are eurlex, ledgar, and unfair-tos from lexglue @xcite @xcite . eurlex dataset consists of a pair of european union legislation (input) and corresponding legal concepts (output) from the eurovoc thesaurus. in ledgar task, a model needs to classify the paragraphs from contracts originated from us securities and exchange commission fillings. similarly, unfair-tos is a task of predicting 8 types of unfair contractual terms for given individual sentences from 50 terms of service. these 3 classification tasks are used to demonstrate nestle on common (english) legal ai benchmark and also to show nestle can be applied to general ai tasks that can be represented in text-to-text format @xcite . ## analysis we have shown that nestle can extract information with accuracies comparable to gpt-4 on many tasks. in this section, we extend our comparison of nestle to commercial llms focusing on two additional real-world metrics: cost and time. as a case study, we select fraud task from korprec-ie where all models struggled ( ## conclusion we develop nestle, a no-code tool for statistical analysis of legal corpus. to find target corpus, structure them, and visualize the structured data, we combine a search engine, a custom end-to-end ie module, and llm. powered by llm and the endto-end ie module, nestle enables unrestricted personalized statistical analysis of the corpus. we extensively validate nestle on 15 korean precedent ie tasks and 3 english legal text classification tasks while focusing on three real-world metrics: accuracy, time, and cost. finally, we want to emphasize that although nestle is specialized for legal ie tasks, the tool can be easily generalized to various nlp tasks that can be represented in a text-to-text format.