BLT : Can Large Language Models Handle Basic Legal Text? (2024)

Abstract: We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of examples that lawyers and paralegals would expect LLMs to handle zero-shot, such as looking up the text at a line of a witness deposition or at a subsection of a contract. LLMs’ poor performance on this benchmark casts into doubt their reliability as-is for legal practice. However, fine-tuning on our training set brings even a small model to near-perfect performance. This benchmark will be useful for fine-tuning LLMs for downstream legal tasks, as well as for tracking LLMs’ reliability as-is for basic legal tasks.

## introduction there has been much excitement about the potential of large language models (llms) to handle complex reasoning tasks. in the legal domain, recent work has focused on llms' ability to perform high-level tasks such as passing the bar exam, identifying hearsay evidence, or navigating the tax code. these results have led to predictions that llms will imminently revolutionize law practice. however, no prior work has investigated llms' ability to handle the most very basic text tasks done in daily legal practice. in this paper, we introduce a benchmark for basic legal text (blt) handling. blt mimics simple tasks done by lawyers and paralegals, including looking up citations to the record, finding statutory citations containing particular text or definitions, or finding the text in a contract that another lawyer has explicitly cited. blt is designed so that a paralegal, new lawyer, or even non-subject matter expert could perform zero-shot at or near 100%. but we find that leading publicly available llms perform poorly out-of-thebox. an example of this poor performance is in figure 1 , where gpt-4o fails to retrieve the text on 1: mr. macejka: object to form. 2: a. the medical invoice from the mao would be 3: the medical claims data. so we're already in 4: receipt of that. 5: by mr. pennekamp: 6: q. that's what you referred to a medical 7: invoice, is the claims data? 8: mr. macejka: object to form. 9: a. that essentially is what the medical 10: claims data is. 11: by mr. pennekamp: 12: q. do you ask the providers for their 13: invoices for the medical services? 14: mr. macejka: object to form. 15: a. we will request for eobs or billing what is the exact text of just line 7 above? the exact text of line 7 is: "q. that's what you referred to a medical" figure 1 : gpt-4o incorrectly answering a simple question about a page from a line-numbered witness deposition transcript. line numbers, exactly as shown above, are passed to gpt-4o, which incorrectly answers 13% of such one-page, 15-line deposition retrieval prompts. the specified line of a deposition transcript excerpt that is a mere 15 lines long. yet we find that fine-tuning a less-advanced model, gpt-3.5-turbo, on blt's training set leads to near-human (i.e., 100%) performance. we also find that this fine-tuned model also performs better on a more complex legal task, demonstrating blt's value for fine-tuning llms for legal applications. the blt dataset has additional strengths. our code foot_0 can generate unlimited new examples never before seen in corpora. moreover, blt is scalable to different window sizes and is one of the few datasets taking full advantage of llms with window sizes of 64k or 128k tokens. ## background law is a largely text-based profession and thus is often used to demonstrate llms' capabilities. openai's gpt-4 technical report @xcite mentioned only a single benchmark in the abstract itself: gpt-4 had passed the bar exam, with the score in the 90th percentile. openai's developer livestream by co-founder greg brockman introducing @xcite used four examples to show gpt-4's capabilities. one involved u.s. tax law, where brockman prompted gpt-4 with several tax-code sections and had it calculate the taxes of hypothetical taxpayers alice and bob. brockman proclaimed that gpt-4 can "do taxes." ## the blt benchmark the blt benchmark involves three different types of legal text, each of which has between two and five different tasks run on it. ## results and discussion we tested four models from openai: gpt-3.5turbo, gpt-4, and gpt-4-turbo, and gpt-4o. from google, we tested the chat-bison-32k variant of gpt-4o and claude-3.5 are broken out separately since their training cutoff was several months after the blt dataset was made public online, so they may have seen the blt test data during training. under @xmath0, the number in parentheses is how often the model erred by returning the unamended u.s. code text rather than the amended text provided to the model in the prompt, results discussed in subsection 4.4. palm 2. from anthropic, we tested claude-2.1 and claude-3.5 sonnet. (due to token limits, we were not able to test claude-3.5 on our two largest test sets, blt-64k and -128k.) crucially, gpt-4o and claude-3.5's training cutoffs were several months after we released the blt test and training data sets on the internet in november 2023. they may have seen the blt test sets during training, artificially boosting their perfor-mance. accordingly, they are broken out separately in we note that accuracy monotonically decreases as prompt size increases, with each model achieving higher accuracy on blt-4k than blt-8k, higher accuracy on blt-8k than blt-16k, and so on. ## fine-tuning we fine-tune the 4,000-token version of gpt-3.5turbo with blt-4k's training set. for each of the 11 task types, blt-4k has a training set with 1,000 prompts and answers, for a total of 11,000 prompts and answers. recall that the training set and test set are generated in the same way, with the same code. of the 11,000 prompts and answers in the training set, we use 90% for training, leaving 10% as a possible evaluation set for future work. we train for two epochs with all hyperparameters, like learning rate, set to the defaults. the results of the fine-tuning are in this result shows the utility of fine-tuning llms on blt's training data to improve performance on higher-level legal tasks. also, this result is in line with the findings of dominguez-olmedo et al. (2024) that small llms that have been lightly finetuned on legal tasks often substantially outperform larger llms on legal tasks. ## llms trained after blt release we generally see the best performance from claude-3.5 and gpt-4o, the two most recently released llms that we test. there are three possible, non-exclusive explanations. first, these models may be more advanced and thus better at most tasks. second, we released the blt test sets on the internet in november 2023, several months before these models' training cutoff dates, so there may have been test-set leakage, with the models already having seen the answers during training. third, the blt training sets (which are ten times larger than the test sets) were also released in november 2023, before these models' training cutoffs. as the fine-tuning experiments in section 5 demonstrate, training on the blt training data can substantially improve performance. seeing the blt training data may have improved these models' basic legal text handling. ## conclusion the chief innovation officer at a large international law firm observed to the new york times of llms, "at its best, the technology seems like a very smart paralegal." @xcite . we find llms are more like sloppy paralegals. currently available llms perform poorly outof-the-box on basic legal text handling. the blt tasks are designed to be truly basic, with humans able to perform them at or near 100%. the gpt-4 family, palm 2, and claude-2.1 all fall far short. only one model, claude-3.5, comes close, with 97.0% aggregate performance, and then only on our smallest test set, blt-4k. yet even claude-3.5 achieves only 80% on retrieving the text on a line of one or two pages of deposition transcript. we find that fine-tuning on our training set brings performance up to near 100%. we expect blt to be a useful resource for those fine-tuning llms for much more complicated legal tasks, as well as a benchmark for llms' ability to do basic legal text handling without fine-tuning.