Large Language Models are legal but they are not: Making the case for a powerful L egal LLM (2023)

Abstract: Realizing the recent advances from Natural Language Processing (NLP) to the legal sector poses challenging problems such as extremely long sequence lengths, specialized vocabulary that is usually only understood by legal professionals, and high amounts of data imbalance. The recent surge of Large Language Models (LLM) has begun to provide new opportunities to apply NLP in the legal domain due to their ability to handle lengthy, complex sequences. Moreover, the emergence of domain-specific LLMs has displayed extremely promising results on various tasks. In this study, we aim to quantify how general LLMs perform in comparison to legal-domain models (be it an LLM or otherwise). Specifically, we compare the zero-shot performance of three general-purpose LLMs (ChatGPT-3.5, LLaMA-70b and Falcon-180b) on the LEDGAR subset of the LexGLUE benchmark for contract provision classification. Although the LLMs were not explicitly trained on legal data, we observe that they are still able to classify the theme correctly in most cases. However, we find that their mic-F1/mac-F1 performance are upto 19.2/26.8% lesser than smaller models fine-tuned on the legal domain, thus underscoring the need for more powerful legal-domain LLMs.

## introduction legal professionals typically deal with large amounts of textual information on a daily basis to make well-informed decisions in their practice. this can become very tedious and demanding due to the overwhelming amount of data they must manage and the meticulous attention to detail necessary to maintain the required precision in their work. thanks to the rise of llms, many tasks such as sentiment analysis, named entity recognition, information retrieval, etc. can now be handled by * these authors contributed equally neural models. though this holds true for the legal domain as well @xcite , they aren't used to make direct decisions. nevertheless, these automated systems that produce legal predictions and generations, are predominantly useful as advisory tools for legal practitioners that can augment their decision-making process. transformers @xcite have become the de facto method for many text classification and multiple choice question answering tasks. bert @xcite , a transformerencoder, and its derived models like roberta @xcite are commonly employed in legal nlp tasks. pre-training such models on legal corpora can help a model adapt to a specific domain by finetuning it with domain-specific data. legalbert @xcite is one such bert model that was trained on legal-oriented data. caselaw-bert @xcite , pol-bert @xcite , and lexlm @xcite are a few more bert-based variants pre-trained for the legal domain. although they show remarkable performance on various legal tasks in comparison with general-purpose bert models, one limit of these models is that bert's input size can only incorporate a maximum of 512 tokens. for short sequences this may seem enough, but in the case of long documents which is commonly found in the legal domain, where input texts can go over 5000 tokens (and requiring even more in few-shot settings), it can be a severe drawback as a lot of important information will get truncated. due to this limit, bert-based models aren't employed as-is in long-document tasks. typically, methods like hierarchical attention are utilized where the long document is split into segments of max length (512 in the case of bert models) and these segments are independently encoded. these segment embeddings are then aggregated with stacked transformers to get the overall encoding of the entire document. similarly, recurrent transformers @xcite @xcite were proposed to process long documents by encoding its representation from individual segments in a recurrent fashion. sparse attention is another method that has been proposed to tackle long sequence inputs @xcite . longformer @xcite uses a combination of local and global attention mechanisms to save on computational complexity and enables the processing of up to 4096 tokens. a number of other works @xcite show that transformer-based architectures that can capture longer text boast major benefits, even more so when augmented with strategies like sparse-attention and hierarchical networks. this again underlines an important direction for verbose legal datasets. our contributions can be summarized as follows: ## related work in this section, we outline the relevant research on llms, efforts using them for legal domain tasks, and finally the benchmarks and datasets. ## experimental setup and results in this section, we describe our experimental approach, along with specifics of our evaluations. ## conclusion in this work, we examine three general-purpose llms' zero-shot performance on a multi-class contract provision classification task using the ledgar dataset of lexglue. our study shows that these llms, even though aren't explicitly trained in legal data, can still demonstrate respectable theme classification performance. the results highlight the need for better legalllms adapted to the specifics of the legal industry, which has been underexplored compared to other domains. in light of this, we also present a review of related datasets and models, which we hope will help get an overview of the field.