L ex S umm and L ex T 5: Benchmarking and Modeling Legal Summarization Tasks in E nglish (2024)

Abstract: In the evolving NLP landscape, benchmarks serve as yardsticks for gauging progress. However, existing Legal NLP benchmarks only focus on predictive tasks, overlooking generative tasks. This work curates LexSumm, a benchmark designed for evaluating legal summarization tasks in English. It comprises eight English legal summarization datasets, from diverse jurisdictions, such as the US, UK, EU and India. Additionally, we release LexT5, legal oriented sequence-to-sequence model, addressing the limitation of the existing BERT-style encoder-only models in the legal domain. We assess its capabilities through zero-shot probing on LegalLAMA and fine-tuning on LexSumm. Our analysis reveals abstraction and faithfulness errors even in summaries generated by zero-shot LLMs, indicating opportunities for further improvements. LexSumm benchmark and LexT5 model are available at https://github.com/TUMLegalTech/LexSumm-LexT5.

## introduction language serves as the bedrock of the legal domain, facilitating precise communication in this complex field. legal systems globally engage in the production, consumption and interpretation of massive volumes of text. legal professionals, comprising lawyers, judges and regulators, continually author a diverse array of complex legal documents, such as briefs, memos, statutes, regulations, contracts, patents and judicial decisions @xcite . in their routines, these professionals not only craft these documents but also immerse themselves in extensive volumes of text, refining their comprehension of the law for effective human behavior management. beyond the realms of consumption and production, the practice of law and the art of lawyering hinge on the analysis and interpretation of textual content @xcite , often perceived by laypersons as legalese or legal gobbledygook @xcite . recent advancements in nlp stand poised to revolutionize legal tasks and significantly benefit stakeholders within the legal domain @xcite . by automating labor-intensive processes, such as document analysis @xcite @xcite @xcite , information extraction (luz de @xcite @xcite , question answering @xcite @xcite @xcite ), text classification @xcite @xcite , information retrieval @xcite @xcite and summarization @xcite @xcite @xcite @xcite @xcite @xcite , nlp with its ability to understand and interpret complex legal language can enhance efficiency and accelerate decision-making. nlp can act as a force multiplier by not only streamlining tasks but also amplifiying the capabilities of legal professionals, leading to increased productivity of legal stakeholders @xcite . to enable a systematic comparison of approaches, legal evaluation benchmarks like lexglue @xcite and lex-treme @xcite have been proposed, focusing on predictive tasks. however, there is an absence of a dedicated benchmark designed for assessing legal generation capabilities. moreover, resources on legal natural language generation (nlg) are sporadic and scattered. in response to this, we introduce lexsumm, a new benchmark curated for training and evaluating legal english summarization models. it includes eight english legal summarization datasets from various jurisdictions, such as the us, uk, eu, and india, for training task-specific models-distinguishing it from legalbench @xcite and lawbench @xcite , oriented towards zero/few-shot llm evaluation. lexsumm represents the distinctive characteristic of legal documents, marked by their long length, posing a challenge for pre-trained models like bart @xcite and t5 @xcite . in our benchmarking efforts, we evaluate lexsumm using long-context models such as led @xcite , longt5 @xcite , and primera @xcite . we also explore contemporary approaches of adopting short-range pre-trained models like t5 @xcite with fusion-in-decoder techniques as in sled @xcite and integration of retrieval techniques, as demonstrated in unlimiformer @xcite , to adopt them for longer documents. additionally, we compare recent long-context zero-shot llms like gpt-3.5 and claude on lexsumm. pre-trained language models such as bert @xcite , roberta @xcite have significantly transformed the nlp landscape, showcasing remarkable efficacy in general-domain text. however, their performance diminishes when applied to domain-specific tasks, leading to concept of continued pre-training with domain-specific unlabeled data @xcite . this resulted in the development of legal-specific pretrained models like legalbert @xcite @xcite . to the best of our knowledge, there has been a lack of sequenceto-sequence model tailored for the legal domain. to address this gap, we introduce lext5, an english legal-oriented sequence-to-sequence model pre-trained on the lexfiles corpus @xcite , from six english-speaking legal systems (eu, european council, canada, us, uk, india). to evaluate the legal knowledge acquired by lext5, we compare its to t5 on legallama @xcite , a zero-shot legal probing suite. we also assess lext5's performance on lexsumm by incorporating into sled and unlimiformer frameworks to accommodate longer inputs. our quantitative and qualitative analysis reveal that lexsumm presents a substantial challenge for existing models including zero-shot llms such as gpt-3.5, leaving much room for the research community to improve upon. to streamline future model evaluations, we will release our benchmark and our pre-trained lext5 model on the hugging face hub, contributing to the advancement of legal nlp research. turning to specific domains, medeval @xcite and m3 @xcite are benchmarks tailored for the medical domain, with classification and generation tasks. in line with these efforts, this work introduces lexsumm, a legal domain-specific summarization benchmark with eight datasets. benchmarks for legal domain lexglue @xcite stands out as the pioneering benchmark in the legal domain, evaluating nlp models on tasks related to legal language understanding. it encompasses seven classification tasks derived from six english legal nlp datasets, spanning jurisdictions such as the us, eu, and the council of europe. lextreme @xcite is a multilingual benchmark for the legal domain, comprising 11 relevant nlu datasets covering 24 languages from two language families (indo-european and uralic). fairlex @xcite , another legal benchmark focuses on assessing fairness across five attributes-gender, age, region, language, and legal area-across legal nlp tasks. fairlex covers four jurisdictions (european council, usa, switzerland, and china), supports five languages (english, german, french, italian, and chinese). lbox @xcite benchmarks korean legal tasks, consisting of two classification tasks, two legal judgment prediction tasks, and one summarization task. legalbench @xcite is construced to assess legal reasoning consisting of 162 tasks covering six different types of legal reasoning, designed for benchmarking zero/few-shot llm paradigm for english language primarily based on american laws. similarly, lawbench @xcite is llm oriented benchmark designed for assessing chinese civillaw system, containing 20 diverse tasks covering 5 task types: single-label, multi-label classification, regression, extraction and generation. in this work, we curate lexsumm benchmark, focusing on eight legal summarization datasets in english, facilitating fine-tuning of task-specific models, an important setting for numerous applications. lexsumm benchmark, with generative tasks, complements the lexglue benchmark @xcite for legal text understanding in english. legal pre-trained models @xcite demonstrated continuing pre-trained models on domain-specific text improves performance on domain tasks. subsequently, there have been efforts to continue pre-training on diverse english legal text like legislation, court cases, and contracts, spanning us, eu, and uk jurisdictions resulting in the creation of legalbert @xcite . in a similar vein, caselawbert @xcite is another law-specific bert model trained using the harvard law case corpus from us federal and state courts. @xcite compiled an extensive corpus known as pile of law, incorporating documents from the us, canada, and eu and trained bert-large on this corpus, giving rise to the polbert. paul et al. 2023 extended pretraining on the indian legal corpora, culminating in inlegalbert. recently, chalkidis et al. 2023 introduced lexlms, pre-trained on lexfiles, a diverse multinational english legal corpus from six primarily english-speaking legal systems. while the aforementioned models focus on english legal corpora, parallel endeavors have emerged to develop legal pre-trained models other languages. french legal model, juribert @xcite is trained using corpora from the court of cassation, france's highest court. similar initiatives include jurbert for romanian @xcite , lamberta, italianlegalbert for italian @xcite , robertalex for spanish @xcite , lawformer for chinese @xcite , aralegalbert for arabic @xcite , lcube for korean @xcite , and legalbert-pt, bertbr for portuguese @xcite . recently, @xcite introduced legalxlm, a multilingual model pre-trained on the multilegalpile, a diverse legal corpus comprising 24 languages from 17 jurisdictions. it is noteworthy that the aforementioned legal domain-specific pre-trained language models predominantly adhere to the bert-style encoderonly architecture and currently, there is a lack of sequence-to-sequence models specifically adapted for legal text. addressing this gap, we present lext5, legal-oriented sequence-to-sequence model pre-trained on the lexfiles corpus for english. ## lexsumm benchmark lexsumm benchmark is a collection of eight legal nlg datasets in english language spanning across us, eu, uk and india jurisdictions. in this section, we describe these datasets and their characteristics. billsum @xcite is a summarization dataset of us congressional bills, sourced from the govinfo service by the us government publishing office along with humanwritten summary from the congressional research service. it consists of 22218 document-summary pairs split into training (16664), validation (2222) and test (3322) sets. eurlexsum @xcite eur-lex platform provides access to various legal documents published by various european union organs. this dataset focuses on the enforced eu legislation along with their summaries, available across all 24 european languages. we restrict to english version of the dataset spanning 1504 document-summary pairs, split into 1128/151/225 for training, validation and testing respectively. govreport @xcite ) contains 19,465 national policy reports published by u.s. government accountability office an expert-written summary is provided along with each report and it is split into 14598, 2919, 1946 for training, validation and test sets respectively. multilexsum-tiny/short/long @xcite consists of 9280 expert-written summaries for 4500 documents from u.s. federal civil rights lawsuits. it has summaries at three different granularities for the same source: (a) long (l) summaries contain multiple paragraphs, covering the case background, parties involved, major case events and proceedings. (b) short (s) summaries have only one paragraph with a shorter description of the background, parties involved and the outcome of the case. (c) tiny (t) summaries have one sentence intended to appear on twitter. input spans across multiple sources such as first complaint, last amended complaint, settlement agreements, opinions, orders etc. three different summarization tasks at each granularity are proposed emulating real-world tasks at the civil rights litigation clearinghouse. long, short and tiny versions have a total of 4539, 3138 and 1603 document-summary pairs respectively which are split into (3404/454/681), (2340/312/486) and (1207/145/251) for train, validation and test. inabs @xcite consists of indian supreme court judgements collected from the website of legal information institute of india . it provides summaries (also called 'headnotes') for some of the cases resulting in total of 7150 case document-summary pairs, which are split into training (5346), validation (713) and test (1069) sets. ukabs @xcite dataset is collected from the uk supreme court website which provides all judgements that were ruled since 2009. for most of the cases, along with the judgements, it also provides the official press summary of the cases. it consists of 793 document-summary pairs which are split into 595, 79, 119 for training, validation and test respectively. ## lext5 we build lext5, a legal-specific seq2seq pretrained model. t5 is an encoder-decoder model initially pre-trained in an unsupervised manner on the c4 corpus @xcite , using span denoising objective which involves replacing 15% of the tokens with sentinel tokens along with consecutive tokens marked for removal being replaced by a single sentinel token. the resulting corrupted text serves as input to the model to predict the masked-out span. then the model is further finetuned using supervised training on various downstream tasks, including those from the glue and superglue @xcite benchmarks, casting them into text-to-text format for training. we initialize the model with t5-base checkpoint of @xcite and continue pre-training using the span denoising objective on the train split of lexfiles @xcite . lexfiles is a diverse legal corpus across 6 primarily englishspeaking legal systems (eu, european court of human rights, canada, us, uk, india) covering various legal documents such as legislation, case law and contracts. it comprises approx. 6 million documents totalling up to approx. 19 billion tokens. we employ a sentence sampling rate from each sub-corpora proportional to number of tokens with exponential smoothing factor of 0.5 @xcite . implementation details in app b. ## benchmarking experiments we benchmark 8 lexsumm tasks using the following seq2seq models, designed to handle longer inputs. implementation details are in app. c. led @xcite ) is based on longformer, an efficient transformer model with linear complexity relative to input length. it features encoder and decoder components, employing efficient local+global attention in the encoder and full quadratic attention in the decoder. led is initialized from pre-trained bart @xcite , with the position embedding matrix initialized by duplicating bart's 1k position embeddings 16 times to handle 16k input tokens. primera @xcite is initialized with the led model and pre-trained with a novel summarization-specific masking objective based on the entity pyramid evaluation method, inspired by the gap sentence generation objective of pegasus @xcite . it can handle 4096 tokens. longt5 @xcite employs transient global attention, inspired by local+global attention from etc @xcite and integrates summarization-specific pre-training from pega-sus into the t5 model to handle longer sequences. we use longt5-base which can handle flexible lengths (unless constrained by memory) due to relative positional embeddings, unlike bart archi-tecture with absolute position embeddings. sled @xcite processes long sequences by partitioning them into overlapping chunks and encoding each chunk with a short-range pre-trained encoder. information across chunks is fused by the decoder by attending to all input tokens, akin to fusion-in-decoder @xcite . sled can be applied on top of any short-range model, resulting in sled-t5 and sled-lext5 derived from their respective base models. while it can handle any input length, it is ultimately memory-bound. unlimiformer @xcite utilizes a retrieval-based approach to enable short-range pretrained models to process inputs of unbounded length. it adopts a strategy akin to sled but focuses solely on the top-k tokens retrieved from a k-nearest-neighbor index constructed over the hidden states of all input tokens at each standard cross-attention head in every decoder layer. this distinguishes unlimiformer from sled which is limited by memory when attending to all input tokens in the decoder. we derive unlimiformer-t5 and unlimiformer-lext5 from their base models. evaluation metrics: we use rouge-1/2/l @xcite and bertscore @xcite to measure the lexical and semantic overlap between the model generated output and the reference summary. ## conclusion in this work, we curate lexsumm benchmark for training and evaluating legal summarization tasks in english. lexsumm can serve as an evaluation platform to foster development of approaches dealing with long legal text using efficient transformer architectures or retrieval-based methods adopted for longer context, legal-oriented pre-training or masking schemes, faithful decoding strategies. we pre-train lext5, a legal seq2seq model and evaluate on legallama probing task and lexsumm downstream benchmark. we compare lext5 wrapped in long-range adaptation frameworks such as sled and unlimiformer with t5 model in longrange adaptation, other long-range pre-trained models, and even zero-shot llms. we release lext5 to the community, hoping it will serve as a backbone model for various legal generative tasks. additionally, we envision lexsumm evolving into a dynamic benchmark, expanding with new datasets over time.