L ex GLUE : A Benchmark Dataset for Legal Language Understanding in E nglish (2022)

Abstract: Laws and their interpretations, legal arguments and agreements are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

## introduction law is a field of human endeavor dominated by the use of language. as part of their professional training, law students consume large bodies of text as they seek to tune their understanding of the law and its application to help manage human behavior. virtually every modern legal system produces massive volumes of textual data @xcite . lawyers, judges, and regulators continuously author legal documents such as briefs, memos, statutes, regulations, contracts, patents and judicial decisions @xcite . beyond the consumption and production of language, law and the art of lawyering is also an exercise centered around the analysis and interpretation of text. natural language understanding (nlu) technologies can assist legal practitioners in a variety of legal tasks @xcite ; aletras * corresponding author: ilias.chalkidis@di.ku.dk et @xcite @xcite , from judgment prediction @xcite @xcite @xcite , information extraction from legal documents @xcite @xcite and case summarization @xcite to legal question answering @xcite @xcite ) and text classification @xcite @xcite . transformer models @xcite pre-trained on legal, rather than generic, corpora have also been studied @xcite @xcite . pre-trained transformers, including bert @xcite , @xcite , t5 @xcite , bart @xcite , deberta @xcite and numerous variants, are currently the state of the art in most natural language processing (nlp) tasks. rapid performance improvements have been witnessed, to the extent that ambitious multi-task benchmarks @xcite are considered almost 'solved' a few years after their release and need to be made more challenging @xcite . recently, @xcite named these pre-trained models (e.g., bert, dall-e, gpt-3) foundation models. the term may be controversial, but it emphasizes the paradigm shift these models have caused and their interdisciplinary potential. studying the latter includes the question of how to adapt these models to legal text @xcite . as discussed by @xcite and @xcite , legal text has distinct characteristics, such as terms that are uncommon in generic corpora (e.g., 'restrictive covenant', 'promissory estoppel', 'tort', 'novation'), terms that have different meanings than in everyday language (e.g., an 'executed' contract is signed and effective, a 'party' is a legal entity), older expressions (e.g., pronominal adverbs like 'herein', 'hereto', 'wherefore'), uncommon expressions from other languages (e.g., 'laches', 'voir dire', 'certiorari', 'sub judice'), and long sentences with unusual word order (e.g., "the provisions for termination hereinafter appearing or will at the cost of the borrower forthwith comply with the same") to the extent that legal language is often classified as a 'sublanguage' @xcite @xcite . furthermore, legal documents are often much longer than the maximum length state-ofthe-art deep learning models can handle, including those designed to handle long text @xcite @xcite . inspired by the recent widespread use of the glue multi-task benchmark nlp dataset @xcite , the subsequent more difficult superglue @xcite , other previous multi-task nlp benchmarks @xcite , and similar initiatives in other domains @xcite , we introduce lexglue, a benchmark dataset to evaluate the performance of nlp methods in legal tasks. lexglue is based on seven english existing legal nlp datasets, selected using criteria largely from superglue (discussed in section 3.1). we anticipate that more datasets, tasks, and languages will be added in later versions of lexglue. 1 as more legal nlp datasets become available, we also plan to favor datasets checked thoroughly for validity (scores reflecting real-life performance), annotation quality, statistical power, and social bias @xcite . as in glue and superglue @xcite , one of our goals is to push towards generic (or 'foundation') models that can cope with multiple nlp tasks, in our case legal nlp tasks, possibly with limited task-specific fine-tuning. another goal is to provide a convenient and informative entry point for nlp researchers and practitioners wishing to explore or develop methods for legal nlp. having these goals in mind, the datasets we include in lexglue and the tasks they address have been simplified in several ways, discussed below, to make it easier for newcomers and generic models to address all tasks. we provide python apis integrated with hugging face @xcite to easily import all the datasets we experiment with and evaluate the performance of different models (section 4.4). by unifying and facilitating the access to a set of law-related datasets and tasks, we hope to attract not only more nlp experts, but also more interdisciplinary researchers (e.g., law doctoral students willing to take nlp courses). more broadly, we hope lexglue will speed up the adoption and transparent evaluation of new legal nlp methods and approaches in the commercial sector, too. indeed, there have been many commercial press releases in the legal tech industry on high-performing systems, but almost no independent evaluation of the performance of machine learning and nlpbased tools. a standard publicly available benchmark would also allay concerns of undue influence in predictive models, including the use of metadata which the relevant law expressly disregards. ## related work the rapid growth of the legal text processing field is demonstrated by numerous papers presented in top-tier conferences in nlp and artificial intelligence @xcite @xcite as well as surveys @xcite @xcite . moreover, specialized workshops on nlp for legal text @xcite @xcite ) are regularly organized. a core task in this area has been legal judgment prediction (forecasting), where the goal is to predict the outcome (verdict) of a court case. in this direction, there have been at least three lines of work. the first one @xcite @xcite predicts violations of human rights in cases of the european court of human rights (ecthr). the second line of work @xcite @xcite considers chinese criminal cases where the goal is to predict relevant law articles, criminal charges, and the term of the penalty. the third line of work @xcite @xcite includes methods for predicting the outcomes of cases of the supreme court of the united states (scotus). the same or similar tasks have also been studied with court cases in many other jurisdictions including france @xcite , philippines @xcite , turkey @xcite , thailand @xcite , united kingdom @xcite , germany @xcite , and switzerland @xcite . apart from predicting court decisions, there is also work aiming to interpret (explain) the decisions of particular courts @xcite @xcite . another popular task is legal topic classification. @xcite highlighted the challenges of legal document classification compared to more generic text classification by using a dataset including docket entries of us court cases. @xcite classify eu laws into eu-rovoc concepts, a task earlier introduced by @xcite , with a special interest in few-and zero-shot learning. luz de araujo et al. (2020) also studied topic classification using a dataset of brazilian supreme court cases. there are similar interesting applications in contract law @xcite . several studies @xcite @xcite explored information extraction from contracts, to extract important information such as the contracting parties, agreed payment amount, start and end dates, applicable law, etc. other studies focus on extracting information from legislation @xcite or court cases @xcite . legal question answering (qa) is another task of interest in legal nlp, where the goal is to train models for answering legal questions @xcite @xcite @xcite . not only is this task interesting for researchers but it could support efforts to help laypeople better understand their legal rights. in the general task setting, this requires identifying relevant legislation, case law, or other legal documents, and extracting elements of those documents that answer a particular question. a notable venue for legal qa has been the competition on legal information extraction and entailment (coliee) @xcite @xcite . more recently, there have also been efforts to pre-train transformer-based language models on legal corpora @xcite @xcite , leading to state-of-theart results in several legal nlp tasks, compared to models pre-trained on generic corpora. overall, the legal nlp literature is overwhelming, and the resources are scattered. documentation is often not available, and evaluation measures vary across articles studying the same task. our goal is to create the first unified benchmark to access the performance of nlp models on legal nlu. as a first step, we selected a representative group of tasks, using datasets in english that are also publicly available, adequately documented and have an appropriate size for developing modern nlp methods. we also introduce several simplifications to make the new benchmark more standardized and easily accessible, as already noted. ## lexglue tasks and datasets we present the legal general language understanding foot_1 evaluation (lexglue) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal nlu tasks. ## vision -future considerations beyond the scope of this work and the examined baseline models, we identify four major factors that could potentially advance the state of the art in lexglue and legal nlp more generally: long documents: several transformer-based models @xcite @xcite have been proposed to handle long documents by exploring sparse attention mechanisms. these models can handle sequences up to 4096 sub-words, which is largely exceeded in three out of seven lexglue tasks (figure 2 ). contrary, the hierarchical model of section 4.2 can handle sequences up to 8192 sub-words in our experiments, but a part of the model (the additional transformer blocks that make the paragraph embeddings aware of the other paragraphs) is not pre-trained, which possibly negatively affects performance. structured text: current models for long documents, like longformer and bigbird, do not consider the document structure (e.g., sentences, paragraphs, sections). for example, window-based attention may consider a sequence of sentences across paragraph boundaries or even consider truncated sentences. to exploit the document structure, @xcite proposed smith, a hierarchi-cal transformer model that hierarchically encodes increasingly larger blocks (e.g., words, sentences, documents). smith is very similar to the hierarchical model of section 4.2, but it is pre-trained endto-end with two objectives: token-level masked and sentence block language modeling. large-scale legal pre-training: recent studies @xcite @xcite introduced language models pre-trained on legal corpora, but of relatively small sizes, i.e., 12-36 gb. in the work of @xcite , the pre-training corpus covered only a narrowly defined area of legal documents, us court opinions. the same applies to lawformer @xcite , which was pre-trained on chinese court opinions. future work could curate and release a legal version of the c4 corpus @xcite , containing multijurisdictional legislation, court decisions, contracts and legal literature at a size of hundreds of gbs. given such a corpus, a large language model capable of processing long structured text could be pre-trained and it might excel in lexglue. even larger language models: scaling up the capacity of pre-trained models has led to increasingly better results in general nlu benchmarks @xcite , and models have been scaled up to billions of parameters @xcite @xcite . in appendix e, we observe that using the large version of roberta leads to substantial performance improvements compared to the base version. the results are comparable or better -in some casescompared to the legal-oriented language models (legal-bert, caselaw-bert). considering that the two legal-oriented models are much smaller and have been pre-trained with (5-10×) less data (section 2), we have a strong indication for performance gains by pre-training larger legal-oriented models using larger legal corpora. ## limitations and future work although, our benchmark inevitably cannot cover "everything in the whole wide (legal) world" @xcite , we include a representative collection of english datasets that also ground to a certain degree in practically interesting applications. in its current version, lexglue can only be used to evaluate english models. as legal documents are typically written in the official language of the particular country of origin, there is an increasing need for developing models for other languages. the current scarcity of datasets in other languages (with the exception of chinese) makes a multilingual extension of lexglue challenging, but an interesting avenue for future research. beyond language barriers, legal restrictions currently inhibit the creation of more datasets. important document types, such as contracts and scholarly publications are protected by copyright or considered trade secrets. as a result, their owners are concerned with data-leakage when they are used for model training and evaluation. providing both legal and technical solutions, e.g., using privacy-aware infrastructure and models @xcite is a challenge to be addressed. access to court decisions can also be hindered by bureaucratic inertia, outdated technology and data protection concerns, which collectively result in these otherwise public decisions not being publicly available @xcite . while the anonymization of personal data provides a solution to this problem, it is itself an open challenge for legal nlp @xcite . in lack of suitable datasets and benchmarks, we have refrained from including anonymization in this version of lexglue, but plan to do so at a later stage. another limitation of the current version of lexglue is that human evaluation is missing. all datasets rely on ground truth labels automatically extracted from data (e.g., court decisions) produced as part of official judicial or archival procedures. these resources should be highly reliable (valid), but we cannot statistically assess their quality. in the future, re-annotating part of the datasets with multiple legal experts would provide an estimation of human level performance and inter-annotator agreement, though the cost would be high, because of the required legal expertise. while lexglue offers a much needed unified testbed for legal nlu, there are several other critical aspects that need to be studied carefully. these include multi-disciplinary research to better understand the limitations and challenges of applying nlp to law @xcite , while also considering fairness and robustness @xcite @xcite @xcite , and broader legal considerations of ai technologies in general @xcite @xcite .