Extractive Summarization of Legal Decisions using Multi-task Learning and Maximal Marginal Relevance (2022)

Abstract: Summarizing legal decisions requires the expertise of law practitioners, which is both time- and cost-intensive. This paper presents techniques for extractive summarization of legal decisions in a low-resource setting using limited expert annotated data. We test a set of models that locate relevant content using a sequential model and tackle redundancy by leveraging maximal marginal relevance to compose summaries. We also demonstrate an implicit approach to help train our proposed models generate more informative summaries. Our multi-task learning model variant leverages rhetorical role identification as an auxiliary task to further improve the summarizer. We perform extensive experiments on datasets containing legal decisions from the US Board of Veterans’ Appeals and conduct quantitative and expert-ranked evaluations of our models. Our results show that the proposed approaches can achieve ROUGE scores vis-à-vis expert extracted summaries that match those achieved by inter-annotator comparison.

## introduction in common-law systems, law practitioners research large numbers of legal decisions from past cases to find similar precedents that justify their arguments and lead to favorable outcomes. the analysis can be time-consuming and expensive as these documents are long and verbose, and understanding them requires legal expertise. automatic summarization of legal documents can help expedite the process cost-effectively. however, the limited availability of expert-annotated summaries makes it challenging to design such automated systems to assist paralegals, lawyers, and other law practitioners. extractive summarization aims to identify and extract essential sentences from the source document to compose the corresponding summary. it is more common in the legal domain due to the complexity of the legal language and the scarcity of labeled data. by contrast, abstractive summarization generates an abstract representation that captures the salient ideas of the source text and might contain new words and phrases not present in the source document. one of the main challenges of extractive summarization is the redundancy in legal documents, as legal decisions can often contain several semantically similar sentences. our objective is to generate summaries that provide maximum information while minimizing redundancy. maximal marginal relevance (mmr) @xcite has proved to be an effective tool to tackle redundancy explicitly @xcite by balancing the importance of query relevance and diversity. however, more recent methods like mmr-select can use neural models as a substitute for query relevance. additionally, we can train the neural models to handle the redundancy implicitly by adding a redundancy loss term @xcite . another challenge is the low availability of expert annotated summarization datasets in the legal domain. in this work, we leverage large amounts of unlabeled data along with the small annotated datasets to gain maximum performance. pretrained transformers like bert @xcite can improve the performance of downstream tasks, such as summarization, even with limited labeled data. however, such models trained on the general domain may fail to capture the intricacies of the domain-specific vocabulary used in legal decisions. the domain-specific variants of bert @xcite pre-trained on large corpora of legal texts can help better embed the legal terms and achieve robust performance in various legal-specific downstream tasks like argument mining @xcite , rhetorical role labeling @xcite , and legal citation recommendation @xcite . to maximize the summarization performance, we also leverage multi-task learning (mtl) by aggregating training samples from several smaller datasets of multiple related tasks. mtl helps the model learn shared representations between the primary task (summarization) and the auxiliary task (rhetorical role identification) to generalize better. the identification of rhetorical roles involves identifying the function of different sentences to understand underlying reasoning and argument patterns in legal decisions. previous works have often used rhetorical role labeling as a precursor to extractive summarization to improve performance @xcite . in this paper, we explore the idea of using rhetorical role identification as an auxiliary task to augment our annotated dataset and help generate better summaries. in brief, we consider our contributions to the extractive summarization of legal documents as follows: the quantitative evaluation demonstrates that the multi-task models perform better than the single-task models. ## data we use the dataset containing single-issue post-traumatic stress disorder decisions from the us board of veterans' appeals foot_0 (bva) by @xcite . these cases focus on veterans' appeals for benefits for a ptsd disability connected to stressful experiences during military service. the dataset is a sample from the bva database that has been constrained to single-issue cases focusing on ptsd. in the texts, the bva reviews the available evidence and either makes a finding that it warrants an award for service-connected ptsd (granted) or not (denied), or refers the case back to a lower administrative division for further @xcite . the dataset consists of 112 decisions and the corresponding expert annotated gold-standard summaries. we have 92 cases (48 remanded, 28 denied, 16 granted) in the training set with one annotated summary each. another 20 cases (10 remanded, 6 denied, 4 granted) constitute the test set, for which there are four extractive summaries by different annotators and two drafted abstractive summaries. each annotator chose a 6-10 sentence long summary based on predefined guidelines, out of which they selected 1-3 sentences each from the reasoning and evidence annotation type. the reasoning sentences connect the outcome to the facts, while evidence sentences add more information to support the former. for rhetorical role labeling, we use two different datasets containing 50 plus 25 annotated bva decisions 3 (walker et al., 2019). the decisions in the larger dataset have partial annotations, so we keep only decisions that have annotations for at least 60% foot_2 of the sentences in each decision. it results in 28 decisions consisting of 17 denied and 11 granted outcomes, while the smaller dataset contains 10 denied and 15 granted decisions. we map the different annotation types of the two datasets to a uniform type system of six annotation types. we merged the different annotation types for our experiments, resulting in 1889 evidence/reasoning and 3728 others sentences. therefore, the final dataset has 53 decisions with 7473 binary sentence-level annotations. the datasets are from different time periods; therefore, the decisions have a slightly different document structure. we remove the metainformation like the case number, dates, judge names, names of the witnesses, and other similar information to have a more uniform layout. we keep only the following sections (if present) from each decision in the dataset: ## discussion our proposed approaches generate higher-scoring summaries than baseline methods and expert annotators in terms of rouge scores and appear to be competitive in a qualitative expert ranking evaluation. the domain-specific pre-trained transformers help produce sentence embeddings capable of capturing the semantics of legal decisions in a way conducive to being used as a component in our proposed setup for summary generation. we can use these embeddings to train simple models like gru effectively. adding explicit methods like mmr helps tackle redundancy to generate more informative summaries even for verbose legal decisions. we further improve the performance by using the weighted cross-entropy loss function combined with redundancy loss (rdloss). the additional loss term helps train models that can handle redundancy implicit. the quantitative measures show the effectiveness of both the single-task and multi-task models, even with a limited dataset containing just 120 legal decisions. the supplementary dataset used to train the same model for the rhetorical role labeling task helps learn better representations and achieve better summary rouge scores. such correlated tasks and datasets prove helpful in low-resource settings such as ours to improve the performance further at no additional annotation costs, assuming they stem from the same legal domain. further qualitative analysis with the help of an expert annotator shows our proposed approaches rank at least as good as human annotators. it also indicates the need for better quantitative metrics to evaluate the quality of the summaries. the specialized bva domain we experiment in is relatively narrow in scope and highly regular in its document structure. our proposed methods seem promising as they work well with limited annotated datasets and computational resources but warrant further investigation and validation on larger, more diverse datasets. ## conclusion training models that can generate extractive summaries of legal opinions comparable to expert annotators in a low-resource setting can be challenging. we demonstrate that domain-specific pre-trained transformers and multi-task training with rhetorical role labeling can effectively train sequential extractive summarizaton models (in our case, grus) on a relatively constrained domain of cases. the proposed methods implement implicit and explicit redundancy checks to maximize the information and minimize the redundancy in summaries. in our experiments, we systematically analyze the performance of different techniques, both quantitatively and qualitatively. the results verify the efficacy of our model design. in the future, we plan to extend our work to other decision types and jurisdictions in the legal domain. we further plan to explore the discrepancy between our quantitative results favoring multi-task models and qualitative evaluation preferring summaries by single-task architectures.