Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation (2022)

Abstract: Summarization of legal case judgement documents is a challenging problem in Legal NLP. However, not much analyses exist on how different families of summarization models (e.g., extractive vs. abstractive) perform when applied to legal case documents. This question is particularly important since many recent transformer-based abstractive summarization models have restrictions on the number of input tokens, and legal documents are known to be very long. Also, it is an open question on how best to evaluate legal case document summarization systems. In this paper, we carry out extensive experiments with several extractive and abstractive summarization methods (both supervised and unsupervised) over three legal summarization datasets that we have developed. Our analyses, that includes evaluation by law practitioners, lead to several interesting insights on legal summarization in specific and long document summarization in general.

## introduction in common law systems (followed in india, uk, usa, etc.) law practitioners have to read through hundreds of case judgements/rulings in order to identify relevant cases that they can cite as precedents in an ongoing case. this is a time-consuming process as case documents are generally very long and complex. thus, automatic summarization of legal case documents is an important problem @xcite @xcite . it is additionally challenging due to two primary reasons as demonstrated in a plethora of solutions exists for text summarization, for e.g., extractive and abstractive, supervised and unsupervised, etc. @xcite . also, several legal domain-specific methods have been designed for case document summarization @xcite . however, detailed systematic analyses are rare on how the different families of summarization models perform on legal case documents. our prior work @xcite took an early step in this direction, but it mostly considered extractive methods. the state-of-the-art in document summarization has advanced rapidly in the last couple of years, and there has not been much exploration on how recent transformer-based summarization models perform on legal documents @xcite . to bridge this gap, we (1) develop three legal case judgement summarization datasets from case documents from the indian and uk supreme courts (see we show that simply computing document-wide metrics gives an incomplete picture of the quality of legal document summarization. in particular, we see some differences between automatic evaluation and evaluation by domain experts. for instance, supervised methods like summarunner, and finetuned bart usually achieve higher rouge scores, but the law practitioners often prefer the summaries generated by simpler unsupervised methods such as dsdr and casesummarizer. again, the rouge scores achieved by the best extractive models are at par with those achieved by the best abstractive models. however, the practitioners often prefer the extractive summaries over the abstractive ones. ## related work we give an overview of existing summarization algorithms @xcite . ## datasets for legal summarization there are very few publicly available datasets for legal case document summarization, especially in english (see since 50 document-summary pairs are not sufficient for training supervised models, when applying these models on in-ext, they were trained over the 7, 030 document-summary pairs in the in-abs train set. we ensure that there is no overlap between this training set and the in-ext dataset. ## experimental setup and evaluation target length of summaries: during inference, the trained summarization models need to be provided with the target length of summaries l (in number of words). for every document in the in-ext dataset, we have two reference summaries (written by two experts). for a particular document, we consider l to be the average of the number of words in the two reference summaries for that document. for in-abs and uk-abs datasets, l is taken as the number of words in the single abstractive reference summary for a given document. given a document, every model is made to generate a summary of length at most l words. some algorithms (e.g. kmm, gist) return a ranking of sentences according to their summary-worthiness. the final summary is obtained by selecting sentences in descending order of the ranked list till the limit of l words is reached. evaluation of summary quality: we report rouge-1, rouge-2, and rouge-l fscores (computed using https://pypi.org/ project/py-rouge/ , with max_n set to 2, parameters limit_length and length_limit not used, and other parameters kept as default), and bertscore @xcite (computed using https://pypi.org/project/ bert-score/ version 0.3.4) that calculates the semantic similarity scores using the pretrained bert model. we calculate two kinds of rouge and bertscore as follows: ## extractive summarization methods we consider some representative methods from four classes of extractive summarizers: (1) legal domain-specific unsupervised methods: let-sum, kmm, casesummarizer, and mmr. (2) legal domain-specific supervised methods: gist. short descriptions of all the above methods are given in section 2. the implementation details for the domain-specific methods we implemented, and publicly available code repositories are stated in the appendix (section a.2 and section a.3). training supervised extractive models: the supervised methods (gist, summarunner and bertsum) require labelled training data, where every sentence must be labeled as 1 if the sentence is suitable for inclusion in the summary, and 0 otherwise. as stated in section 3, we use parts of the in-abs and uk-abs datasets for training the supervised methods. however, since both these datasets have abstractive summaries, they cannot be directly used to train the extractive summarizers. we explore three methods -maximal, avr, and tf-idf -for converting the abstractive summaries to their extractive counterparts. best performances for the supervised methods are observed when the training data is generated through the avr method; hence we describe avr here and report results of the supervised methods trained on data generated through avr. descriptions of maximal and tf-idf are stated in the appendix (section a.4). @xcite . for each sentence in the abstractive goldstandard summary, we select 3 sentences from the source document (full text) that have the maximum average of rouge-1, rouge-2 and rouge-l scores w.r.t. the sentence in the abstractive summary. then we take the union of all the sentences thus selected, and label them 1 (to be included in the summary). all other sentences in the source document are assigned a label of 0. ## abstractive summarization methods we apply several abstractive methods for legal document summarization, including both pretrained models and models finetuned for legal document summarization. a key challenge in applying such methods is that legal documents are usually very long, and most abstractive summarization models have restrictions on the number of input tokens. ## results and analyses this section analyzes the performance of different summarization models. for in-ext, in-abs and uk-abs datasets, ## concluding discussion we develop datasets and benchmark results for legal case judgement summarization. our study provides several guidelines for long and legal document summarization: (1) for extractive summarization of legal documents, dsdr (unsupervised) and summarunner (supervised) are promising methods. (2) for abstractive summarization, legal-pegasus (pretrained and finetuned) is a good choice. (3) for long documents, fine-tuning models through chunking seems a promising way. (4) document-wide evaluation does not give the complete picture; domain-specific evaluation methods, including domain experts, should also be used.