Computing and Exploiting Document Structure to Improve Unsupervised Extractive Summarization of Legal Case Decisions (2022)

Abstract: Though many algorithms can be used to automatically summarize legal case decisions, most fail to incorporate domain knowledge about how important sentences in a legal decision relate to a representation of its document structure. For example, analysis of a legal case sum- marization dataset demonstrates that sentences serving different types of argumentative roles in the decision appear in different sections of the document. In this work, we propose an unsupervised graph-based ranking model that uses a reweighting algorithm to exploit properties of the document structure of legal case decisions. We also explore the impact of using different methods to compute the document structure. Results on the Canadian Legal Case Law dataset show that our proposed method outperforms several strong baselines.

## introduction single document summarization aims at rephrasing a long text into a shorter version while preserving the important information @xcite . while recent years have witnessed a blooming of abstractive summarization models that can generate fluent and coherent new wordings @xcite @xcite , abstractive summaries often contain hallucinated facts @xcite . in contrast, extractive summarization models directly select sentences/phrases from the source document to form a summary. in certain domains such as the law or science @xcite , using exact wordings may be needed. in this work, we focus on extractive summarization of legal case decisions. different from texts in the news domain, case texts tend to be longer (e.g., in canadian legal case decisions @xcite there are on average 3.9k words, while standard news articles @xcite range from 400 -800 words) and also have more complicated document structures (e.g., legal cases are likely to be split into sections while news articles are not). in contrast to scientific domains, which also have long and structured texts, large training sets of case decisions and reference summaries are generally not freely available given the restrictions of the legal field. a currently used case dataset has less than 30k training examples @xcite , which is ten times less than scientific datasets such as arxiv and pubmed @xcite . thus, for the legal domain, it is not surprising that unsupervised extractive summarization methods are of interest. unfortunately, when researchers @xcite have attempted to directly apply standard unsupervised models to legal data, they have obtained mediocre results. however, most such attempts have failed to utilize the document structure of legal texts. in case law, important sentences about the issues versus the decisions of the court occur in different places in the document structure. in contrast, summarization algorithms typically flatten any structure during initial processing (i.e., they concatenate sentences from different sections/paragraphs of a document to form a sentence list), or select sentences using structural biases from other domains (e.g., the importance of leading sentences in news @xcite ). as shown in figure 1 , while lexrank correctly extracts the legal issue from the beginning of the source text, it incorrectly extracts several redundant sentences (i.e., @xcite , @xcite which talk about similar content) as well as ignores a large part of the article (e.g., no sentences are extracted from the last section of the original case: recapitulation of calculation of damages). in contrast, the human summary focuses on sentences related to the argument of the legal decision (e.g., what are the issues, reasoning and conclusions of this court case?), which tend to be spread across the document structure. recently, the hiporank model was proposed to exploit discourse structure patterns during unsupervised extractive summarization @xcite . however, the model was designed for long scientific articles, and the experiments were based on data where the articles were already split into document sections. for case decisions, document structures are generally either missing or only implicitly conveyed by text formatting. for example, in figure 1 , document sections are conveyed by bolding in the source html file. moreover, algorithms such as pacsum and hiporank compute sentence centrality just once and greedily select the top-k candidates as the extractive summary. such a greedy selection algorithm fails to match the distribution of the argumentative sentences that ultimately appear in human case law summaries. to address these limitations, we investigate the utility of different methods for automatically segmenting the sentences of legal case decisions into the sections of a document structure. we posit that incorporating better views of document structure could bring improvements in summarization quality when discourse-aware methods such as hipo-rank are applied to legal case decisions. we also propose a novel reweighting algorithm to improve how hiporank selects sentences when creating extractive summaries of legal decisions. the algorithm takes the history of already selected summary sentences into account, and gradually updates the importance score of a sentence. we posit that reweighting will decrease the selection of redundant sentences as well as increase the selection of argumentative sentences from less-represented document segments (e.g., in the middle). we evaluate our proposed method foot_0 for summarizing legal decisions using an annotated canadian case summarization dataset (canlii) @xcite . based on the belief that argumentative sentences will capture the important sentences to summarize in a legal decision @xcite , a portion of the can-lli dataset comes with gold-standard sentencelevel labels identifying which sentences are related to the issue/reasoning/conclusion of the court's decision in both source and summary documents. we use these labels to additionally propose a metric that can better evaluate the quality of the generated summary from a legal expert's perspective. empirical results show that our method improves performance over previous unsupervised models @xcite @xcite in automatic evaluation. unsupervised extractive summarization traditional extractive summarization methods are mostly unsupervised @xcite @xcite , where a large portion apply the graph-based algorithms @xcite @xcite or are based on term frequencies such as n-gram overlaps @xcite to rank the sentences' importance. more recently, pretrained transformer-based models @xcite @xcite have provided better sentence representations. for instance, @xcite built directed unsupervised graph-based models on news articles using bert-based sentence representations and achieved comparable performance to supervised models on multiple benchmarks. @xcite augmented the document graph of @xcite with sentence position and section hierarchy to reflect the document structure of scientific articles. different from these two works which are based on assumptions of news and scientific article structures, our method uses reweighting to better utilize the document structure of legal cases. extractive summarization of legal texts despite the success of supervised neural network models in news and scientific article summarization @xcite @xcite , they face challenges in legal document summarization given the longer texts, distinct document structure, and limited training data @xcite . instead, prior work has tackled legal extractive summarization by applying domain independent unsupervised algorithms @xcite @xcite , or designing domain specific supervised approaches @xcite @xcite . one recent work @xcite frames the task as integer linear programming and demonstrates the importance of in-domain structure and legal knowledge. in another line of research, @xcite propose a sentence classification task with the hope of exploiting the court decision's argument structure by making explicit its issues, conclusions, and reasons (i.e., argument triples). our work is unsupervised and implicitly reveals the relations between argument triples to generate better summaries. ## case decision summarization dataset recent work has introduced a number of legal document summarization or salient information identification tasks with associated datasets, e.g., for bill summarization @xcite and for case sentence argumentive classification @xcite and rhetorical role prediction @xcite . similarly to @xcite , we use the canlii (canadian legal information institute) dataset of legal case decisions and summaries foot_2 . full corpus statistics are provided in ## method and models we propose a reweighting model that employs a graph-based ranking algorithm to exploit the structures encoded in long legal case decisions. ## experimental setup for supervised models, we split the training data in an 80:20 ratio for training and validation. for unsupervised models, we tune the hyperparameters on the validation set. model training details can be found in appendix b. upper bound oracles based on figure 2 , we create a domain-dependent irc_oracle model where test sentences manually annotated with the irc labels are concatenated to form the summary. following @xcite , we also report results for ext_oracle, a domain-independent summarizer which greedily selects sentences from the original document based on the rouge-l scores compared to the abstractive human summary. extractive baselines for unsupervised models, we compare with lsa @xcite , lexrank @xcite , tex-trank @xcite , and pacsum @xcite . we also include hiporank @xcite with document views. for supervised methods, we compare with bert_ext @xcite . although not our focus, abstractive baselines are in appendix d.4. ## results in this section, we aim to deal with three research questions: rq1. how well do the extractive baselines including the hiporank backbone deal with legal documents? rq2. how well do the different views of document structures perform with the hiporank backbone? rq3. can the reweighting algorithm help select important argumentative sentences and improve summary quality? ## conclusion we presented an unsupervised graph-based model for the summarization of long legal case decisions. our proposed approach incorporated diverse views of the document structure of legal cases and utilized a reweighting scheme to better select argumentative sentences. our exploration of document structure demonstrates how using different types of document structure impacts summarization performance. moreover, a document structure inspired reweighting scheme yields performance gain on the canlii case dataset.