Towards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Reranking (2023)

Abstract: We propose a simple approach for the abstractive summarization of long legal opinions that takes into account the argument structure of the document. Legal opinions often contain complex and nuanced argumentation, making it challenging to generate a concise summary that accurately captures the main points of the legal opinion. Our approach involves using argument role information to generate multiple candidate summaries, then reranking these candidates based on alignment with the document’s argument structure. We demonstrate the effectiveness of our approach on a dataset of long legal opinions and show that it outperforms several strong baselines.

## introduction legal opinions contain implicit argument structure spreading across long texts. existing summarization models often struggle to accurately capture the main arguments of such documents, leading to summaries that are suboptimal @xcite . we propose an approach for the abstractive summarization of long legal opinions that leverages argument structure. legal opinions often follow a specific argumentative structure, with the main points of the argument being presented clearly and logically @xcite @xcite . prior work has shown that by considering this structure during summarization, it is possible to generate extractive and abstractive summaries that more accurately reflect the original argumentation in the document @xcite @xcite . in this paper, we present a framework for abstractive summarization of long legal opinions that extends this literature by leveraging argument structure during summary reranking to both generate and score candidates. our method involves utilizing the longformer-encoder-decoder (led) @xcite model to generate multiple candidate summaries by training it on various input formats. this allows for the consideration of different argument representations in the summary generation process. additionally, we use beam search to further diversify the output. finally, we rank the candidate summaries by measuring their lexical similarity to the input's main arguments. we evaluate our approach on a dataset of long legal opinions obtained from the canadian legal information institute (canlii) 1 and demonstrate that our method outperforms competitive baselines. our results with rouge and bertscore @xcite suggest that considering the argumentative coverage of the original opinions can lead to a more effective selection of summaries. our contributions are: (1) we propose a simple reranking approach that takes into account the argumentative structure of legal opinions to improve over the standard finetuning of generation models. (2) we demonstrate through empirical results and ablation analysis reasons for the effectiveness of our approach for summarizing long legal opinions. our code can be accessed through this repository: https://github.com/ engsalem/legalsummreranking 2 related work long legal document summarization legal documents have a distinct format, with a hierarchical structure and specialized vocabulary that differs from that of other domains @xcite . they also tend to be longer in length @xcite @xcite , which has led to the use of transformer models with sparse attention mechanisms @xcite @xcite to reduce the complexity of encoding lengthy text. legal opinions, in particular, have a complex argu-mentative structure that spans across the text, making it crucial to address in summaries @xcite @xcite . we use prior legal opinion summarization methods as evaluation baselines. summarization and argument mining using a dialogue summarization dataset with argument information, @xcite converted an argument graph into a textual format to train a summarizer. for legal documents, @xcite used argument role labeling to improve extractive summarization using multitask learning. @xcite blended argument role labeling and abstractive summarization using special markers, generating summaries that better aligned with legal argumentation. we incorporate the models of @xcite into summary reranking and further improve performance. second stage reranking generating multiple outputs and reranking them according to certain criteria has been successfully applied in nlp downstream applications including abstractive summarization. some methods use different input formats to generate multiple outputs. oved and levy (2021) perturbed input multi-opinion reviews to generate multiple candidate summaries, then ranked them using coherency. ravaut et al. (2022) used a multitask mixture of experts to directly model the probability that a summary candidate is the best one. @xcite ranked candidate summaries generated from 16 diverse beam searches to improve news summarization in terms of rouge score. @xcite presented a novel technique for summary reranking that involves a non-deterministic training objective. their approach enables the model to directly rank the summaries that are probable from beam-search decoding according to their quality. we rely on distinct argument-aware input formats in addition to diverse beam decoding to develop our argument-aware reranking method. ## annotated dataset we employ the annotated subset @xcite of the canlii dataset (zhong and litman, 2022) used in prior summarization research of legal opinions. this subset contains 1049 opinion/summary pairs annotated with sentence-level argument role labels for both input documents and reference summaries. the input opinions have mean/max lengths of 4375/62786 words, motivating us to use models for long text. recent work has proposed argument role taxonomies aligned with structures commonly found in legal text @xcite . the canlii data was annotated for argument roles using the irc scheme for legal opinions @xcite , which divides argument roles into issues (legal questions which a court addressed in the document), reasons (pieces of text which indicate why the court reached the specific conclusions), and conclusions (court's decisions for the corresponding issues). we use these 3 fine-grained irc labels, as well as collapse them into a single argumentative label, to incorporate argument structure into our models. an irc-annotated opinion and summary pair can be found in appendix a. ## model and methods our proposed method follows the generate and ranking paradigm and can be split into two parts. first, we explore techniques to utilize an argumentation augmented led model to generate multiple candidate summaries s. second, we propose a function µ that scores a summary s where @xmath0. the best candidate figure 1 shows an overview of our approach. ## experiments all models use led-base checkpoint as a base model. led-base encodes up to 16k tokens, which fits our long inputs. all experiments use 5-fold cross-validation, with the 4-fold documents split into 90% training and 10% validation; the validation split is used to select the best checkpoint. 4 we compare all rank-based methods (baseline and proposed) to abstractive baselines previously explored in legal opinion summarization: finetune led-base (which refers to vanilla model finetuning using our dataset), and arg-led-base (elaraby and litman, 2022) (which finetunes led on the dataset blended with argument markers that mark the start and the end of each argument role in the input). 5 we also compare our proposed rank-based approach from section 4 with ranking baselines that use different input formats or diverse decoding alone. specifically, we have employed ranking on top of the output of the three led models outlined in @xcite which are trained on distinct argument aware input formats (we refer to this model as "baseline ranking"). additionally, for diverse decoding, we have employed different beam widths within the range of 1 and 5 6 on top of the model trained on the input with fine-grained markers (arg-led-fine-grained), which achieved the best abstractive baseline rouge results. all models utilizing argument markers employed both oracle and predicted conditions during inference time, using human annotations or argument mining respectively, to produce the markers. ## results and discussion ## conclusion and future work we proposed a framework for improving the summarization of long legal opinions by combining distinct argument formats of the input with diverse decoding to generate candidate summaries. our framework selects the summary with the highest lexical overlap with the input's argumentative content. our results indicate that ranking alone can improve over abstractive baselines. moreover, combining ranking with our proposed candidate generation method improves results while maintaining robustness to noisy predictions. in future research, we plan to incorporate human expert evaluations to compare automatic metrics with human ratings. also, we aim to explore the impact of using noisier argument roles during training on a larger corpus by using the predicted markers obtained from our smaller dataset to experiment with the remaining unannotated portion of the canlii dataset.