BB 25 HL egal S um: Leveraging BM 25 and BERT -Based Clustering for the Summarization of Legal Documents (2023)

Abstract: Legal document summarization aims to provide a clear understanding of the main points and arguments in a legal document, contributing to the efficiency of the judicial system. In this paper, we propose BB25HLegalSum, a method that combines BERT clusters with the BM25 algorithm to summarize legal documents and present them to users with highlighted important information. The process involves selecting unique, relevant sentences from the original document, clustering them to find sentences about a similar subject, combining them to generate a summary according to three strategies, and highlighting them to the user in the original document. We outperformed baseline techniques using the BillSum dataset, a widely used benchmark in legal document summarization. Legal workers positively assessed the highlighted presentation.

## introduction pending judicial processes are a prevalent and significant issue affecting legal systems worldwide. the number of pending cases can vary significantly depending on population size, legal system, and backlog of cases. while in some countries, there may be only a few thousand pending processes, it can amount to millions in others. this scenario motivates the research of computational techniques that can help accelerate judicial analysis, select similar cases for judging in batches, or identify patterns that could lead to better decision-making. the automatic summarization of legal documents to synthesize their essence is critical in this context. the goal of automatic text summarization is to create summaries that are similar to human-created summaries @xcite . this is a challenging task since natural language is complex and nuanced. text summarization algorithms must consider the intended audience, the purpose of the summary, as well as the type and format of the original text. text summarization is valuable for various applications, such as news aggregation, document management, and legal document summarization. most works use extractive summarization to generate summaries, defined in @xcite , as "the generation of a summary containing a sentence subset of the original text after identifying the important sentences". several techniques were explored for extractive legal text summarization, including word relevance @xcite , graph-based ranking models @xcite , statistical models @xcite , and deep learning @xcite . more recently, bert @xcite has been leveraged in the legal area @xcite , inspired by state-of-the-art results achieved in general extractive text summarization @xcite . another approach used in the legal documents area is bm25 @xcite , a ranking function commonly used in information retrieval to determine the relevance of a document concerning a search query. the combined use of bert and bm25 is recurrent for information retrieval in legal documents @xcite , but it is still in the initial stages in the legal documents summarization area. bert is a powerful language model that captures complex relationships between words and sentences, while bm25 is an effective information retrieval algorithm to rank documents. the strengths of these techniques can be joined to produce high-quality summaries and help to overcome some of the traditional methods' hurdles (e.g., feature engineering, long documents). according to @xcite , there needs to be more analysis of the readability of the generated summaries, and how to present them. in the legal area, summary presentation is addressed using highlighting @xcite and heatmaps @xcite representing the relevance of sentences within the original document. however, the relevance of a sentence may be a secondary aspect for legal workers, who seek the main arguments within their context. in this article, we propose bb25hlegalsum (bert + bm25 + highlighting legal documents summarization), a novel method for the extractive summarization of legal documents. it leverages bert and bm25 to identify relevant sentences in a legal document and combine clusters of sentences to generate candidate summaries, which are selected using metrics against a reference summary. we generate summaries using three strategies to identify the best parts of a document, focused on the precision of the selected sentences, their coverage of the text (recall), and a trade-off between these two criteria. another distinctive feature is the presentation of the generated summary. we propose a subsidiary highlighting approach that represents, using different colors, the sentences contained in the summaries generated according to each strategy. in this way, the user can identify and distinguish in their original context the relevant sentences of the document according to distinct points of view that emphasize precision, coverage, or both. our experiments address the following research questions: (1) how does the performance of bb25hlegalsum compare to baseline methods for legal document summarization? (2) how does the length of the reference summary impact the recall and precision of the generated summary using bb25hlegalsum? (3) which type of document summary is more readable in the legal context: focused on precision, recall, or f-measure? our method outperformed baseline works in a benchmark dataset @xcite . we observed that the length of the reference summary impacts the recall and precision of the generated summaries and that bb25hlegalsum performs better for larger-than-average summaries. a qualitative assessment by legal workers has shown that highlighting with distinct colors enables identifying different types of information captured by each summarization strategy. they pointed out that higher recall is the most critical criterion for summarization in the legal context, since it avoids missing relevant information. the main contributions of our article are: (1) a method that leverages bert and bm25 to generate legal document summaries. it outperforms baselines @xcite @xcite in a benchmark dataset; the remaining of this work is structured as follows. section 2 presents related work. section 3 describes bb25hlegalsum in detail. section 4 presents our experiments. section 5 outlines the conclusions and future work. ## related work extractive summarization forms summaries by selecting and concatenating the most important spans (typically sentences) in a document @xcite . legal document summarization has explored various techniques. casesummarizer @xcite combines standard summary methods based on word relevance (i.e., tf-idf) with domain-specific knowledge to summarize legal documents. graphbased ranking models, notably lexrank @xcite and textrank @xcite , explore the relationships and similarities between nodes representing the text to select the relevant portions of legal documents. statistical models have been utilized for scoring the relevance of sentences in legal documents, including bayesian optimization @xcite , kullback-leibler @xcite , and latent semantic analysis @xcite . the contextual nuances and semantic dependencies in legal documents are explored for generating summaries using deep learning @xcite . more recently, a trend is to deploy pre-trained models such as bert @xcite , which capture complex relationships between words and sentences. the focus in some works is the presentation of the generated legal summary. @xcite uses different colors to highlight the top-5 sentences, and @xcite proposes a heatmap to distinguish the importance of sentences. however, the relevance of a sentence may be a secondary aspect for legal workers, given that they generally seek the key arguments within a legal document. the quality of generated summaries is typically assessed by comparing the generated summary against some reference summary using rouge (recall-oriented understudy for gisting evaluation) @xcite . in the context of rouge, recall refers to how much of the reference summary is captured in the system summary, precision measures how much of the system summary is relevant, and f1 combines recall and precision. necessary assessments on legal text summarization remain unaddressed, such as properties of the readability of the summaries (e.g., the trade-off between conciseness and completeness) and the relationship between performance efficiency and reference summaries, typically used as the gold standard to evaluate the proposed summary systems @xcite . bm25 @xcite is a wellestablished information retrieval algorithm that ranks documents based on their relevance concerning a query. the combined use of bert and bm25 is recurrent for document retrieval in the competition on legal information extraction/entailment (coliee) @xcite @xcite , but its potential has not been fully examined for legal document summarization. the resulting summarization model can benefit from the strengths of both approaches to produce high-quality summaries and help to overcome some of the traditional methods' hurdles, such as the reliance on feature engineering and the difficulty in handling long documents. our work contributes with a solution that leverages bert and bm25 to produce legal document summaries, and with a method for presenting the generated summaries using highlighting that enables the examination of the trade-off between conciseness and completeness for readability of legal documents summaries. ## bb25hlegalsum overview bb25hlegalsum is a novel method for the extractive summarization of legal documents. it assumes as input a legal document d, composed of a legal description (desc), and a reference summary (refsum). given a document d, the goal is to select from desc a set of relevant sentences and to combine them to produce a generated summary, hereafter gsum. our premise is that, for a lawyer, the most important aspect of legal document summarization is the extraction of the most relevant arguments and the ability to identify their importance within a context. hence, the refsum may synthesize the document, but it does not necessarily provide all the useful information a legal worker needs. our method comprises four main steps: (1) select from d.desc a set of unique, relevant sentences by leveraging bert to explore similarity thresholds and bm25 to rank sentences; (2) aggregate relevant sentences to select a set of candidate summaries (candsum m ) by combining clusters of related sentences; and (3) select among the candidate summaries the most representative one, as measured by rouge against the reference summary (d.refsum); (4) present the generated summary gsum in the original document by highlighting the selected sentences using different colors, combining multiple perspectives of importance. a significant concern in our work is understanding the trade-off of conciseness and completeness as a measure of the quality of the generated summaries. hence, our method proposes and assesses three strategies to select the best-generated summary, given a set of possible candidates, according to the metrics used for the selection (precision, recall, and f-measure, respectively). the remaining of this section provides details on our method. ## conclusions in this paper we described bb25hlegalsum, a method that leverages bm25 and the combination of bert clusters to summarize legal documents. we generate a summary using three strategies to understand the role of preciseness and completeness in legal documents: posum, rosum, and fosum. the summaries are presented to users within the original document with three-colored highlighted sentences that indicate the relevant sentences ac-cording to a summarization perspective. our experiments revealed that this unsupervised method outperforms the baselines for the billsum dataset (us and ca test data), and that the length of the reference summary impacts the recall and precision of the generated summaries. the larger the reference summary, the better is the performance of our system. we also conducted a qualitative assessment with three lawyers, who evaluated that summaries that target higher recall (rosum) are more appropriate in the legal context, since they avoid missing relevant information. they also positively evaluated the three-coloring approach proposed, arguing that it provides the context of the sentences and the relevance perspective. future work includes improving the combination of clusters to generate summaries, and a more comprehensive readability assessment.