Towards an Automated Pointwise Evaluation Metric for Generated Long-Form Legal Summaries (2024)

Abstract: Long-form abstractive summarization is a task that has particular importance in the legal domain. Automated evaluation metrics are important for the development of text generation models, but existing research on the evaluation of generated summaries has focused mainly on short summaries. We introduce an automated evaluation methodology for generated long-form legal summaries, which involves breaking each summary into individual points, comparing the points in a human-written and machine-generated summary, and calculating a recall and precision score for the latter. The method is designed to be particularly suited for the complexities of legal text, and is also fully interpretable. We also create and release a small meta-dataset for the benchmarking of evaluation methods, focusing on long-form legal summarization. Our evaluation metric corresponds better with human evaluation compared to existing metrics which were not developed for legal data.

## introduction generative text models, including large language models (llms), have made huge strides in performance in the last few years, and are now increasingly deployed in many domains in business and science. however, research on effective automated evaluation metrics for generated text has yet to catch up, and basic methodologies such as rouge @xcite and others (see section 2) are still used to judge the performance of new models. in the sub-field of text summarization, existing research on and meta-datasets for the evaluation of generated text summaries have focused mainly on shorter summaries consisting of a few sentences, while very little work has been done on long-form summaries (see the survey @xcite . long-form abstractive summarization is a task that has particular importance in the legal domain. legal documents such as court judgments (which are documents written by judges, detailing the background of a court case and the reasons for a ruling) are often many tens of pages long, and summaries of these can be several pages long. the uk supreme court, for instance, releases press summaries of 2-3 pages for the cases it decides (the supreme court of the united kingdom, 2024). modern llms, with their long context windows, are a natural tool for automatically generating such summaries from the original legal document. there is a pressing need, therefore, for effective automated evaluation metrics for the resulting longform summaries. in this paper, we propose an automated method for the evaluation of long-form generated legal summaries, which involves breaking each summary into individual points, comparing the points in a human-written reference and machine-generated candidate summary, and calculating a recall or precision score. we call our method the pointwise evaluation methodology. the idea of splitting summaries into discrete units to obtain reliable manual evaluation scores is well-known @xcite , and automated methods based on this idea have been explored @xcite . our proposed method expands upon previous work by: 1) adapting the methodology to be usable for long-form summaries and 2) using more advanced models to deal with the greater nuance and complexity of legal text. to evaluate this method against existing ones, we also create and release a small meta-dataset for benchmarking evaluation methodologies for longform legal summarization. to our knowledge, this is the first such dataset to be made available. ## meta-dataset for evaluation of long-form legal summaries we create a small meta-dataset for the evaluation of evaluation methods for long-form legal summarization, consisting of 7 cases from the uk supreme court (uksc) foot_0 . for each decided case, the uksc writes and releases a 2-3-page-long press summary. 2. even if the goods were not targeted at consumers in the uk, the company nonetheless infringed the trademarks. the court of appeals disagreed with the high court's conclusion that the listings were not targeted at uk consumers. ## automating the pointwise evaluation method each step in the two-step approach described in section 3.1 can be automated. ## discussion and conclusions the improvement in correlation of our method over the baselines is particularly pronounced for the population-level correlation. this is an indication that our method produces consistent results across all the court cases in our dataset. in other words, it does not merely rank the candidate summaries for each case in the correct order from best to worst, but also gives a recall score that is well-correlated with the human score on an individual candidate summary level. in addition, the root mean squared error between the llm-based pointwise metric and the human metric is much smaller than that of the baselines. this indicates that our method produces absolute recall scores that are close to the human scores, thus giving an accurate idea of the absolute quality of a single llm summary (and not just the comparatative quality of multiple llm summaries). the plots in figure 3 illustrate these points further. apart from performance, one of the advantages of the pointwise evaluation method over existing ones its its interpretability and explainability. the method allows us to see exactly which reference points are included or missing in the candidate summary. this allows us to improve the candidate summaries in a targeted manner, for example by editing the prompts to tell the summarizing llm to focus more on the type of information that the current summaries do not include. we can also see which points in each candidate summary were not included in the reference (and thus are probably irrelevant), and can thus also improve the llm prompts to avoid these.