Abstractive Summarization of D utch Court Verdicts Using Sequence-to-sequence Models (2022)

Abstract: With the legal sector embracing digitization, the increasing availability of information has led to a need for systems that can automatically summarize legal documents. Most existing research on legal text summarization has so far focused on extractive models, which can result in awkward summaries, as sentences in legal documents can be very long and detailed. In this study, we apply two abstractive summarization models on a Dutch legal domain dataset. The results show that existing models transfer quite well across domains and languages: the ROUGE scores of our experiments are comparable to state-of-the-art studies on English news article texts. Examining one of the models showed the capability of rewriting long legal sentences to much shorter ones, using mostly vocabulary from the source document. Human evaluation shows that for both models hand-made summaries are still perceived as more relevant and readable, and automatic summaries do not always capture elements such as background, considerations and judgement. Still, generated summaries are valuable if only a keyword summary or no summary at all is present.

## introduction given the increasing availability of legal information and the fact that many legal documents are often relatively long and dense, there is an increasing need for systems that can automatically summarize these documents. such summaries can help not only lawyers and judges, but also citizens, companies and researchers to process case law. two key approaches exist for automatic summarization: extractive summarization involves identifying important text spans from the document and combining them into a summary, and abstractive summarization involves generating new sentences that explain in more general terms what the text is about @xcite . abstractive summaries are potentially more readable and more efficient than extractive summaries. for example, consider the following sentence: by letter of 18 june 2012, the appellant addressed a request to the defendant to take enforcement action against [a] inc. and @xcite inc. for (alleged) violation of the provisions of the quarantine facilities for live bivalve molluscs regulation 2007. an abstractive model can retain only the information that the appellant requested enforcement action based on the quarantine facilities for live bivalve molluscs regulation 2007, while an extractive model would retain the full sentence. in this paper we apply a reinforcement learning approach with a bilstm (referred to as rl) as well as a deep learning approach based on the bart @xcite transformer model (referred to as bart) to abstractive summarization of the dutch case verdict database rechtspraak.nl. we show that generated summaries are useful, but not yet on par with human-generated summaries. the rest of this paper is structured as follows. in section 2 we consider current state-of-the-art models for legal summarization, section 3 describes our dataset, and section 4 discusses the design and implementation of the deep learning pipelines. here we also highlight the different evaluation methods used: in addition to the common rouge metric, we also look at abstractiveness @xcite , i.e., the amount of novelty introduced in the wording of the summary, and perform a human evaluation on the aspects of summary relevance and readability. finally, section 5 and 6 will elaborate upon the results and implications of this research. ## related work legal text differs from common document types such as news articles @xcite , which has prompted the development of custom word embeddings for legal vocabulary in english @xcite . however, general pre-trained embeddings or embeddings trained on-the-fly are also commonly used. early approaches used pattern-based heuristic segmentation approaches @xcite . machine learning was used by @xcite to classify sentences as different legal rhetorical structures @xcite such as fact, proceedings or background. @xcite applied probablistic graphical models based on conditional random fields (crf) to segment and label a legal decision given various rhetorical roles (e.g., argument or final decision). @xcite used a naive bayes algorithm with a set of heuristic features to identify sections (introduction, context, reasoning, conclusion) and create a summary. instead of identifying specific sections or elements, @xcite use a rule-based approach, where rules created by domain experts are used to identify important phrases in a decision. more recently, @xcite create summaries by classifying sentences in a decision as, e.g., issues, decision, etc. similarly, @xcite use a number of different machine learning techniques to classify the issues, conclusions and reasons in a legal verdict. @xcite use an lstm classifier on sentences from the 'reasoning' section of taiwanese supreme court judgements to determine which sentences belong to the 'gist' of the judgement, achieving an f1-score of around 0.9. @xcite used a combination of supervised sentence-scoring and tf-idf in an ensemble method on their billsum dataset. regarding abstractive approaches, @xcite use the pointer model by @xcite on their supreme court of india dataset. @xcite use their pre-trained pegasus language model to generate abstractive summaries on the billsum dataset of @xcite . previous work on abstractive summarization of uk court verdicts was performed by @xcite . ## data for the current research, data from the dutch judicial system is used. on average, around 1.6m cases are handled in the netherlands every year, of which a small percentage is published on the official website rechtspraak.nl. for the rl experiments in this research a pre-processed version of the rechtspraak data provided by pandora intelligence foot_0 is used, providing easy access to the type, summary and verdict of each case. for the bart experiments a separate preprocessing pipeline is used that exposes only the case text and the summary. in total, this dataset contains around 430k legal court cases. 94% of these cases contain a summary, and we included only these cases in the data exploration discussion in the current section. an example document can be found in table a1 . on average, case texts contain ∼ 650 words with summaries of length ∼ 50. however, a significant amount of summaries has less than 25 words, containing only keywords or a single sentence. a small amount of summaries is over 250 words long. to provide more uniform data to the models, for training we used only cases that have a summary containing between 40 and 150 words, and consisting of a minimum of three and a maximum of six sentences with at least 5 words in each sentence. note that very short summaries are reintroduced in the dataset for human evaluation. ## research method we use two deep learning pipelines on the rechtspraak.nl data: a hybrid reinforcement learning method and a transformer-based method. ## conclusion & discussion in this work, a dataset containing over 400k dutch court verdicts was used to train a hybrid reinforcement learning-based model, as well as a transformer-based bart model. we evaluated generated summaries based on rouge, abstractiveness, and through a human evaluation experiment using legal experts. our experiments report an f1 score of 46.52 (rouge-1), 33.74 (rouge-2) and 44.88 (rouge-l) for the bart model, com-parable to state-of-the-art results achieved on the cnn/daily mail dataset. the models did not introduce many novel ngrams, but showed good performance in rewriting and shortening sentences. the evaluation also showed the potential to improve the model, following observations that the model may cut sentences off too early and does not always include all elements (background, considerations and judgement) in the summary. furthermore, while important facts were generally included and the rewriting process showed adequate results, still unnecessary case details are found in the generated summaries. considering the level of abstractiveness the models showed the capability of rewriting long and redundant sentences found in legal text to much shorter ones. quantitatively it was shown that the model generates a large number of novel ngrams compared to the reference summaries from the dataset. due to the inverse relationship of the abstractiveness and rouge score of a document, a good performance in producing novel n-grams actually reduces the score on the summary quality evaluation measured with rouge, which was confirmed by the analysis of the evaluation results. therefore, we argue that rouge scores are not fully representative as a metric for abstractive summarization. while alternative methods are being developed @xcite rouge is likely to remain an influential evaluation approach, however these results should be interpreted carefully when comparing models and approaches. to complement rouge scores, a human evaluation study was conducted to evaluate both generated and reference summaries on readability and content. especially for the rl model the results show a large difference in relevance between reference summaries (6.7/10) and generated summaries (4.6/10), and a slightly smaller difference in readability (7.0/10 vs 5.6/10). however, the participants in the rl experiments noted that the generated summaries did contain key information about the case and preferred it to using a reference summary consisting of only keywords. for the bart experiments the difference across the four dimensions informativeness, relevance, fluency and readability were perceived to be smaller but still the reference summaries were preferred for all dimensions. it has been argued in the literature that an abstractive summary may be less accurate and can lead to misinterpretations of a judge's intent @xcite . furthermore, as argued by @xcite , there are many citations (to e.g. previous cases or articles of law) which cannot be ignored. however, with more data being available, improved hardware and matured algorithms, the accuracy of abstractive models is increasing. furthermore, citations can often be extracted from legal texts using basic regular expressions and the relevant legal articles or precedent cases can be provided as metadata, which can then be presented in combination with the abstractive natural language summary. furthermore, we argue that even less accurate summaries can be useful as a tool for quickly searching through huge databases of cases. furthermore, there is also the possibility to combine abstractive models with more domain-specific constraints, such as citing law articles and structuring the summary into facts, arguments and decision. this study fills the following gaps in current research on (legal) text summarization. first, very few research on legal summarization has made use of an abstractive summarization model. the authors are aware of two approaches only, of which the first shows comparatively low evaluation scores @xcite , and the second is based on us congressional bills @xcite , which, while they can be considered legal documents, are rather different from the case verdicts and decisions we consider. for example, bills -essentially numbered lists of laws and statutes -are much more structured than verdicts, and the language used in bills is much more generic because it does not pertain to individual cases like verdicts. second, like @xcite , our work shows that unsupervised neural models originally developed for news articles can be successfully used on legal documents, which differ significantly from news articles both in terms of length and in terms of internal structure and distribution of relevant content elements. furthermore, no previous research has applied an abstractive summarization model on a dataset of legal documents in dutch, showing that our unsupervised language models are robust considering the legal language of the documents presented to the model. for both models, long texts are still difficult to process due to technical limitations on input representation. a case verdict document can easily surpass such length constraints and will be truncated (e.g., to 1024 words) as a result prior to summarization. with respect to future research, models designed to process longer text @xcite therefore seems promising. also, even though pre-trained language models are known for their ability to generalize across domains, the model of @xcite used to obtain the high levels of performance on congressional bills shows a relatively average performance on the cnn/daily mail news dataset, which might support the hypothesis that the document structure (rather than the model itself) is the predominant factor for the summary evaluation scores. applying the current two methods on the congressional bills would provide more insight into the reasons behind the performance differences. currently, the rl model uses static word2vec word embeddings created on the fly on the rechtspraak dataset. in contrast, the bart contextual embeddings were pretrained on the generalpurpose c4 dataset. while the bart model already outperforms the rl model by a significant margin for both rouge scores and human ratings, it would be interesting to investigate whether pretraining bart on domain-specific data (i.e., dutch legal text) would result in an additional performance increase. a practical problem however is data availability: the c4 subset currently used contains 6b words of crawled web pages, which is difficult to match with dutch legal text. other future work includes a more detailed analysis of summaries generated by the bart model. we have observed that the overall quality of the bart summaries is higher compared to the summaries generated by the rl model, in terms of grammaticality and topicality. it would be interesting to compare the detailed observations made for the rl model, such as the abstractiveness, relation to law categories and time frames, and missing legal aspects in the summaries, to establish whether the bart model supports these observations as well. case (ecli:nl:cbb:crvb:2012:bw3333) [...] considerations [...] the job expert has selected functions which he considers the appellant capable of performing and has established the loss of earning capacity to less than 35%. as decided on january 7th, 2010, it is established that appellant starting october 20, 2009 is not entitled to benefits regarding the law work and income. the insurance physician has concluded that the impairments are established correctly. he has studies the letter by the general practitioner dated december 31, 2009. according to the insurance physician there are no grounds to assume further impairments. [...] the court has judged that the decision was based on a sufficiently appropriate medical foundation. there is no reason to doubt the conclusion by the insurance physicians that appellant is capable of performing simple, routine tasks. [...] for the appeal appellant has entered information by the general practitioner and treating psychologist. [...] the insurance physician has stated that this information is not a reason to assume further impairments. the council follows this conclusion. [...] establishing sickness-or handicap-based impairments to performing labor based on the medical condition of the appellant should in principle be considered the expertise of the insurance physician. [...] appellant is rightfully considered to be able to perform the selected tasks. [...] it did not become apparent that the workload exceeds the capabilities of the appellant.