MILDS um: A Novel Benchmark Dataset for Multilingual Summarization of I ndian Legal Case Judgments (2023)

Abstract: Automatic summarization of legal case judgments is a practically important problem that has attracted substantial research efforts in many countries. In the context of the Indian judiciary, there is an additional complexity – Indian legal case judgments are mostly written in complex English, but a significant portion of India’s population lacks command of the English language. Hence, it is crucial to summarize the legal documents in Indian languages to ensure equitable access to justice. While prior research primarily focuses on summarizing legal case judgments in their source languages, this study presents a pioneering effort toward cross-lingual summarization of English legal documents into Hindi, the most frequently spoken Indian language. We construct the first high-quality legal corpus comprising of 3,122 case judgments from prominent Indian courts in English, along with their summaries in both English and Hindi, drafted by legal practitioners. We benchmark the performance of several diverse summarization approaches on our corpus and demonstrate the need for further research in cross-lingual summarization in the legal domain.

## introduction legal judgment summarization is an important and challenging task, especially considering the lengthy and complex nature of case judgments @xcite @xcite . in the context of the indian judiciary, there is an additional requirement -indian legal case judgments are mostly written in complex english due to historical reasons, but a significant portion of india's population lacks a strong command of the english language. hence, it is important to summarize case judgements in indian languages to ensure equitable access to justice. there exist a few indian legal case judgment summarization datasets, e.g., the datasets developed by @xcite , but all of them contain english case documents and summaries only. in this work, we introduce mildsum (multilingual indian legal document summarization), a dataset consisting of 3,122 case judgments from multiple high courts and the supreme court of india in english, along with the summaries in both english and hindi, drafted by legal practitioners. there have been previous efforts in compiling document-summary pairs in the legal domain, such as the eur-lex @xcite dataset containing 1,500 document-summary pairs per language, for several european languages. however, to our knowledge, there has not been any similar effort in the indian legal domain. thus, mildsum is the first dataset to enable cross-lingual summarization of indian case judgments. to construct mildsum, we utilize the website livelaw ( https://www.livelaw.in/ ), a popular site among indian law practitioners, that publishes articles in both english and hindi summarizing the important case judgments pronounced by the supreme court and high courts of india. according to the site, these articles are written by qualified law practitioners; hence, they can be considered as high-quality summaries of the case judgments. we carefully extract the case judgments (in english) from this website, along with the english and hindi articles summarizing each judgment. a major challenge that we had to address in this process is to link the english article and the hindi article corresponding to the same judgment. the mildsum dataset is available at https://github.com/law-ai/mildsum . we also benchmark a wide variety of summarization models on the mildsum dataset. we consider two broad approaches -(1) a summarize-thentranslate pipeline, where a summarization model is used over a case judgment (in english) to generate an english summary, and then the english summary is translated into hindi, and (2) a direct crosslingual summarization approach, where a state-ofthe-art cross-lingual summarization model @xcite is finetuned over our dataset to directly generate hindi summaries from the english documents. we observe that the summarizethen-translate approach performs better (e.g., best rouge-2 score of 32.27 for english summaries and 24.87 for hindi summaries on average) than the cross-lingual summarization approach (best rouge-2 score of 21.76 for the hindi summaries), in spite of reduction in summary quality due to the translation stage. thus, our experiments demonstrate the need for better cross-lingual summarization models for the legal domain. to this end, we believe that the mildsum dataset can be very useful in training and evaluating cross-lingual summarization models, for making legal judgments accessible to the common masses in india. also note that, apart from training summarizing models, the mildsum dataset can also be used to train / evaluate machine translation (mt) models in the legal domain, given the paired english and hindi summaries. mt in the indian legal domain is another challenging problem that has not been explored much till date. ## related works there has been limited prior work on crosslingual summarization in indian languages. a popular large-scale multi-lingual summarization dataset massivesumm was introduced by @xcite which includes some indian languages as well. the ilsum dataset @xcite encompasses hindi and gujarati, in addition to english. @xcite recently introduced a large-scale cross-lingual summarization dataset crosssum, while @xcite released wikilingua as a benchmark dataset for cross-lingual summarization. however, none of these datasets have specifically addressed crosslingual summarization within the legal domain. there exist some multi/cross-lingual summarization datasets in the legal domain for non-indian languages, such as eur-lex @xcite and clidsum @xcite . in particular, eur-lex is a multi-and cross-lingual corpus of legal texts originating from the european union (eu); it covers all 24 official languages of the eu. but to our knowledge, no such effort has been made in domain #doc avg. #tokens doc en_sum hi_sum court judgments 3,122 4,696 724 695 ## the mildsum dataset in this work, we develop mildsum (multilingual indian legal document summarization), a collection of 3,122 indian court judgments in english along with their summaries in both english and hindi, drafted by legal practitioners. statistics of the dataset are given in ## experiments and results we divided the mildsum dataset into 3 parts in a 70:15:15 ratio -train set (2185 data points), validation set (469 data points), and test set (468 data points). only the test split is used to benchmark the performance of several state-of-the-art summarization models. we consider two broad approaches -(i) a summarize-then-translate pipeline approach, and (ii) a direct cross-lingual summarization approach -that are detailed next. ## conclusion this study develops the first multi-and crosslingual summarization dataset for indian languages in the legal domain (available at https://gith ub.com/law-ai/mildsum ). we also benchmark a variety of summarization models on our dataset. our findings emphasize the need for better crosslingual summarization models in the indian legal domain.