Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization (2024)

Abstract: Legal professionals face the challenge of managing an overwhelming volume of lengthy judgments, making automated legal case summarization crucial. However, prior approaches mainly focused on training and evaluating these models within the same jurisdiction. In this study, we explore the cross-jurisdictional generalizability of legal case summarization models. Specifically, we explore how to effectively summarize legal cases of a target jurisdiction where reference summaries are not available. In particular, we investigate whether supplementing models with unlabeled target jurisdiction corpus and extractive silver summaries obtained from unsupervised algorithms on target data enhances transfer performance. Our comprehensive study on three datasets from different jurisdictions highlights the role of pre-training in improving transfer performance. We shed light on the pivotal influence of jurisdictional similarity in selecting optimal source datasets for effective transfer. Furthermore, our findings underscore that incorporating unlabeled target data yields improvements in general pre-trained models, with additional gains when silver summaries are introduced. This augmentation is especially valuable when dealing with extractive datasets and scenarios featuring limited alignment between source and target jurisdictions. Our study provides key insights for developing adaptable legal case summarization systems, transcending jurisdictional boundaries.

## introdution legal professionals, including lawyers, judges, and paralegals, are often inundated with an overwhelming amount of case judgements containing intricate textual content encompassing multiple issues and arguments, making it time-consuming to read and comprehend every document in detail manually. to address this, many legal information systems offer case summaries authored by legal experts. there has been a lot of academic research over the years to automatically produce the summaries of legal case judgements, which can help reduce the human cost effectively @xcite @xcite . traditional legal case judgement summarization methods have primarily focused on extractive summarization, where crucial sentences are selected from source documents. initially, unsupervised approaches were prominent, tailored to capture legal discourse nuances without requiring training data. more recently, supervised extractive summarizers have emerged, relying on expert-annotated summaries for training. recognizing the limitations of extractive methods in providing comprehensive context and coherence, researchers have shifted towards abstractive summarization, leveraging recent advances in unsupervised pre-training (e.g., @xcite , legal domain-specific continued pre-training (e.g., @xcite @xcite , and summarization task-specific pre-training (e.g., @xcite . the task of summarizing legal case judgements presents distinctive challenges due to several factors, such as the legal domain vocabulary being slightly different in different jurisdictions (e.g., country / court), as well as the complex sentence and discourse structures influenced by various writing styles that are in use in different jurisdictions. the current strategy to design a legal case summarization system for a new jurisdiction (which we call target jurisdiction) either involves unsupervised methods or collecting expert annotated datasets to fine-tune supervised summarization models. though these supervised models usually show improved performance compared to unsupervised methods, this often demands a large number of expert summaries, making it costly and less adaptable to new jurisdictions. this prompts us to ask how we can build an effective summarization system for a target jurisdiction without data annotation efforts. in this regard, we seek to investigate whether models trained on some other jurisdiction for which training data exists (which we call as source jurisdiction) can generate better summaries than unsupervised methods for the target jurisdiction. thus, we evaluate the crossjurisdiction generalizability of different legal case summarization systems and further propose an adversarial training-based setup to improve the crossjurisdiction transfer performance, paving the way for the development of summarization systems that are generalizable in real-world legal scenarios. we summarize our three primary research questions: rq1: in situations where reference summaries are unavailable to train supervised models for a specific target jurisdiction, can supervised summarization models, trained using data from a different jurisdiction (source jurisdiction) outperform unsupervised methods in generating more effective summaries? what criteria should guide the selection of the most suitable source jurisdiction for a specific target jurisdiction? which models exhibit better cross-jurisdiction summarization capabilities? rq2: can we leverage unlabeled judgment data (only documents, not reference summaries) from the target jurisdiction to improve transfer performance during training supervised summarization models on source jurisdiction? this approach is akin to unsupervised domain adaptation, where unlabelled target inputs are used when fine-tuning on a labelled source corpus. we employ an adversarial domain discriminator method @xcite to learn the jurisdiction-agnostic feature representations to facilitate cross-jurisdictional transfer. rq3: can extractive silver summaries on the target jurisdiction data, obtained using unsupervised summarization algorithms without manual human annotation, further improve transfer performance in addition to unlabelled target jurisdiction data? these silver summaries aid the decoder to grasp semantic nuances in target jurisdiction, complementing adversarial domain discriminator. to answer these questions, we conducted a comprehensive experimental analysis on three datasets from different jurisdictions. we find that supervised models trained on a different source jurisdiction can yield better summarization in the target jurisdiction, compared to unsupervised models. further, we observe that the choice of source dataset is highly influenced by jurisdictional similarity rather than the dataset size or abstractiveness (rq1). we witness an improvement in transfer performance using domain discriminator-based adversarial training. however, this improvement is contingent on the choice of backbone model. for instance, we observe a decline in transfer performance when applied to legal pre-trained pegasus in contrast to improvement in general pre-trained bart model (rq2). finally, we notice an improvement when adding silver summaries in scenarios when the dataset is extractive and there is less similarity between source-target jurisdictions (rq3). while extractive approaches excel in faithfully representing source document content, they face inherent challenges, including incomplete or incorrect discourse, coreference resolution issues, and a lack of context, impacting the readability of generated summaries @xcite . hence, there is a growing interest in exploring abstractive summarization methods for legal case summarization @xcite @xcite . @xcite applied transformerbased pre-trained abstractive models like bart, legal-led, and legal-pegasus to legal case summarization. to address long legal documents, moro and ragazzi 2022 proposed chunking input documents into semantically coherent segments, enabling these pre-trained transformer-based models to summarize lengthy documents without truncation. recent efforts also aim to enhance factual accuracy in generated summaries by employing entailment modules to select faithful candidates @xcite . additionally, there are efforts to understand the argumentative structure in the legal documents to improve the performance of legal case summarization models @xcite @xcite . in contrast to the approaches mentioned, which train and evaluate summarization models within specific jurisdictions, our work focuses on evaluating the cross-jurisdiction generalizability of legal case summarization systems. cross domain/dataset generalization: this assessment in nlp has been undertaken across various tasks, including constituent parsing @xcite , reading comprehension @xcite , named entity recognition @xcite , summarization @xcite , sentiment analysis @xcite and relation extraction @xcite , among others. recently, thakur et al. @xcite created a robust benchmark called beir (benchmarking ir), for evaluation of retrieval model generalization capabilities across various domains. in the legal domain, @xcite explored the cross-domain transferability of text generation models across four legal domains, including congressional bills, treaties, legal contracts, and privacy policies. however, the tasks they investigated within each domain exhibited significant variations in terms of generated text, such as titles in the privacy policy domain and actual summaries in congressional bills. in contrast, our work specifically focuses solely on summarizing legal case judgments. we consider each jurisdiction as a distinct domain characterized by its unique vocabulary and discourse writing style. furthermore, we explore strategies to enhance cross-jurisdictional transfer performance through adversarial learningbased domain adaptation. ## datasets there are very few publicly available english legal case summary corpora available. we use the following three datasets from prior works. (i) uk-abstractive dataset (abbreviated as 'uk-abs') @xcite : this dataset includes 793 case documents and their summaries from the uk supreme court, spanning from 2009 onwards. the abstractive summaries are sourced from the official press-released version available on the court's website. we use the provided split, with 693 documents for training and 100 for testing. ## models & evaluation metrics we use the following representative models from different legal case summarization methods in @xcite for our study. we briefly describe each method and refer the reader to the aforementioned work for implementation details. unsupervised extractive methods: (i) domainagnostic methods which employ graph-based salient sentence identification, namely lexrank @xcite and reduction (mihalcea and tarau, 2004), matrix factorization-based lsa @xcite and tf-idf based luhn @xcite . (ii) legal domain-specific methods include letsum @xcite , cas-esummarizer @xcite , which ranks sentences based on their tf-idf weights coupled with legal-specific features, maximal marginal relevance (mmr) @xcite uses iterative selection of predictive sentences and finally uses mmr to select the final summary sentences. supervised extractive methods: summarunner @xcite uses binary classification for sentence selection to determine whether it should be part of the summary. to build the training data for the classifier from a source document d and its abstractive reference summary s, each sentence in s is matched with the top sentences from d, chosen based on the highest average rouge-1, 2 and l scores. the extractive pseudo-reference summary is created by combining all selected sentences and are then used for supervised training of the model. ## rq1: cross-jurisdiction generalizability unlike previous studies @xcite that primarily evaluated these summarization algorithms in in-domain settings, where fine-tuning and evaluation occur on the same dataset, we focus on assessing their generalization capabilities across domains/jurisdictions. unsupervised models can be applied directly to summarize documents of any dataset (jurisdiction). for supervised models, we train the model on a specific dataset (source jurisdiction) and evaluate its performance on other datasets (target jurisdictions) in a blind zero-shot manner. results ( abstractive & hybrid models, due to their pretraining, demonstrated better performance compared to unsupervised methods even without any fine-tuning, as reflected over uk-abs and in-abs primarily due to their abstractive nature of datasets and pre-training objective (block 3 in tab.4). led is semi-pretrained as it has been initialized by repeatedly copying bart without undergoing endto-end general pre-training and then it has been fine-tuned on sec legal corpus to create legal-led, making it overfit to that specific domain, as reflected in its lower zero-shot performance. finally, abstractive & hybrid models fine-tuned on datasets from source jurisdictions generally performed well compared to the unsupervised methods, emphasizing the importance of legal adaptation during fine-tuning in addition to their pretraining either generally as with bert/bart-bert or legal-specific as with legal-pegasus (block 4 in choice of source jurisdiction: for both uk-abs and in-abs, training on each other's data yielded better results compared to training on bva-ext (e.g., bart applied on uk-abs test set, achieves 0.532/0.837 when trained on in-abs, compared to 0.519/0.835 when trained on bva-ext) due to their larger size, abstractive nature, and jurisdictional similarity between in-abs and uk-abs, as stated earlier in section 3. interestingly, for bva-ext, uk-abs proved to be a better source domain than in-abs (for most supervised methods), despite the latter's larger size and more extractive nature. this transferability is mainly attributed to the similarity between jurisdictions, as observed in section 3. this result shows that, while choosing the source jurisdiction (whose data will be used to train supervised summarization models), blindly choosing the jurisdiction with the largest dataset size is not an optimal approach. rather, the source jurisdiction should be chosen based on its lexical and semantic similarity with the target jurisdiction. to this end, the metrics we used in section 3 can be useful to quantify the similarity between jurisdictions. main takeaways: our findings reveal that supervised models trained on a different source jurisdiction can yield better summarization in the target jurisdiction, compared to unsupervised models. specifically, pre-trained models exhibit superior cross-jurisdictional generalizability due to their pretraining. legal-oriented training further enhances their generalization capabilities. the choice of the source dataset significantly influences the development of an effective system for a specific target jurisdiction, with jurisdictional similarity playing a critical role than dataset size and abstractiveness. ## rq2: leveraging unlabelled target jurisdiction corpus in this section, we explore whether we can improve the cross-domain performance of supervised abstractive summarization models using the unlabeled judgement documents from the target jurisdiction, while training the models on source jurisidiction judgement-summary pairs. to this end, we adopt domain-adversarial training from unsupervised domain adaptation literature @xcite . note that, we still do not use any reference summary from the target jurisdiction. in this technique, we additionally introduce an auxiliary jurisdiction classifier that tries to discriminate the source jurisdiction embeddings from the target ones. the encoder of the abstractive model is updated not only to facilitate the decoder in producing summaries for source jurisdiction documents, but is also adversarially trained to confuse the jurisdiction classifier to learn jurisdictioninvariant feature representations. put differently, we want our models to learn how to summarize the given input document with minimal encoding of the jurisdiction-specific information contained in the texts, facilitating model to generalize to documents from target jurisdictions. in our adaptation scenario, we have access to labelled judgement-summary pairs of a source jurisdiction {x s , y s } and unlabelled judgement corpus from target jurisdiction {x t }. let θ e and θ d be the encoder e and decoder d parameters of a summarization model. we introduce an auxiliary jurisdiction classifier j with parameters θ j , implemented as a linear layer, which takes the document rep-resentation from the encoder and performs binary classification to identify the document's jurisdiction. we train the classifier in an adversarial fashion to maximize the encoder's ability to capture information required for the summarization task while minimizing its ability to predict the jurisdiction. instead of the two-step adversarial min-max objective of gan @xcite , @xcite proposed to jointly optimize all the components using a gradient reversal layer (grl). the grl is inserted between the encoder and jurisdiction classifier, where it acts as the identity during the forward pass but, during the backward pass, scales the gradients flowing through by -λ, making the encoder receive the opposite gradients from the jurisdiction classifier. here, hyperparameter λ is the adaptation rate controlling the influence of the jurisdiction classifier on the encoder during training. thus, the overall objective function can be compactly represented as: ## rq3: leveraging silver summaries of target jurisdiction in this section, we explore whether incorporation of silver summaries from the target jurisdiction, obtained via unsupervised extractive summarization algorithms without the need for human annotations, can enhance effectiveness of cross-domain transfer. we employ the same domain-adversarial training framework, with a slight modification accounting for availability of judgment-summary pairs from both the source jurisdiction (expert-annotated) and the target jurisdiction (extractive silver summaries obtained via unsupervised algorithms). the first term in overall objective, as described in eqn. 1, is now used for target judgement-summary pairs as well. in other words, the x s in first term of eqn. 1 can now be x ? (i.e., either x s or x t ). this modification also strengthens the robustness of the adversarial learning approach. in the previous setup, the decoder's updates were solely influenced by source jurisdiction summaries, potentially leading to overfitting to source jurisdiction semantics and limiting its generalizability to new jurisdictions. the inclusion of silver summaries helps the decoder align with the semantics of the target jurisdiction, thereby facilitating more effective transfer performance. experimental setup: similar to rq2, we replicate the setup using three abstractive models bart, legal-pegasus and bert-bart for each dataset pair, designating one as the source jurisdiction with expert-annotated summaries and the other as the target jurisdiction with silver summaries. we obtain the silver summaries for the target jurisdiction using the mmr algorithm. foot_2 this can be attributed to the semantic similarity between uk-abs and in-abs, allowing effective learning without additional silver summaries from each other. however, while using bva-ext as train dataset, the semantics specific to uk-abs (in-abs) necessitate the inclusion of their silver summaries to facilitate the learning of target semantics. with respect to legal-pegasus on both uk-abs and in-abs as test set, we witness improvements in all training setups. for bva-ext as the test set, incorporation of silver summaries led to greater performance improvements across all three models and both source datasets consistently (e.g., 0.460/0.868 to 0.471/0.872 on uk-abs train set on bart). this lends to the nature of bva-ext dataset being extractive getting boost from addition of silver summaries which are extractive. main takeaways: adding silver summaries significantly improved performance, particularly for extractive datasets like bva-ext and when dealing with less similar jurisdictions (e.g., using bva-ext for training and in-abs/uk-abs for testing). this, along with our rq2 findings, highlights the efficacy of silver summaries in mitigating representational erasure in legal pre-trained models, as seen in the improved performance of legal-pegasus. ## case study tables 7 and 8 (in the appendix) illustrate some errors found in the summaries generated by different configurations of bart models for a specific indian supreme court judgement. one notable error involves indian jurisdiction-specific terms such as i.p.c. (indian penal code). models with blindzero transfer (rq1) and those with access to the source text only in grl (rq2) often output terms like k.p.c. or k.c. (in place of i.p.c.), demonstrating a limitation in the semantics of the decoder to understand indian-specific jargon. however, upon adding silver summaries (rq3), the decoder becomes exposed to such jargon, leading to the correct output of i.p.c. another common error across all models is incomplete sentences and coherence issues. in some cases, the model abruptly shifts to a new aspect without completing the previous one, potentially resulting in misrepresentations of the input document. these observations highlight the challenges in long text summarization, and the need for future improvement. ## conclusion our study reveals the following practical insights on building an effective legal summarization system for a target jurisdiction without annotated data: (i) fine-tuning on non-target datasets outperforms unsupervised methods, but success depends on the similarity between source and target jurisdictions.