An Empirical Study on Cross- X Transfer for Legal Judgment Prediction (2022)

Abstract: Cross-lingual transfer learning has proven useful in a variety of Natural Language (NLP) tasks, but it is understudied in the context of legal NLP, and not at all in Legal Judgment Prediction (LJP). We explore transfer learning techniques on LJP using the trilingual Swiss-Judgment-Prediction (SJP) dataset, including cases written in three languages. We find that Cross-Lingual Transfer (CLT) improves the overall results across languages, especially when we use adapter-based fine-tuning. Finally, we further improve the model’s performance by augmenting the training dataset with machine-translated versions of the original documents, using a 3× larger training corpus. Further on, we perform an analysis exploring the effect of cross-domain and cross-regional transfer, i.e., train a model across domains (legal areas), or regions. We find that in both settings (legal areas, origin regions), models trained across all groups perform overall better, while they also have improved results in the worst-case scenarios. Finally, we report improved results when we ambitiously apply cross-jurisdiction transfer, where we further augment our dataset with Indian legal cases.

## introduction rapid development in cross-lingual transfer (clt) has been achieved by pre-training transformer-based models in large multilingual corpora @xcite , where these models have state-of-the-art results in multilingual nlu benchmarks @xcite . moreover, adapter-based fine-tuning @xcite has been proposed to minimize the misalignment of multilingual knowledge (alignment) when clt is applied, especially in a zero-shot fashion, where the target language is unseen during training. clt is severely understudied in legal nlp applications except for * equal contribution. to the best of our knowledge, clt has not been applied to the legal judgment prediction (ljp) task @xcite @xcite , where the goal is to predict the verdict (court decision) given the facts of a legal case. in this setting, positive impact of cross-lingual transfer is not as conceptually straight-forward as in other general applications (nlu), since there are known complications for sharing legal definitions and interpreting law across languages @xcite @xcite . following the work of @xcite , we experiment with their newly released trilingual swiss-judgment-prediction (sjp) dataset, containing cases from the federal supreme court of switzerland (fscs), written in three official swiss languages @xcite . the dataset covers four legal areas (public, penal, civil, and social law) and lower courts located in eight regions of switzerland (zurich, ticino, etc.), which poses interesting new challenges on model robustness / fairness and the effect of cross-domain and crossregional knowledge sharing. in their experiments, @xcite find that the performance in cases written in italian is much lower compared to the rest, while also performance varies a lot across regions and legal areas. ## related work legal judgment prediction (ljp) is the task, where given the facts of a legal case, a system has to predict the correct outcome (legal judgement). many prior works experimented with some forms of ljp, however, the precise formulation of the ljp task is non-standard as the jurisdictions and legal frameworks vary. @xcite ; @xcite predict the plausible violation of european convention of human rights (echr) articles of the european court of human rights (ecthr). @xcite study chinese criminal cases where the goal is to predict the ruled duration of prison sentences and/or the relevant law articles. another setup is followed by @xcite ; @xcite ; @xcite , which use cases from supreme courts (french, indian, swiss, respectively), hearing appeals from lower courts relevant to several fields of law (legal areas). across tasks (datasets), the goal is to predict the binary verdict of the court (approval or dismissal of the examined appeal) given a textual description of the case. none of these works have explored neither cross-lingual nor cross-jurisdiction transfer, while the effects of cross-domain and crossregional transfer are also not studied. cross-lingual transfer (clt) is a flourishing topic with the application of pre-trained transformer-based models trained in a multilingual setting @xcite @xcite excelling in nlu benchmarks @xcite . adapter-based fine-tuning @xcite has been proposed as an antimeasure to mitigate misalignment of multilingual knowledge when clt is applied, especially in a zero-shot fashion, where the target language is unseen during training (or even pre-training). meanwhile, clt is understudied in legal nlp applications. @xcite experiment with standard fine-tuning, while they also examined the use of adapters @xcite for zeroshot clt on a legal topic classification dataset comprising european union (eu) laws. they found adapters to achieve the best tradeoff between effectiveness and efficiency. their work did not examine the use of methods incorporating translated versions of the original documents in any form, i.e., translate train documents or test ones. recently, @xcite used an updated, unparalleled version of chalkidis et al. dataset to study nmt -augmented clt methods. other multilingual legal nlp resources @xcite have been recently released, although clt is not applied in any form. ## motivation and challenges for cross-jurisdiction transfer legal systems vary from country to country. although they develop in different ways, legal systems also have some similarities based on historically accepted justice ideals, i.e., the rule of law and human rights. switzerland has a civil law legal system (walther, 2001), i.e., statutes (legislation) is the primary source of law, at the crossroads between germanic and french legal traditions. contrary, india has a hybrid legal system with a mixture of civil, common law, i.e., judicial decisions have precedential value, and customary, i.e., islamic ethics, or religious law @xcite . the legal and judicial system derives largely from the british common law system, coming as a consequence of the british colonial era @xcite @xcite . based on the aforementioned, cross-jurisdiction transfer is challenging since the data (judgments) abide to different law standards. although the supreme court of india (sci) rules under the indian jurisdiction (law), while the federal supreme court of switzerland (fscs) under the swiss one, we hypothesize that the fundamentals of law in two modern legal systems are quite common and thus transferring knowledge could potentially have a positive effect, and thus it is an experiment worth considering, while we acknowledge that from a legal perspective equating legal systems is deeply problematic, since the legislation, the case law, and legal practice are different. our empirical work and experimental results shows that cross-jurisdiction transfer in this specific setting (combination of swiss and indian decisions) has a positive impact in performance, but we cannot provide any profound hypothesis neither we are able to derive any conclusions on the importance of this finding on legal literature and practice. we leave these questions in the hands of those who can responsibly bear the burden, the legal scholars. rq2: do models benefit or not from cross-regional and cross-domain transfer? in section 3.4, we find that models benefit from cross-regional transfer across all cases, since they are exposed to (trained in) many more documents (cases). we believe cross-regional diversity is not a significant aspect, compared to the importance of the increased data volume and language diversity. cross-domain transfer is beneficial in three out of four cases (legal areas), with comparable results on penal (criminal) law, where the application of law seems to be more straight-forward / standardized (higher performing legal area). cross-regional and cross-domain transfer lead to more robust models. rq3: can we leverage data from another jurisdiction to improve performance? in section 3.5, we find that cross-jurisdiction transfer in our specific setup, i.e., very similar ljp tasks, is beneficial. again, we believe that this is mostly a matter of additional unique data (cases), rather than a matter of jurisdictional similarity. cross-jurisdiction transfer leads to a better performing model. rq4: how does representational bias (wrt. language, origin region, legal area) affect model's performance? we observe that representational bias -in non-extreme cases (e.g., w.r.t. language) -does not always explain performance disparities across languages, regions, or domains, and other characteristics also need to be considered. ## ethics statement the scope of this work is to study ljp to broaden the discussion and help practitioners to build assisting technology for legal professionals and laypersons. we believe that this is an important application field, where research should be conducted @xcite to improve legal services and democratize law, while also highlight (inform the audience on) the various multi-aspect shortcomings seeking a responsible and ethical (fair) deployment of legal-oriented technologies. in this direction, we study how we could better exploit all the available resources (from various languages, domains, regions, or even different jurisdictions). this combination leads to models that improve overall performance -more robust models -, while having improved performance in the worstcase scenarios across many important demographic or legal dimensions (low-resource language, worst performing legal area and region). nonetheless, irresponsible use (deployment) of such technology is a plausible risk, as in any other application (e.g., online content moderation) and domain (e.g., medical). we believe that similar technologies should only be deployed to assist human experts (e.g., legal scholars in research, or legal professionals in forecasting or assessing legal case complexity) with notices on their limitations. the main examined dataset, swiss-judgment-prediction (sjp), released by @xcite , comprises publicly available cases from the fscs, where cases are pre-anonymized, i.e., names and other sensitive information are redacted. the same applies for the second one, indian legal documents corpus (ildc) of @xcite . cated in