A sy L ex: A Dataset for Legal Language Processing of Refugee Claims (2023)

Abstract: Advancements in natural language processing (NLP) and language models have demonstrated immense potential in the legal domain, enabling automated analysis and comprehension of legal texts. However, developing robust models in Legal NLP is significantly challenged by the scarcity of resources. This paper presents AsyLex, the first dataset specifically designed for Refugee Law applications to address this gap. The dataset introduces 59,112 documents on refugee status determination in Canada from 1996 to 2022, providing researchers and practitioners with essential material for training and evaluating NLP models for legal research and case review. Case review is defined as entity extraction and outcome prediction tasks. The dataset includes 19,115 gold-standard human-labeled annotations for 20 legally relevant entity types curated with the help of legal experts and 1,682 gold-standard labeled documents for the case outcome. Furthermore, we supply the corresponding trained entity extraction models and the resulting labeled entities generated through the inference process on AsyLex. Four supplementary features are obtained through rule-based extraction. We demonstrate the usefulness of our dataset on the legal judgment prediction task to predict the binary outcome and test a set of baselines using the text of the documents and our annotations. We observe that models pretrained on similar legal documents reach better scores, suggesting that acquiring more datasets for specialized domains such as law is crucial.

## introduction while large language models (llms) have gained significant attention in nlp, many real-world applications have yet to leverage their capabilities fully. one of the main challenges lies in the fact that these models heavily rely on training or fine-tuning refugee law is a specific area of law that currently lacks publicly available datasets and benchmarks to evaluate nlp applications and compare their performance with standard baselines. lawyers specializing in refugee law devote a significant amount of time to reviewing past cases to prepare for new ones. this process of legal research and case review is not only time-consuming but also expensive. given the limited resources and funding available in refugee law, where a majority of claimants rely on legal aid, it becomes even more critical to find ways to free up time and speed up the case review process. the availability of a comprehensive dataset in refugee law would have significant benefits. it would enable lawyers to efficiently search and analyze relevant cases, extract valuable insights, and apply them to new cases. by streamlining the case review process, lawyers could save time, reduce costs, and ultimately provide more efficient legal assistance to those seeking refugee protection. the absence of a refugee law dataset affects lawyers and claimants, as refugee status determination (rsd) plays a pivotal role in determining refugee protection. refugee protection decisions directly impact the lives of the approximately 4.6 million asylum seekers worldwide as of mid-2022 foot_0 . to provide a concrete example, 48,014 new claims and 10,055 appeals were filed foot_1 in canada alone in 2021. processing times of refugee claims range from a few months to several years. this variation in processing times exacerbates the uncertainty and instability experienced by asylum seekers. by enhancing access to data, we aim to promote fairness, effectiveness, and improved outcomes for individuals seeking refuge globally. automating aspects of case review and providing high-quality data can facilitate better access to legal counseling and support for claimants. our dataset is particularly valuable because we collected a total of 76,523 gold annotations (both from human labeling and rule-based) and trained specific information extraction models to gain insights and structure the dataset (section 3) to facilitate two tasks: (1) entity extraction for legal search and (2) legal judgment prediction. both tasks, according to lawyers we consulted, have the potential to help draft new claims for refugee protection. we experiment with judgment prediction on different baseline masked language models to showcase the dataset's potential (section 4). asylex contributes directly to applying nlp in high-stakes legal domains and as a novel research asset to the nlp research community. specifically, our contributions are as follows: in addition to the data, the code of the experiments is public and can be found at: https: //github.com/clairebarale/asylex . ## related work legal nlp legal nlp is an active and promising field of research. legal information is predominantly conveyed through text, with crucial details typically documented in written form. in principle, this makes the law an ideal domain for leveraging natural language processing techniques. however, applying nlp in the legal domain brings about notable challenges. these challenges arise from the distinct structure, specialized vocabulary, contextual nuances, importance of legal citations, and high stakes in legal applications. a wide range of tasks and functionalities have been explored in legal nlp (zhong et al., 2018), such as legal summarization (elaraby and litman, 2022; aumiller et al., 2022), legal information extraction and retrieval (bommarito ii et al., 2021; brüninghaus and ashley, 2001; chalkidis et al., 2021), legal question answering, automatic text generation, text classification @xcite , legal judgment prediction @xcite @xcite or multitask benchmarks @xcite . with this dataset, we target two tasks for case review: (i) entity extraction and (ii) legal judgment prediction. legal judgment prediction is a common task in legal ai. work has been conducted on legal datasets gathering decisions from the supreme court of the united states @xcite @xcite or the european court of human rights @xcite . similar studies have been conducted on asylum decision data sets @xcite @xcite . however, most of these works rely on statistical data rather than text documents. one reason is the lack of specialized language resources and datasets available in law. we recognize this gap and aim to address it by providing a new dataset of legal texts, thereby contributing to legal nlp research. legal information extraction and classification are difficult tasks because approaches are mostly supervised and, therefore, require precisely labeled datasets. legal nlp datasets while large language models have recently made natural language tasks more accessible, their utilization in the legal domain has been relatively limited. one of the reasons behind this is the need to adapt these models to legal applications @xcite . these are typically trained on general corpora and may not fully capture the intricacies of legal text, as shown by previous studies @xcite . in our information extraction pipeline, we observed that legalbert outperformed roberta in accuracy, highlighting the significance of fine-tuning models specifically for legal tasks. a range of datasets for legal nlp tasks in english exists in tax law @xcite , european legislation and the european court of human rights @xcite , corporate and contract law @xcite , supreme court cases and us court cases @xcite . lexfiles is the most comprehensive to date @xcite . however, no such dataset is available for refugee decisions. to further investigate how well large language models perform on specialized tasks and to ensure advances also benefit the field of refugee law, we curated the firstever dataset of refugee status determination cases in english. this dataset will serve as a valuable resource to train and evaluate models tailored to the unique challenges and requirements of refugee law. the dataset contains labeled data suited for two nlp tasks: (1) entity extraction and (2) legal judgment prediction. for the first task of interest, we offer a total of 19,115 human-labeled annotated samples, data spanning across 22 categories, an additional 57,408 annotations using rule-based extraction on the case cover, along with 6,278,028 silver-standard labeled samples inferred through a named-entity recognition model presented in details in @xcite . for the case cover, which constitutes the first page of each case and contains meta-data about each document, we annotated 346 documents. details of the number of labeled samples per category are shown in for the second task of interest, we provide a comprehensive collection of data for legal judgment prediction, which comprises a test set with 1,682 gold-standard classified documents and a train set with 30,944 classified documents. in addition, we provide the raw text of the document cases in full text and split it into sentences, each presented with their corresponding case number (decisionid). ## experiments: text classification for outcome prediction figure 4 : distribution of the documents over the years by the date of the decision of success. for judges, engaging in this task offers the potential to analyze their decisions, leading to potential improvements and increased consistency. furthermore, researchers can utilize this task to assess the fairness of refugee status determination (rsd) decisions and uncover potential biases. task: judgment prediction the task consists of predicting whether the decision outcome was positive (refugee status granted) or negative (the claim was rejected). ## conclusion we introduce a high-quality dataset of annotated refugee claims to streamline case review procedures and contribute to the overall effectiveness of legal practitioners' workflows. furthermore, we explore the performance of generic models in specialized domains, particularly in the context of refugee law. this dataset aims to not only enable advancements in legal nlp but also specifically address the scarcity of resources in refugee law. by providing researchers and practitioners access to high-quality labeled data, we hope to foster further research and innovation in this crucial area, ultimately facilitating the development of intelligent legal systems with applications in refugee law and beyond.