Automated Refugee Case Analysis: An NLP Pipeline for Supporting Legal Practitioners (2023)

Abstract: In this paper, we introduce an end-to-end pipeline for retrieving, processing, and extracting targeted information from legal cases. We investigate an under-studied legal domain with a case study on refugee law Canada. Searching case law for past similar cases is a key part of legal work for both lawyers and judges, the potential end-users of our prototype. While traditional named-entity recognition labels such as dates are meaningful information in law, we propose to extend existing models and retrieve a total of 19 categories of items from refugee cases. After creating a novel data set of cases, we perform information extraction based on state-of-the-art neural named-entity recognition (NER). We test different architectures including two transformer models, using contextual and non-contextual embeddings, and compare general purpose versus domain-specific pre-training. The results demonstrate that models pre-trained on legal data perform best despite their smaller size, suggesting that domain-matching had a larger effect than network architecture. We achieve a F1- score superior to 90% on five of the targeted categories and superior to 80% on an additional 4 categories.

## introduction the retrieval of similar cases and their analysis is a task at the core of legal work. legal search tools are widely used by lawyers and counsels to write applications and by judges to inform their decision-making process. however, this task poses a series of challenges to legal professionals: (i) it is an expensive and time-consuming task that accounts for 30% of the legal work on average @xcite , (ii) databases can be very large, with legal search tools gathering billions of documents, and (iii) selection of cases can be imprecise and may return many irrelevant cases, which creates the need to read more cases than necessary. in canada, from the date of the first claim to the final decision outcome, a claimant can expect to wait 24 months for refugee claims and 12 months for refugee appeals foot_0 . long processing times are due to a significant backlog and to the amount of work required from counsels that help claimants file their claims, and who are frequently legal aid or ngo employees. we find that these challenges are well-suited for nlp-based solutions and investigate the feasibility of automating all steps of the legal search for past similar cases. we construct an end-to-end pipeline that aims at facilitating this multi-step process, thereby supporting and speeding up the work of both lawyers and judges in refugee status determination (rsd). we provide a level of granularity and precision that goes beyond that of existing legal search tools such as westlaw, lexisnexis, or refworld foot_1 @xcite , which operate at the document level. refworld is an online database maintained by the united nations which helps retrieve relevant precedent cases and legislation. however, the level of precision with which one can search for cases is limited. moreover, our pipeline guarantees increased transparency, enabling end users to choose the criteria of legal search they find most relevant to their task among the proposed categories that act as filters for a search. specific literature studying refugee law and ai is sparse. attention has been given to the classification and prediction of asylum cases in the united states @xcite . on canadian data, research has been conducted to analyze the disparities in refugee decisions using statistical analysis @xcite @xcite . however, those studies rely mostly on tabular data. we propose to work directly on the text of refugee cases. to the best of our knowledge, no previous work implements an end-to-end pipeline and state-of-the-art nlp methods in the field of refugee law. we provide an nlp-based end-to-end prototype for automating refugee case analysis built on historical (already decided) cases, which are currently available only in unstructured or semi-structured formats, and which represent the input data to our pipeline. the end goal of our approach is to add structure to the database of cases by extracting targeted information described in the pipeline described in figure 1 begins by searching and downloading cases (information retrieval, paragraph 4.1), pre-processing them (paragraph 4.2), extracting items previously identified as relevant by legal professionals. it then outputs a structured, precise database of refugee cases (information extraction, paragraph 4.3). in the information extraction step, we test different training and pre-training architectures in order to determine the best methods to apply to the refugee case documents. we construct each step with the aim of minimizing the need for human effort in creating labeled training data, aiming to achieve the best possible accuracy on each extracted information item. we discuss technical choices and methodologies in section 5. finally, we evaluate the information extraction step on precision, recall, and f1 score, and present detailed results in section 6. we demonstrate that annotation can be sped up by the use of a terminology base while incorporating domain knowledge and semi-automated annotation tools. we find that domain matching is important for training to achieve the highest possible scores. we reach satisfactory token classification results on a majority of our chosen categories. the contributions of this paper are as follows: at the core of the ongoing refugee crisis is the legal and administrative procedure of refugee status determination (rsd), which can be summarized in three sub-procedures: (i) a formal claim for refugee protection by a claimant who is commonly supported by a lawyer, (ii) the decision-making label a description example case cover general date 1,219 absolute or relative dates or periods date of the hearing and date of the decision gpe 871 cities, countries, regions place of the hearing org 278 tribunals "immigration appeal division", "refugee protection division" person 119 names name of the panel and counsels ## research approach our approach is guided by investigating the hypothesis that ner methods can be used to extract structured information from legal cases, i.e. we want to determine whether state-of-the-art methods can be used to improve the transparency and processing of refugee cases. consistency of the decision-making process and thorough assessment of legal procedure steps are crucial aspects ensuring that legal decision outcomes are transparent, high-quality, and well-informed. consequently, key research questions we need to address include: training data requirements how many labeled samples are needed? can keyword-matching methods or terminology databases be leveraged to reduce the need for human annotation? extraction what methods are best suited to identify and extract the target information from legal cases? replicability to what extent might our methods generalize to other legal data sets (other legal fields or other jurisdictions)? pre-training how important is the pre-training step? how important is domain matching: do domain-specific pre-training perform better than general-purpose embeddings, despite their smaller sizes? architectures how important is the architecture applied to the information extraction tasks, in terms of f1 score, precision, and recall? in this section, we detail each step of the pipeline as presented in figure 1 and how it compares to the current legal search process. subsequently, in 5 we describe the training data creation process, and the network architectures tested. the code for our implementation and experiments can be found at https://github.com/clairebarale/ refugee_cases_ner . ## results and evaluation our experimental results are presented in for the case cover, we obtain satisfactory results on all labels with f1 scores above 90% for three of them and 84.78% for name extraction. apart from names, cnn architectures perform better, with dates achieving the highest score with randomly initialized embeddings. we explain this with the specific layout of this page (annex a). the only gain of using a transformer-based model is to achieve a higher recall compared to the cnnbased architectures. for the main text, results vary across labels: we obtain a score above 80% for date, gpe, person, org with the best score on roberta, but legal-bert-base-uncased scores lower than 60% on explanation, law, law_case. overall, when using transformers, we observe a better precision-recall balance. results on three labels determination, law_report, norp are unreliable because of the limited sample both for training and testing. determination appears only once per case, and law_report appears in a few cases only. further annotation would require selecting the paragraphs of cases where these items appear to augment the size of the sample. we leave this task to future work. explanations for other low scores are partly to be found in the tokenization errors reported during the human-labeling task. figure 2 shows an example of wrong tokenization on two categories law and law_case for which we believe bad tokenization is the primary explanation for low scores (similarly reported by sanchez). in the first sentence of the figure, words are not correctly split between "under" and "section" and between the section number and "of". on the lower part of the figure, sentence tokenization does not correctly split the case reference as it is confused by the dot present in the middle. in this example, the case name is displayed as three different sentences, making the labeling task impossible. the most appropriate pre-training varies across labels: for categories on which cnn performs best such as credibility, doc_evidence, law, we find that fine-tuning static vectors performs better than randomly initialized embeddings or dynamic vectors, which suggests that context was not essential when retrieving spans of text (pre-training relies on tri-grams). this could derive from the methods of annotation that were terminology-based for those labels. while the target items may contain particular vocabulary such as "personal information form" for doc_evidence, context is of minimal importance since those phrases would not appear in another context or under another label. on the contrary, context seems much more important for retrieving procedural steps (procedure), which is the only category where the pre-training layer with contextual embeddings significantly increases the f1 score. in the majority of categories, we find that the content of the pre-training is important (claimant_event, in other categories, roberta performs better than legalbert and cnns, suggesting that the size of the pre-trained model is more important than domain matching. while legalbert has a size of 12gb, roberta is over 160gb and outperforms legalbert on traditional ner labels (gpe, org, person and also claimant_info, law_case). ## related work because of the importance of language and written text, applications of nlp in law hold great promise in supporting legal work, which has been extensively reviewed by zhong et al.. however, because of the specificity of legal language and the diversity of legal domains, as demonstrated in our work with the results on legalbert-based transformer, general approaches aiming at structuring legal text such as @xcite or general legal information extraction @xcite are unfit for specific domains such as international refugee law and are not able to achieve a high degree of granularity. earlier methods of statistical information extraction in law include the use of linear models such as maximum entropy models @xcite and hidden markov models @xcite . however, state-of-the-art results are produced by methods able to capture some context, with an active research community investigating the use of conditional random fields @xcite @xcite and bilstms @xcite @xcite @xcite for legal applications. scope and performance increased with the introduction of new architectures of deep learning using recurrent neural networks (rnn), cnns, and attention mechanisms as demonstrated by chalkidis et al., even though we find that transformers do not always perform best on our data. we therefore fo- ## conclusion and future work our pipeline identifies and extracts diverse text spans, which may vary in quality across different categories. we acknowledge that certain entities we identify are more valuable than others for legal search purposes. additionally, due to the complexity of the text, some noise is to be expected. however, this noise does not hinder the search for relevant items. users have the flexibility to search and retrieve cases using any combination of our 19 categories of flagged entities. additionally, work is required for the evaluation of the prototype by legal practitioners beyond traditional machine learning metrics @xcite . however, we believe the work presented here is an important first step and has the potential to be used for future nlp applications in refugee law. our approach provides significant contributions with newly collected data, newly created labels for ner, and a structure given to each case based on lawyers' requirements, with nine categories of information being retrieved with an f1 score higher than 80%. compared to existing case retrieval tools, our pipeline enables endusers to decide what to search for based on defined categories and to answer the question: what are criteria of similarity to my new input case ?