Information Extraction from Legal Documents: A Study in the Context of Common Law Court Judgements (2020)

Abstract: ‘Common Law’ judicial systems follow the doctrine of precedent, which means the legal principles articulated in court judgements are binding in subsequent cases in lower courts. For this reason, lawyers must search prior judgements for the legal principles that are relevant to their case. The difficulty for those within the legal profession is that the information that they are looking for may be contained within a few paragraphs or sentences, but those few paragraphs may be buried within a hundred-page document. In this study, we create a schema based on the relevant information that legal professionals seek within judgements and perform text classification based on it, with the aim of not only assisting lawyers in researching cases, but eventually enabling large-scale analysis of legal judgements to find trends in court outcomes over time.

## introduction the law is reason free from passion 1 -but you'll have to dig through hundreds of pages to find it. in common law countries such as australia, a core legal principle is the doctrine of precedentevery court judgement contains legal rulings that are binding upon subsequent cases in lower courts, though how legal rulings apply in subsequent cases is dependent on the facts of the case. when preparing to give a legal opinion or argue a case, lawyers spend many long hours reading lengthy judgements to identify therein the precedents that are salient to the case at hand. this time-consuming manual process has formed a barrier to large-scale analysis of * meladel mistica, geordie z. zhang, and hui chia contributed equally to this paper. 1 aristotle, politica (politics by aristotle), written 350 b.c.e, translated by benjamin jowett legal judgements. even though thousands of court judgements are published in australia every year, 2 lawyers are only able to analyse small numbers of judgements, potentially missing broader trends hidden in the vast numbers of judgements that are published by the courts. there is a growing body of research at the intersection of law and natural language processing, including prediction of court opinion about a case @xcite , classification of legal text by legal topics or issues @xcite , and legal entity recognition @xcite . however, our ultimate goal is to assist lawyers in identifying sections of judgements relevant to their case at hand, as well as bulk analysis of cases to identify relationships between factual patterns and decision outcomes. for this reason, we model our initial study on the sentence-by-sentence identification of argumentation zones within academic and scientific texts @xcite . however, these zoning papers do not account for the complex document structure of legal judgements, which have the potential to be structured as multiple sub-documents within the one court decision (see section 3). the overall goal of the project is to automate the extraction of information from legal judgements, to assist lawyers to more easily and quickly identify the type of information that they are looking for from a large number of judgements. the project also aims to enable the large-scale analysis of judgements by legal researchers in order to identify trends or patterns that may be occurring within judgments, for example identifying patterns of facts that lead to particular results. this kind of analysis is relevant in predicting the outcome of complex cases and may also inform law reform. this part of the study reports on the initial phase of experimenting with the granularity of the annotation labels in developing our schema, as well as our initial experiments in automatically identifying these labels. ## background legal research as a broad term can include any form of research that is undertaken for the purpose of advancing legal advice, litigation or law reform, and can include research activities such as community surveys, comparative studies of legislation and the study of court judgments. this project focuses solely on the activity of studying court judgments, as it is a crucial component of legal research in common law countries. for lawyers and legal researchers, court judgments are a key source of data for the purpose of legal research, though legal research in general can encompass other sources of data, such as legislation, international treaties, government reports etc. when lawyers or legal researchers read a court judgment, what they are looking for is observations, opinions or decisions that the judge has made about how the law should be interpreted and applied in the particular context of the case before it. for example, what are the rules to resolve conflict between competing values, or what are the rules for resolving ambiguities of the meaning of a word in legislation? these observations, opinions and decisions by judges can be conceptualised as "law data" -data that legal researchers collect in order to understand how laws are being applied by courts to specific factual patterns and to predict how it may be applied in future scenarios. collecting data about how laws are interpreted is important at both the individual and the societal level. at the individual level, much of a lawyer's work is advising clients on what they need to do to comply with the law. lawyers will research past court judgments to collect data about how the law has been interpreted in similar factual situations, in order to make an informed opinion about how the law is likely to be applied to the case at hand. at the societal level, legal researchers in academia, regulatory agencies and government collect data on how laws are being interpreted and applied to specific facts, in order to assess whether laws are delivering the desired social outcomes. the field of legal research has conventionally relied mostly on qualitative data, and if there is quantitative data it is usually at a small scale. the reason for this is because "law data" is expressed in court judgments that are generally very long and complex free-form text. the only method for collecting "law data" has been through the manual reading of legal judgments by people with legal expertise. this is a very time-consuming process and therefore legal research has generally had to rely on small quantities of data. the contribution that nlp can make to the legal field is to enable the automatic extraction of "law data" from court judgements, to increase the number of court judgments that legal researchers can analyse. the challenge for this project has been the novelty of the task of extracting complex data from court judgments. there is no established schema for extracting information from court judgments. the schema proposed in this study is the result of a multi-disciplinary approach to merging the categories of data that are useful to legal researchers and lawyers, with the categories of information that can be accurately labelled using text classification. ## corpus development we developed our initial proof-of-concept corpus from court judgements from the high court of australia, foot_1 which is the highest court in the australian judicial system hierarchy. a court case may be decided by a single judge or a group of judges. in the case of a single judge, the court judgement is single-authored with one voice. when there are multiple judges, they can write a single judgement as a group, particularly if they are in agreement, or they can give separate reasons. in the latter case, the court judgement will then consist of multiple sets of reasons, structured as sub-components from the different judges, which together make up the entire judgement for that court case. to legal domain experts, there are general patterns or sequences by which different types of information tend to appear within a judgement. however there is a high degree of variation between court judgements according to the writing style of the judge. for instance, one common document pattern begins with the explanation of the facts of the case, followed by the reasoning on how the rele-label description fact specific facts of that case, e.g. the applicant entered australia as an unauthorised maritime arrival on 5 september 2011. reasoning legal principles considered, e.g. the question that arises is whether the tribunal failed to consider that the applicant faced a real probability of irreparable harm. conclusion outcome of the case, e.g. the tribunal committed a jurisdictional error, the appeal should be allowed. figure 1 : description of the label set vant legal principles were applied, and then ending with their conclusion. but this is not always the case. some judges will state their conclusions at the beginning, and then provide a detailed examination of the facts and legal reasoning. where there are multiple sets of reasons within a single judgement, each set of reasons will have its own structure particular to that judge's writing style. we limit our corpus to immigration law cases, and randomly selected 55 of these high court judgements. these 55 documents contain over 9.5k sentences in total. each of them was annotated at the sentence level with either fact, rea-soning or conclusion, which capture different aspects of the case as shown in figure 1 . in this initial corpus, reasoning made up half of the labelled sentences. of the remaining sentences, three quarters were labelled fact, and one quarter conclusion. the fact and conclusion segments of the case are usually what lawyers are most interested in. these portions of the document (judgement) contain unique details pertaining to the case, while the reasoning category is a combination of original insights of this case and a recapitulation of previous relevant judgements. annotation for the annotations, we had 1 primary annotator (annotator a), a qualified lawyer and legal researcher, who marked up all of the sampled high court judgements. anno-tator a had a label distribution of fact: 38%, reasoning: 50%, and conclusion: 12%. we also had 2 secondary annotators (annotators b and c): the first is a practising immigration lawyer, and the second has some legal training, but is not a fully qualified lawyer. we randomly selected 3 documents (judgements) for the secondary annotators to mark up. this made up 5% of the number of sentences of the whole corpus. of those sentences, there were no three-way disagreements between the annotators. the cohen's kappa (κ) between all three annotators shows very good 2-way agreement between all pairs of annotators. the inter-annotator agreement between a-b and b-c were 0.70, and between a-c was 0.73. a large majority of the 2-way disagreements involved reasoning, with 81.5% of the disagreements being reasoningvs-fact and reasoning-vs-conclusion, split roughly 50:50. ## experiments in order to assess the feasibility of using our corpus in a supervised setting, we perform experiments using a range of different models for sentence-level classification. the goal is to have a reasonable understanding of how difficult the task is, both in terms of our initial schema and training data size. ## discussion and future work in this paper, we have presented the preliminary investigations of our interdisciplinary collaboration. the main focus was to scope out the areas in which nlp can assist in the task of interpreting legal judgments -a task that every lawyer must do in researching a case. the main contribution of this paper is developing and testing the annotation schema. in future work, we aim to extract trends over time for a given aspect of the annotation, e.g. how the presentation reasoning changes over time as new cases are judged with each new @xcite given that australia has a common law system, these judgements in effect shape the interpretation and understanding of the law and set a precedence for subsequent cases. the results of the sentence-level text classification are promising despite the inherent confusability within the reasoning class: even professional lawyers with years of training can disagree in ascertaining whether a sentence is indeed a reason-ing rather than a conclusion or in some cases a reasoning or a fact sentence, as there can be elements of either within a reasoning sentence. although the results do show promise, in future work, we intend to experiment with the annotation schema to explore more detailed sub-categories under reasoning. this will assist us in identifying more targeted zones within the judgements, which may better assist in legal information extraction tasks, and in better characterising the structure of these legal documents. from an application perspective, we plan to test the newly released legalbert @xcite and compare this to our adaptation of a domain-specific bert and xlnet for legal texts. we note that legalbert was pre-trained on a variety of legal texts that are different from the legal texts in our database, which consisted solely of australian court judgments. the data used to pretrain legalbert included legislation and contracts, which are different to court judgments in terms of structure and content. also, the data used to pretrain legalbert was from multiple legal jurisdictions, being the united states, united kingdom and europe, with each jurisdiction having unique nuances to the language used in its legal texts. given these differences between our data and the training data of legalbert, it remains an open question as to whether legalbert would have any advantage over bert, and whether a custom-tuned bert for our purposes may be more advantageous.