Challenges for Information Extraction from Dialogue in Criminal Law (2021)

Abstract: Information extraction and question answering have the potential to introduce a new paradigm for how machine learning is applied to criminal law. Existing approaches generally use tabular data for predictive metrics. An alternative approach is needed for matters of equitable justice, where individuals are judged on a case-by-case basis, in a process involving verbal or written discussion and interpretation of case factors. Such discussions are individualized, but they nonetheless rely on underlying facts. Information extraction can play an important role in surfacing these facts, which are still important to understand. We analyze unsupervised, weakly supervised, and pre-trained models’ ability to extract such factual information from the free-form dialogue of California parole hearings. With a few exceptions, most F1 scores are below 0.85. We use this opportunity to highlight some opportunities for further research for information extraction and question answering. We encourage new developments in NLP to enable analysis and review of legal cases to be done in a post-hoc, not predictive, manner.

## introduction our criminal justice system struggles to balance "the value of treating like cases alike, and the value of treating each case individually." @xcite in criminal law, machine learning has been proposed as a tool to improve consistency in decision making, but to date, research efforts have primarily focused on codified justice -processes that make a determination given a limited set of case factors and using specifiable rules, such as a risk assessment used for a probation classification. however, various legal contexts balance a standard of codified justice with a standard of equitable justice, which requires decision-makers to apply moral principles to individuals' unique situations. how can natural language processing aid equitable justice? equitable justice centers human discretion and the uniqueness of each individual, but nonetheless is based on factual information. the facts of each case are typically discussed and interpreted through dialogue. often, the dialogue produces transcripts, which are available as public records. usually, the sheer length of transcribed conversational text all but prohibits any meaningful form of quantitative review, because of the immense effort involved in manually annotating case factors. nlp methods for information extraction over speech can assist in identifying the underlying facts of a case from hearing transcripts. the factors can then be used in statistical analyses of a decision-making process to (a) provide historical understanding over case records that are otherwise locked away in a filing cabinet, and (b) identify specific outlier cases for reconsideration of fair and equitable decision-making where human capacity for review is constrained. by applying information extraction post-hoc rather than filling in a data table or computing a risk score at the time of a hearing, the decision-maker retains full autonomy in conducting a legal process using their own discretion. in this role, information extraction supplements, but never fully supplants, the need for dialogue and transcripts. a broad set of stakeholders can then contribute to identifying the factors that may be relevant in comparing cases. 1 we present a case study of the capabilities of information extraction methods for dialogue and identify areas for further research in the criminal law context. we have obtained a nearly complete dataset of 35,105 parole hearing transcripts from the state of california for individuals serving life sentences between 2007 and 2019. the california parole hearing system serves as a useful case study because (1) california has one of the largest prison systems in the u.s., (2) the hearings are transcribed and available on the public record, (3) the hearings are relatively long (about 20,000 words) and illustrate the challenges of long dialogue, (4) human annotation of the hearings is expensive, and ( 5 ) the hearings are one continuous dialogue in a single sitting between a decision-maker and a parole candidate, with brief statements from the candidate's attorney. in comparison, criminal trials are much longer, present many forms of exhibits which are often not digitally available, and contain many additional complexities. we have identified 11 case factors representative of the types of features (binary, multi-class, date, and numerical) that are relevant to the parole decision-making system and illustrate a range of challenges in information extraction. we evaluate three families of models on this task: (1) an unsupervised data programming paradigm @xcite extended to weak supervision, (2) pretrained question answering models based on distil-bert @xcite and longformer @xcite , and (3) classification models based on bert @xcite that are each fine-tuned to predict a single task. most models fall below an f1 score of 0.85 for most of the features. the different feature types challenge each of the models in different ways. data programming remains a largely rule-based approach and works best when the keywords indicative of a label are clear, such as the penal code or a numerical education score. pre-trained question answering models maintain or improve performance on most categories, except for boolean questions, which remains an area of active development. surprisingly, all models perform poorly on extracting the risk assessment score, which relies on three simple keywords "low," "moderate," or "high." information extraction from long dialogues remains an open challenge, especially when the extraction tasks are not entity-based. we call on research in information extraction to move beyond entity-based tasks in order to tackle the range of tasks relevant for legal dialogue. we also emphasize the need for all methods to handle longer context windows. long context windows are not merely a byproduct of underdeveloped retrieval methods; they are inherent to the level of personal detail required to apply equitable justice. ## data our text corpus consists of 35,105 parole hearing transcripts, averaging 18,499 words each, covering 15,852 unique individuals from 2007-2019 parsed from pdf documents. each hearing is attended by a presiding and a deputy parole commissioner, the parole candidate, and typically an attorney for the candidate. often, hearings also include a district attorney representative from the county of the commitment offense, who makes a statement, and a victim or their next-of-kin, who may make a statement. some hearings are attended by visitors who do not participate in the dialogue. the majority of the conversation occurs between the parole candidate, their attorney, and the presiding commissioner. ## human performance to compute a human performance baseline for the reliability with which the selected features can be extracted from transcripts, we use cohen's κ coeffi- ## results because snorkel, wslf, and task-ft models are trained for a given class, their results are given in the space of the label of the task, whether that is a binary label or a date, for example. however, both qa1 and qa2 models are extractive question answering models, i.e. the answers returned are taken from the text of the hearing. in some cases, the text needs additional processing to be transformed into a label. the transformation may be human intervention, such as in the case of edu level, where the extractive answer "ninth grade" and needs to be translated into a categorical answer "no high school or ged." in other cases, such as with dates, the transformation can be partially or fully automated, such as by parsing answers like "march the 6th, 2019" into the mepd year, 2019, using tools such as sutime @xcite . overall, wslf does well on most classification tasks, though it is beaten by qa2 on risk assess and by the more powerful classifier task-ft on off mur1. qa2 is strongest on dates and generally outperforms qa1. task-ft performs best on a variety of tasks, but surprisingly, it does not always improve over wslf and snorkel, even though its training process uses the very labels produced by the data programming methods, but augmented with even more information, the underlying text itself. ## discussion our case study on extracting features from parole hearings illustrates many outstanding challenges in question answering, information extraction, and text classification. addressing these challenges is key to using nlp for positive impact in criminal law. the tasks posed by the parole dataset do not fall neatly into relation extraction, which has been the focus of neural information extraction. for legal domain tasks, human labels are scarce and expensive, which raises the question of whether weak supervision may be a more efficient allocation of labels than direct supervision. legal hearings are long and don't fit neatly into the context window size of a neural model, which raises questions about how neural question answering systems can address this task. we answer the questions in turn. can weakly supervised methods be successfully used to reduce the cost of data annotation? data programming provides the opportunity to produce a large number of labels, but it still comes in general, including on date features, the most common failure mode for qa1 and qa2 is to return an incorrect answer of a correct type. for example, for yrserved, the models frequently returned any number they found in the context passage, such as the sentence (e.g. "15 years to life") or any other time range (e.g. "it was around two years i was part of that gang.") how big a problem is document length? long context windows continue to challenge all models present, especially neural models. although developing retrieval models for dialogue can help narrow the context window for downstream question answering applications, an even bigger challenge is the fact that even with an ideal retrieval model, the "correct" context window can still be long. in conversation, speakers are free to go on tangents. more importantly, in the case of legal hearings, speakers elaborate on case factors, attending to detail (as they should), which can greatly prolong a hearing. for example, in discussions of the psychological risk score, both data generation methods and neural question answering systems fail to identify the sentence and keyword containing "low," "moderate," or "high." we suspect that this is because discussions of all risk factors are usually several thousand words long. the score can be mentioned at the very beginning or very end, but often it is tucked away somewhere in the middle. ## conclusion parole hearing transcripts go into a great amount of detail in discussing numerous case factors centered around a single named entity, an incarcerated individual who has reached their parole eligibility date. the lack of relational structure and long format of these hearings makes information extraction from transcripts very challenging using several very different approaches from modern nlp. we estimate that an f1 score of 0.80-0.85 across a broad set of features would provide the ability to conduct meaningful downstream research on a hearing-driven decision-making process like parole. to flag individual cases for reconsideration, we believe that the bar likely lies even higher, since misclassifications often cause outliers. the performance of present models approaches the level at which we can provide useful automatic extraction tools to parole stakeholders for some features, especially certain binary ones. however, for other, seemingly simple medium-and high-cardinality tasks, much work remains. we plan to conduct future experiments to provide more transparency to model performance. the opaque nature of nlp modeling perplexes our legal collaborators: "how can you identify whether a candidate has participated in gang-related rehabilitation programming but not pick out the risk assessment score from a choice of three words?" the largest challenge moving forward remains natural language understanding in the face of document length. of course, length is not the only problem and other artifacts of spoken dialogue cause challenges, including interruptions, corrections, and colloquial speech. improved retrieval techniques or even summarization methods can help assess the extent to which document length remains a challenge and possibly mitigate its impact. however, there is no getting around the level of detail that is regarded as due process. one solution is to incorporate the hierarchical nature of dialogue @xcite . within a discussion about risk assessment, a parole commissioner may ask about various sub-factors, such as mental illness, or behavior toward other individuals in prison. we suspect that the word "low," "moderate," or "high" can appear in any of those sub-topics without referring to the risk score. we hope to conduct further research to assess the need for and viability of a hierarchical model. conversely, an extractive model sometimes picks up on risk-related words in the sub-topics, rather than returning to the higher level question of the risk scores. common sense knowledge will also play a role in solving this challenge. in one section of a hearing, the commissioner says, "and, uh, i note that you -you have both a high school diploma and ged, is that correct?" over the course of the next eight thousand words, the parole candidate describes his life, from playing sports in high school, to having a child, to the chaos of teenage co-parenting, to night school, to getting married, and to moving cities to protect his children. later on, the commissioner revisits the record and says, "you've taken some college classes," which the candidate himself failed to mention. in addition to understanding the topics and sub-topics in which education occurs, the edu level task benefits from real-life knowledge about educational levels. the wslf model performs well because of tailored labeling functions that encode information about "high school" and "college." finally, cross-sentence reference resolution remains important. in figure 1 , the question of the most recent form 115 can be answered in a short context window. yet, extracting the answer requires resolving the reference of "these" in " you picked up a couple of these in 2013." while the amount of attention to personal detail in these hearings presents the biggest challenge to our extraction models, individualized attention is also precisely what defines equitable justice. we hope that the nlp community will take up this challenge.