Learning from Limited Labels for Long Legal Dialogue (2021)

Abstract: We study attempting to achieve high accuracy information extraction of case factors from a challenging dataset of parole hearings, which, compared to other legal NLP datasets, has longer texts, with fewer labels. On this corpus, existing work directly applying pretrained neural models has failed to extract all but a few relatively basic items with little improvement over rule-based extraction. We address two challenges posed by existing work: training on long documents and reasoning over complex speech patterns. We use a similar approach to the two-step open-domain question answering approach by using a Reducer to extract relevant text segments and a Producer to generate both extractive answers and non-extractive classifications. In a context like ours, with limited labeled data, we show that a superior approach for strong performance within limited development time is to use a combination of a rule-based Reducer and a neural Producer. We study four representative tasks from the parole dataset. On all four, we improve extraction from the previous benchmark of 0.41–0.63 to 0.83–0.89 F1.

## introduction in many judicial processes such as legal hearings and criminal trials, decisions are made as a result of lengthy dialogues, in which case factors are discussed in great detail. to study such dialogues, scholars typically invest immense effort to hand label a small number of transcripts with some case factors; the factors are then used in downstream analysis. in most cases, the sheer length of transcribed conversational text all but prohibits any large-scale analysis of the process. information extraction over dialogues can assist in identifying the underlying factors of a case from transcripts. the benefits of information extraction are twofold. automating the extraction of case factors means that a historical legal analysis can now be comprehensive, containing all available transcripts, rather than being limited to the several dozen or hundred transcripts that a single researcher can label by hand. the second advantage is to open the door to counterdata applications in law (d'ignazio and @xcite . to date, most machine learning applications in the law have been predictive: given case factors up front, make a prediction of an outcome. in domains where case factors cannot or should not be known prior to the hearing, information extraction can produce case factors after a hearing, which enables machine learning to play an alternative role to the role of prediction, the role of oversight @xcite . in our application, information extraction allows the public to audit the parole process, whose case records are otherwise locked away in a filing cabinet. to be useful for such downstream research, the consensus in legal domain nlp is that information extraction should produce labels that achieve an f1 of at least 0.80 @xcite . our corpus, a set of historical california parole hearings, is a particularly difficult application, but also representative of many challenges in criminal law: (1) parole hearings are longer than documents in existing benchmarks. (2) existing benchmarks source from written text; parole documents are loosely-structured dialogue. (3) existing benchmarks contain at least an order of magnitude more labels. (4) information extraction from formal written documents centers around named entities and relation extraction. by contrast, much of the text in the criminal context serves the purpose of surfacing, discussing, and correcting case factors, which are not necessarily relational. this means parole hearings pose both extractive and abstractive tasks, often across multiple sentences, which is known to be challenging even in more structured settings @xcite . the scarcity of labels and specificity of the domain suggest that subject matter experts (smes) can be helpful. on the parole corpus, weak supervision-based data programming approaches @xcite achieve f1 scores of only 0.41-0.63 @xcite . we propose an alternative way to involve smes, in which we split the problem into two components: a reducer model which extracts relevant text segments from a hearing, and a producer model which generates answers from the text segments selected by the reducer. our methods effectively achieve extraction at 0.83-0.89 f1. we show that using an approach with a rulebased reducer and neural producer outperforms other commonly-used approaches. focusing sme effort on developing rules for the reducer is thus more time-efficient than requiring smes to provide additional target labels, whether manually or via data programming. with quality text segments, a neural producer model can be effectively fine-tuned on just one thousand labels. ## related work a review of data programming literature suggests that semi-supervised techniques might be a good fit for our problem space. several existing pipelines combine a limited amount of training data, rulebased systems and neural models to achieve strong results on benchmark datasets @xcite and in various medical fields @xcite @xcite . by comparison, weak supervision-based data programming methods tend to focus on bootstrapping in the absence of data @xcite , which is a nontrivial performance constraint. regardless of supervision strength, an architecture based on rule-based systems may be useful for generating "candidates" as input to downstream neural models; @xcite explores the time efficiency of manual labeling compared with rule-writing (via regular expressions) for named entity recognition (ner), where results are compared over a bidirectional lstm-based classifier, finding that in most circumstances, a combination of rulebased and machine-learning classifiers optimizes human time investment. we therefore adopt the approach of using a rulebased system for candidate generation. one new challenges with our corpus is that parole hearings generally center around one individual, so the candidates for downstream models are not named entities, but more loosely defined segments of the hear-ing. compared to ner, there is less prior work exploring rule-based methods for more general retrieval and segmentation. our goal of achieving 0.80 f1 in an abstractive format is currently beyond the capabilities of stateof-the-art (sota) neural models on comparable tasks, only one of which is in the legal domain. on natural questions (nq; @xcite , sota models achieve f1 scores of 0.79 and 0.64 on its long and short answer tasks, respectively. however, nq is purely extractive and averages only 7,300 words per input. on the doc2edag financial statements dataset @xcite , the graph-based interaction model with a tracker @xcite surpasses 0.80 f1 when extracting events from documents averaging 912 tokens in length, but this sota result drops to 0.76 f1 in the longest quartile. on open-domain question answering, the sota dense passage retrieval @xcite has an extractive top-5 accuracy of just 0.66. for downstream applications, a model must have a robust top-1 accuracy. the closest comparable legal dataset is the contract understanding atticus dataset (cuad) @xcite . over cuad, a sota model like roberta @xcite achieves a lower, and extractive, question answering performance of 0.80 recall at 0.31 precision, representing an f1 score of only 0.45, with documents still averaging one-quarter the length of parole transcripts. ## data we have obtained a corpus of 35,105 parole hearing transcripts, averaging 18,499 words each from 2007-2019. 1 each hearing is a dialogue, primarily between one or more commissioners and the parole candidate. most case factors are embellished with history and context, which is important for the procedure of a parole hearing, but challenging for information extraction. @xcite identified eleven fields for information extraction. we study the four fields that the previous study failed to extract with near 0.80 f1: job_offer (whether the parole candidate has a job offer upon release), edu_level (the candidate's educational level), risk_assess (a psychological assessment score), and last_writeup (the date of the candidate's last disciplinary writeup in prison). ## methods we use a reducer-producer paradigm (figure 2 ) in the spirit of the document retriever-reader model used in open-domain question answering (odqa; @xcite , with two differences: (1) the reducer selects one or more relevant passages from within a single document @xcite , and (2) the producer model is not necessarily a qa model. we use separate reducers and producers for each field. prior applications of data programming to this corpus used smes to write noisy labels for training a neural model; it does not significantly reduce the input text into shorter segments and instead relies on an end-to-end neural approach @xcite . by contrast, our approach uses smes to focus on the smaller task of reducing input text and relies on only gold labels, however few, for training the neural model. one subproblem is designed to be tractable for an sme (the reducer), and the other for a pretrained language model (the producer). ## results our methods achieve the 0.80 f1 benchmark 23 for all four fields, as shown in ## conclusion our corpus of parole hearings poses the challenge of information extraction with few gold labels: one thousand labels is not enough to locate and identify the answer in a long document. parole, like many other applications, requires domain-specific knowledge, which raises the question of how best to incorporate the labor of subject matter experts to assist neural models in making optimal use of available labels, in order to achieve high performance on extraction tasks. we identified two problems with existing work on the parole dataset, which fell short of the 0.80 f1 on many tasks: (1) text segments remained too long for many sota neural models to digest, and contained many spurious signals. (2) question answering was a useful first approach to handle a wide range of different feature types. however, out-of-the-box, it was rarely the best way to handle each individual feature type. we present an approach that uses an smedesigned rule-based reducer to identify relevant text segments, and a neural producer to generate labels using those text segments. we argue that it is time-efficient and performant for human smes to write mostly keyword-based rules for finding relevant parts of a parole transcript. in a parole transcript, a field of interest might be discussed in practically infinite different ways, but is usually somewhat well-defined by a limited set of words and patterns that are almost always used (for example, "ged", "college courses", "did not graduate" for a parole candidate's level of education). these keywords are relatively easy for a human to identify and write combinations of regular expressions to identify. however, training a neural model to recognize the phrases over the course of 20,000-word documents requires at least an order of magnitude more labels than are available @xcite . therefore, we focus sme energy on the reducer, and only the reducer. for the producer model, the role of human and machine are reversed. when the text is shortened to a sufficiently succinct context, neural models can be successfully fine-tuned to extract labels at an f1 of 0.80. it is practically impossible for a human to write rules to interpret every possible phrasing of, for example, someone's educational journey. however, pretrained language models excel at producing labels from small, targeted pieces of text. the 1,000 available labels are sufficient for good performance on this task @xcite . we use a base model that can handle relatively long tokens. we also explore a range of different fine-tuning heads. our architecture shows the effectiveness of a modular, two-step approach, where not every module needs to be a neural or machine learning model. such efforts to involve subject matter experts are especially important in applications that require substantial domain expertise. we hope that this work encourages additional research to better understand other legal processes whose workings are yet opaque to the public.