Discovering Explanatory Sentences in Legal Case Decisions Using Pre-trained Language Models (2021)

Abstract: Legal texts routinely use concepts that are difficult to understand. Lawyers elaborate on the meaning of such concepts by, among other things, carefully investigating how they have been used in the past. Finding text snippets that mention a particular concept in a useful way is tedious, time-consuming, and hence expensive. We assembled a data set of 26,959 sentences, coming from legal case decisions, and labeled them in terms of their usefulness for explaining selected legal concepts. Using the dataset we study the effectiveness of transformer models pre-trained on large language corpora to detect which of the sentences are useful. In light of models’ predictions, we analyze various linguistic properties of the explanatory sentences as well as their relationship to the legal concept that needs to be explained. We show that the transformer-based models are capable of learning surprisingly sophisticated features and outperform the prior approaches to the task.

## introduction written laws enacted by legislative bodies set forth the collection of legally binding rules of conduct (e.g., rights, prohibitions, duties). understanding written laws is difficult because the abstract rules must account for a variety of situations, even those not yet encountered. written laws communicate general standards and refer to classes of persons, acts, things, and circumstances @xcite . therefore, legislators use vague @xcite , open textured @xcite terms, abstract standards @xcite , principles, and values @xcite to deal with the inherent uncertainty. for example, let us focus on the two emphasized concepts from the following written provision of law (29 u.s. code § 203): understanding of the provision depends on understanding the meaning of the two emphasized concepts. any doubts about the meaning may be removed by explanation or interpretation @xcite . even small differences in understanding of a single concept may be crucial for determining how a provision applies and what are its effects in a particular context. for example, the meaning of the concept common business purpose could be crucial in determining if two restaurants in different parts of the same city, sharing a single owner, constitute an "enterprise." the explanation of the concept would involve an investigation of how has it been referred to, explained, interpreted, or applied in the past. this is an important step that enables a lawyer to come up with arguments in support of or against particular accounts of meaning @xcite . searching through a database of legal documents a lawyer may retrieve sentences such as the following: some of these sentences are most likely useful for explaining the concept (1 and 2) but others appear to have very little value (3). manually reviewing such sentences is labor intensive. we would like to rank more highly the sentences the goal or effect of which is to elaborate upon the meaning of the selected concept. these include, but are not limited to, (i) definitional sentences (e.g., a sentence that provides a test for when the concept applies), (ii) sentences that state explicitly in a different way what the concept means or state what it does not mean, (iii) sentences that provide an example, instance, or counterexample of the concept, and (iv) sentences that show how a court determines whether something is such an example, instance, or counterexample. ## related and prior work in prior work, we employed a variety of traditional information retrieval (ir) measures and their combinations, e.g., bm25, novelty, topic modeling @xcite @xcite . these turned out to be remarkably successful in finding documents or their parts (e.g., paragraphs) that are likely to contain useful sentences. however, they fell short in performing finer-grained evaluation of the sentences contained in those document parts. using a learningto-rank approaches on hand-crafted features led to only moderate improvements @xcite . in this work, we show that transformer-based pre-trained models (bert family) are capable of such fine-grained evaluation by learning to detect sophisticated semantic features in sentences themselves as well as in their relationships to the explained concepts. furthermore, we show that many of these features are sensible to humans. the models based on the bert architecture have been successfully used in a variety of ir tasks. a comprehensive survey of text ranking with transformers, such as bert, is provided in @xcite . several simple applications of bert to ad hoc document retrieval are presented in @xcite . successful applications of bert for retrieval of short texts such as sentences are presented in @xcite and @xcite . similar to the utilization of provisions of written law in this work, the authors of @xcite demonstrated the effectiveness of using a query context in a re-ranking component based on bert. in nogueira et al. (2019 ## data set we downloaded the complete bulk data from the caselaw access project foot_0 which includes all official, book-published u. s. cases from all federal and state courts as well as from a number of territorial courts (president and of harvard university, 2018). the dataset comprises more than 6.7 million unique cases. for document indexing we used a lemmatizer based on the so-called induced rippledown rules @xcite ). foot_1 using the u.s. case law sentence segmenter @xcite we divided each case into individual sentences (0.8 billion). we queried the system for sentences mentioning 42 selected legal concepts (i.e., terms/phrases, such as "audiovisual work," or "electronic signature") coming from provisions of the u.s. code (the official collection of federal statutes). 3 given the constraints imposed by available resources, we made the best effort to create a well-balanced dataset covering 20 different areas of legal regulation (26,959 retrieved sentences in total). eleven law students classified the sentences in terms of four categories with respect to their utility for explaining the legal concepts: ## discussion while it should be apparent that the sentence detached from the legal concept (and the provision of law it is embedded in) does not provide reliable grounds for determining its value, it appears that the sentences themselves do carry some signal. this is evidenced by the performance of the bert snt model that only considers sentences themselves. this model outperforms the bm25-based baselines. interestingly, the system correctly recognized that very short pieces of text that do not form full grammatical sentences typically have very little value. for example, the following sentences have been placed at the bottom of their respective rankings (the explained context is highlighted in yellow): communication & navigation equipment [no value] b. non-disclosure of preexisting works [no value] furthermore, it appears that the system relies on features such as the presence of numbering in the sentence, complicated sentence structures, abrupt endings or starts of the sentences, and references, to recognize quotations of written provisions of law and assign a low value to such sentences: derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and [f] (2) [no value] this strategy makes sense in general. the quotation could either be the citation of the source provision ('no value') or a citation of a different provision (high chance of different meaning and hence lower value of a sentence). however, there are situations in which the strategy does not work well. the electronic communications privacy act of 1986 (ecpa), pub. l. 99-508, §101(a)(6)(c), 100 stat. 1848, 1849 (1986), codified, as amended, at 18 u.s.c. §2510(18) (1986), defines "aural transfer" to mean "a transfer containing the human voice at any point between and including the point of origin and the point of reception." [high value] the 'aural transfer' is a rare example of a concept for which there is a legal definition. as a result bert snt underperforms the random baseline on this particular concept (ndgc@100 0.53 vs 0.62). bert snt also seems to have developed a certain tendency to rank high sentences where something is claimed to be something else: screen output is considered an audiovisual work that falls within the subject matter of copyright. ## conclusions and future work in this work, we showed that pre-trained language models based on transformers can be fine-tuned for the special task of retrieving sentences for explaining legal concepts. specifically, we demonstrated that a pre-trained roberta base model, fine-tuned on three variations of the task, resulted in effective ranking functions outperforming the bm25 baselines. the promising performance of bert snt reveals the interesting fact that sentences themselves carry certain signal about their usefulness. the even stronger performance of bert qry2snt and bert sp2snt points to the important interactions among a legal concept, the provision of law in which it is embedded, and retrieved sentences, that both need to be accounted for in order to perform well in this challenging task. the whole work demonstrates the effectiveness of methods based on pre-trained language models applied to a legal domain task. this is important because advances in general nlp and ml do not always transfer in a straightforward manner to specialized domains such as automatic processing of legal or medical texts. importantly, we fill the gap in prior work by showing that the transformer based methods are capable of fine-grained evaluations of the individual sentences as to their usefulness. the application of pre-trained language models to the task of discovering sentences explaining legal concepts yielded promising results. at the same time, the work is subject to limitations and leaves much room for improvement. hence, we suggest several directions for future work: