Do Language Models Learn about Legal Entity Types during Pretraining? (2023)

Abstract: Language Models (LMs) have proven their ability to acquire diverse linguistic knowledge during the pretraining phase, potentially serving as a valuable source of incidental supervision for downstream tasks. However, there has been limited research conducted on the retrieval of domain-specific knowledge, and specifically legal knowledge. We propose to explore the task of Entity Typing, serving as a proxy for evaluating legal knowledge as an essential aspect of text comprehension, and a foundational task to numerous downstream legal NLP applications. Through systematic evaluation and analysis and two types of prompting (cloze sentences and QA-based templates) and to clarify the nature of these acquired cues, we compare diverse types and lengths of entities both general and domain-specific entities, semantics or syntax signals, and different LM pretraining corpus (generic and legal-oriented) and architectures (encoder BERT-based and decoder-only with Llama2). We show that (1) Llama2 performs well on certain entities and exhibits potential for substantial improvement with optimized prompt templates, (2) law-oriented LMs show inconsistent performance, possibly due to variations in their training corpus, (3) LMs demonstrate the ability to type entities even in the case of multi-token entities, (4) all models struggle with entities belonging to sub-domains of the law (5) Llama2 appears to frequently overlook syntactic cues, a shortcoming less present in BERT-based architectures.

## introduction during the initial phase of pretraining, language models (lms) are exposed to an extensive corpus of textual data, allowing them to acquire the capacity to represent the probabilistic structure of language. in this process, it has been theorized that they incidentally learn various linguistic signals and patterns, both syntactic and semantic. work by @xcite and subsequent studies @xcite ) make the hypothesis that a side effect of the pretraining stage is that lms also learn factual knowledge. on the other hand, @xcite research demonstrates the significance of both model pretraining and task-specific pretraining; pretraining a model with a specific focus on a particular task or a limited domain corpus yields notable advantages in enhancing model performance and adaptability. entity typing and extraction are crucial tasks for a range of use cases including named-entity recognition (ner), relation extraction, summarization, structuring raw data, and most specifically in law, legal search, and past cases retrieval. to gain more insights into entity typing and extraction, entity probing tasks have been designed for bidirectional lstm conditional random field models @xcite , masked language models @xcite and autoregressive lms @xcite , using gpt-2. conversely, one notable bottleneck of the application of nlp within the legal domain is the lack of resources and annotated datasets. thereby, it is of particular interest to explore the extent to which lms, during their pretraining phase, acquire a sufficient understanding of legal entities, serving as a surrogate for legal knowledge. ultimately, lms could be exploited as a source of weak and indirect supervision in downstream tasks such as legal ner or question answering (qa), as they constitute a good proxy to use natural text incidentally thanks to their pretraining stage. indeed, humans do not exclusively rely on exhaustive supervision but instead make use of occasional feedback and learn from incidental signals originating from various sources. this approach holds potential for pretrained language model pretraining corpus # parameters # tokens corpus size # vocab legal casehold @xcite harvard case law 110m 43b 37 gb 32k pile of law @xcite us, canadian, ecthr 340m 130b 256 gb 32k lexlm @xcite us, canada, eu, uk, india 125 2t + 256b 175 gb 50k ## dataset we use asylex, a dataset of refugee decisions from canada curated for entity typing and extraction foot_0 . this publicly available dataset comprises 19,115 human annotated instances, encompassing 20 distinct categories of entities that hold legal relevance as explained in @xcite . these categories have been identified as categories of interest with the collaboration of legal professionals and experts in the field of refugee law. asylex comprises 59,112 historical decision documents, explanation 1142 1963 1061 5834 legal_ground 2801 4142 868 2189 law 688 2783 1511 5018 law_case 4562 2136 663 2639 law_report 2327 537 2542 459 8970 5269 9849 8473 4181 448 2603 5390 5192 1142 2801 688 4562 2327 122 608 25 961 1617 6013 1233 834 476 5834 2189 5018 2639 459 # 1t # 2t # 3t # 3+ spanning from 1996 to 2022. these documents are derived from the online repository of cases of the canadian legal information institute foot_1 . the documents encompass both initial determinations and subsequent appeals on whether the claimant is granted refugee status or not. it is important to note that the dataset contains entities of varying generality. some entities, such as geographical locations, possess broad applicability and could be pertinent to any text (generic entity types: location, date, norp). others are more specialized within the legal domain, such as procedural steps (generic legal entity types: org, law, claimant_info, procedure, doc_evidence, law_case). finally, certain entities are highly specific to refugee law, such as the assessment of credibility, which frequently determines the acceptance or rejection of a refugee claim (refugee law entity types: credibility, determination, explanation, legal_ground, law_report). this diversity in entity scope presents an opportunity for assessing the impact of pretraining, particularly in scenarios where entity types have received various exposures during the pretraining phase. ## experimental results we evaluate the results in terms of recall since we want to ensure capturing as many true positives as possible, and f1 score to assess the overall performance on the task. in this section, we compare the results in terms of lm used, length of the input entity, prompt type, and entity type, before conducting an error analysis in the section 6.5. the results of experiment mlm are presented in ## conclusion our investigation includes lm selection, input entity length, prompt types, and entity types, in an attempt to understand model strengths and limitations. in summary, our study shows that llama2 performs best on specific entities and displays potential for improvement with better prompting strategies. however, it also seems that llama2 repeatedly overlooks syntactic cues. masked language models mostly appear to be lacking sufficient context within our experimental setup, where they are confronted with a highly challenging task. law-oriented lms exhibit varying results, possibly influenced by training corpus differences, and the pile of law model shows the best performance on our asylex dataset. despite inherent challenges, lms can accurately identify certain entity types, including multi-token ones, but encounter difficulties with legal sub-domains like refugee law. future research may explore optimized prompts and fewshot learning strategies. furthermore, assessing the average precision of the entity type ranking predictions generated by the lm, and conducting experiments on additional datasets, would also be necessary.