Can Domain Pre-training Help Interdisciplinary Researchers from Data Annotation Poverty? A Case Study of Legal Argument Mining with BERT -based Transformers (2021)

Abstract: Interdisciplinary Natural Language Processing (NLP) research traditionally suffers from the requirement for costly data annotation. However, transformer frameworks with pre-training have shown their ability on many downstream tasks including digital humanities tasks with limited small datasets. Considering the fact that many digital humanities fields (e.g. law) feature an abundance of non-annotated textual resources, and the recent achievements led by transformer models, we pay special attention to whether domain pre-training will enhance transformer’s performance on interdisciplinary tasks and how. In this work, we use legal argument mining as our case study. This aims to automatically identify text segments with particular linguistic structures (i.e., arguments) from legal documents and to predict the reasoning relations between marked arguments. Our work includes a broad survey of a wide range of BERT variants with different pre-training strategies. Our case study focuses on: the comparison of general pre-training and domain pre-training; the generalisability of different domain pre-trained transformers; and the potential of merging general pre-training with domain pre-training. We also achieve better results than the current transformer baseline in legal argument mining.

## introduction interdisciplinary natural language processing (nlp) has become one of the most important trends in nlp development. for example, processing of legal text has resulted in research topics such as legal topic classification @xcite , legal information extraction @xcite , judicial decision prediction @xcite , and legal argumentation mining @xcite . among these, legal argumentation mining is especially difficult, but has strong application potential, given that arguments are among the most important language structures used in the law. the goal of argument mining is to automatically detect arguments from raw text as well as to identify the reasoning relationships between these arguments @xcite . argument mining systems that help to identify and analyse argumentative text can assist legal professionals to save time and effort when facing modern document systems with huge quantities of files. however, the shortage of suitable datasets and the high cost of annotating new datasets impede the application of many advanced nlp approaches (such as neural network and deep learning) to legal argument mining, which is a common issue in most interdisciplinary nlp research. creating and constructing annotated corpora is complex and labour intensive @xcite . particularly when the raw text is domainspecific (e.g., legal text), the annotating experts are required to have extensive knowledge of the corresponding field. this leads to a paradoxical situation whereby a domain with enormous quantities of text resources built up over centuries is served by only a small number of suitable corpora that tend to be limited in their scale. this dilemma in interdisciplinary research may be solved by using transformer frameworks, such as bert @xcite : first pre-training (selfsupervised learning) on a large group of roughly labelled text, then fine-tuning on downstream tasks with much smaller datasets that do fine-grained feature annotation. transformers have revolutionised many research fields including legal text processing @xcite . this caused us to examine the potential of reducing the burden of annotation in interdisciplinary research through pre-training. in this work, we use legal argument mining as our case study, because it includes not only the general text classification, but also the relation mining on legal text, and has a strong connection with both the legal field (humanities) and the study of argument mining (nlp). the primary aim of this paper is to explore the extent to which domain pre-training (i.e. pretraining transformers using legal texts) can improve transformers' performance on legal text processing tasks, without the need for large volumes of expensive annotation. legal text has its own distinct characteristics when compared with general englishlanguage writing, this also motivates an investigation as to whether legal-specific pre-training can improve upon transformers pre-trained on generalpurpose corpora (e.g. wikipedia, books). although @xcite set a legal argument mining baseline using roberta @xcite , none of the legal-domain pre-trained transformers @xcite have been applied to legal argument mining tasks to date. in this case study, we try to find (a) whether domain pre-trained transformers outperform general pre-trained transformers in interdisciplinary nlp tasks; (b) whether domain pre-trained transformers maintain good generalisability when applied to tasks using another domain-specific dataset without overlap with the pre-train corpora; and (c) whether merging pre-train corpora from different domains can enhance the transformer performance on interdisciplinary downstream tasks. in our work, we first provide a thorough survey including a wide range of bert variants which emphasise two pre-train domains (generic and legal) and use different pre-train strategies. then we evaluate these transformer models on three legal argument mining tasks: (1) argument clause recognition, (2) argument relation mining, (3) argument component recognition. we discuss the potential of using domain-specific pre-training to adapt stateof-the-art transformer models to interdisciplinary research that lacks large annotated datasets, and we analyse what adjustments in domain-specific pretraining may improve transformers' performance in a complex text processing problem like legal argument mining. ## argument and argument mining long before being treated as a research area in nlp, philosophers and rhetoricians had paid special attention to the logic and reasoning processes embodied in human languages. numerous schemes and theories have been proposed to define and reason about argumentation. in our work, we use the definition given by @xcite that an argument is a set of statements (propositions) that includes three parts: conclusion, premises and inference. as described in @xcite , argument mining is the automatic identification and extraction of the structure of inference and reasoning expressed as arguments presented in natural language. there are two crucial stages in the framework of argument mining: argument extraction and relation prediction @xcite . argument extraction is the first stage where the arguments (with their internal structures) are identified from the input documents. relation prediction is where the support or attack relations between the arguments are predicted. ## bert-based transformers bert (bidirectional encoder representations from transformers) is a contextual word embedding model using the deep transformer architecture @xcite to derive word features @xcite . it leverages a two-step framework: pre-training and fine-tuning. during pre-training, bert is trained on a large corpus using self-supervised learning methods. then, in finetuning, the model is tuned on the downstream task's dataset, which is usually much smaller. bert has achieved state-of-the-art performance on many legal text processing tasks @xcite @xcite . since then, several studies have addressed whether pretraining on legal texts can improve transformers' performance on downstream tasks in the same domain @xcite @xcite , but none have been evaluated on legal argument mining. ## bert pre-train strategies in order to analyse how different pre-trained transformers perform on legal argument mining tasks, we compare five bert base (@xmath0, @xmath1, @xmath2, 110m params @xcite ) variants from two pre-train categories: general pre-trained models using generic english corpora for pre-training, and domain pre-trained models using english legal text in their pre-train corpus. here we first provide an outline background of each transformer. then, based on previous studies @xcite , we compare these models across three key aspects of pre-train strategy: pre-train corpora selection, pre-train procedure, and text encoding. ## echr dataset the european court of human rights (echr) case-law dataset, developed from the hudoc database 1 , is an open-source database of case documents, and has become one of the most commonlyused datasets for legal text processing research such as judicial decision prediction @xcite , court decision event extraction @xcite , and legal argument mining @xcite @xcite . the echr case-law dataset has been used for legal argument mining research from an early stage @xcite . @xcite provides a detailed structural analysis of echr documents. several legal argument mining corpora have been established based on the echr case-laws @xcite @xcite . among them, we choose the recently released echr argument mining corpus (echr-am) @xcite for our experiments. the echr-am corpus contains 42 cases, 20 decisions (the average word length is 3,500 words) and 22 judgements (the average word length is 10,000 words). the entire corpus is annotated at the sentence level using three labels according to the walton argumentation model (see section 2.2): premise, conclusion, and non-argument. the annotation focuses on internal argument relations which includes a total of 1,951 premises and 743 conclusions acting as argument components for individual arguments. ## legal argument mining tasks our case study currently focuses on the argument extraction, which is the first stage within a typical argument mining framework (as mentioned in sec-1 http://hudoc.echr.coe.int/ figure 1 : annotation example of the echr argument mining corpus tion 2). following the example of @xcite , we organise this as three tasks. ## discussion our case study on legal argument mining gives us insights on the potential of using domain pretraining to reduce the data annotation burden in interdisciplinary nlp research, as well as help us better understand the relationship between domain pre-training and domain-specific downstream tasks. to answer question (a) in section 1, it is clear that domain pre-trained transformers work better than general pre-trained transformers in all three legal argument mining tasks, which also exceed the baseline from @xcite . this supports the idea that domain pre-training helps improve transformer's performance on downstream tasks where only small datasets are available. combining the scope of pre-train corpora used by each transformer with its performance, we suggest that using small domain-specific pre-training corpora would be as effective as using a large general corpora. in section 7, the legal-bert family use a much smaller legal text collection (11.5 gb) compared with roberta's 161 gb general pre-train corpus, but also achieve competitive results despite less pre-training steps (see both harvard legal-bert variants present good performance on the echr-am dataset. in contrast to the legal-bert family which includes echr cases as part of its pre-train legal text collection, the pre-train corpus used by harvard legal-bert variants has no overlap with the echr-am dataset. therefore, for question (b) in section 1, our case study indicates that domain pre-trained transformers can maintain good generalisability on downstream tasks focusing on different datasets. this "reusable" characteristic of domain pre-trained models is significant. collecting relevant pre-train corpora and pre-training itself still require sufficient time and computing resources. if the domain pre-trained model is reusable in different domainspecific tasks, sharing domain pre-trained models will be superior to sharing corpora, especially for research groups who do not have the resources or capability for pre-training models on large corpora. with reference to question (c) in section 1 about merging domain pre-training with general pre-training, the experiment results indicate that the transformers using further pre-training work slightly better than those using pre-training from scratch. this indicates the potential of enhancing the transformer's generalisability on downstream domain-specific tasks by merging generic corpora with further pre-training, especially in tasks like argument relation mining that require the model to extract not only the features from the text but also the potential relations between different sequences. our work pays attention to a common issue in interdisciplinary nlp research that the background area (i.e., humanities, law) has considerable volumes of text material, but the annotation work is costly and impedes the process of adapting advanced nlp technologies to assist the research. we suggest that pre-train transformers can help this "data poverty" issue, and domain-specific pretraining will improve transformers' performance when adapting to interdisciplinary tasks with only small fine-annotated datasets. we analyse state-of-the-art transformers in both pre-train categories, and present a comprehensive survey of available models for the legal domain. our case study provides the first comparison between general pre-trained transformers and domain pre-trained transformers on legal argument mining tasks, and demonstrates that domain pre-trained transformers achieve better results on all three tasks than the baseline outlined by @xcite . our case study also compares the performance of two latest groups of transformers in legal domain, and offers an analysis of some key aspects when applying domain pre-training on interdisciplinary nlp tasks.