An Efficient Active Learning Pipeline for Legal Text Classification (2022)

Abstract: Active Learning (AL) is a powerful tool for learning with less labeled data, in particular, for specialized domains, like legal documents, where unlabeled data is abundant, but the annotation requires domain expertise and is thus expensive. Recent works have shown the effectiveness of AL strategies for pre-trained language models. However, most AL strategies require a set of labeled samples to start with, which is expensive to acquire. In addition, pre-trained language models have been shown unstable during fine-tuning with small datasets, and their embeddings are not semantically meaningful. In this work, we propose a pipeline for effectively using active learning with pre-trained language models in the legal domain. To this end, we leverage the available unlabeled data in three phases. First, we continue pre-training the model to adapt it to the downstream task. Second, we use knowledge distillation to guide the model’s embeddings to a semantically meaningful space. Finally, we propose a simple, yet effective, strategy to find the initial set of labeled samples with fewer actions compared to existing methods. Our experiments on Contract-NLI, adapted to the classification task, and LEDGAR benchmarks show that our approach outperforms standard AL strategies, and is more efficient. Furthermore, our pipeline reaches comparable results to the fully-supervised approach with a small performance gap, and dramatically reduced annotation cost. Code and the adapted data will be made available.

## introduction with the advent of pre-trained transformer-based language models @xcite @xcite , training models from scratch has been outperformed by fine-tuning pre-trained language models for several tasks in natural language processing, including text classification @xcite . however, fine-tuning these models still needs large labeled datasets to perform well on the downstream task @xcite @xcite . collecting a large annotated dataset is a highly expensive and time-consuming process in specialized domains, where annotation can only be performed by the domain experts, such as the legal domain @xcite . active learning (al) has been proved effective for data-efficient fine-tuning of pre-trained language models in non-specialized domains like news, emotions, and movies @xcite . in addition, @xcite have shown that the unlabeled data can be used to adapt the pre-trained language model to the downstream task, thereby improving the active learning performance with no extra annotation cost. on the specialized domains, @xcite have evaluated multiple al strategies in the legal domain before the emergence of pre-trained language models. nevertheless, to the best of our knowledge, the effectiveness of active learning in fine-tuning pre-trained language models in the legal domain has been poorly studied. in this work, we focus on efficient legal text classification with roberta @xcite by leveraging existing al strategies. we identify two challenges in deploying al strategies in the legal domain; first, legal texts contain a specialized vocabulary that is not common in other domains, including the ones on which pre-trained language models are trained. second, the annotation of legal texts is highly expensive and time-consuming due to the necessity of specialized training for understanding these texts. for example, @xcite reported a cost of over $2 million for the annotation of the contract understanding atticus dataset (cuad) consisting of around 500 contracts. to account for the specialized vocabulary, inspired by @xcite work, we leverage the available unlabeled data to adapt the pre-trained language model to the downstream task. in addition, considering the limitations of pretrained language models like bert and roberta in capturing semantics @xcite , we use knowledge distillation to further improve the task-adapted model by mapping its embedding space to a semantically meaningful space. our experiments demonstrate that al strategies can benefit from semantically meaningful embeddings. concerning the cost and time constraints, we focus on the fact that many al strategies @xcite @xcite require an annotated set of n positive and negative samples to start with. in practice, acquiring this set is expensive for large and skewed datasets. we propose a strategy to make the first iteration more efficient by clustering the unlabeled samples and limiting the pull of candidates to the cluster medoids. our experiments demonstrate we can achieve comparable results with the standard initial sampling approach with up to 63%, and 25% fewer actions on the skewed contract-nli @xcite , and balanced ledgar benchmarks (tuggener et al., 2020) respectively. our contributions can be summarized as follows: active learning with pre-trained language models multiple works have studied active learning for pre-trained language models like bert. ein-dor et al. ( 2020 ) have evaluated various al strategies for fine-tuning bert for text classification, and showed that al can boost bert's performance especially for skewed datasets. however, they do not leverage the available unlabeled data to adapt the pre-trained language model to the task at hand, and only focus on non-specialized domains like news and sentiment analysis that do not require experts' knowledge. @xcite have shown that taskadaptive pre-training using the available unlabeled data leads to performance gain when using pretrained language models like bert. following this observation, @xcite demonstrated the importance of task-adaptation for active learning for non-specialized texts like news, movies and sentiment analysis. inspired by these works, we leverage the available unlabeled data to effectively adapt roberta to legal text classification, where the annotation demands experts' knowledge. in addition, we propose an additional step to map the embedding space of the task-adapted roberta to a semantically meaningful space using sentence transformers. sentence transformers @xcite have shown that the embedding space of offthe-shelf pre-trained language models like bert @xcite and roberta @xcite is not semantically meaningful, and thus, is not suitable for common sentence comparison measures like cosine similarity. to overcome this limitation, they propose sentence transformers, obtained by adding a pooling layer on top of pre-trained language models, and fine-tuning them in a siamese network architecture with pairs of similar sentences. in this work, we use a roberta-based sentence transformer as a teacher model and distill its knowledge to the task-adapted roberta to produce sentence embeddings that capture the semantics and can be compared using cosine similarity. active learning strategies numerous methods have been proposed to find proper labeling candidates for active learning. majority of them belong to one or both of two categories: diversity-sampling, and uncertainty-sampling. diversity-based methods @xcite @xcite aim to find labeling candidates that best represent the dataset, whereas uncertainty-based methods @xcite @xcite target candidates about which the model is uncertain. badge @xcite ) is a cluster-based al strategy that belongs to both of these categories. it transforms data into gradient embeddings that encode model confidence and sentence feature at the same time. by applying kmeans++ on the gradient embeddings it can find samples that differ both in terms of semantics and predictive uncertainty. alps @xcite is another cluster-based al strategy that leverages both uncertainty and diversity using the surprisal embeddings obtained by passing the sentences to the mlm head of the pre-trained language model, and computing the cross entropy loss for a random set of tokens against the target labels. existing al strategies often require a set of labeled samples to start with, which is expensive to acquire. to overcome this high cost, we propose a clustering-based strategy to reduce the effort required to create the initial set of annotated samples. ## notation and setting in this section, we explain the structure shared between all al strategies used in this work and fix the notation. active learning is an iterative process aiming to obtain a desired performance given an annotation budget. here, we consider the annotation budget to be the number of actions performed by the annotator. in addition, we assume all annotators are legal experts, and that each annotator assigns perfect labels to text segments. let u 0 and l 0 be the starting pool of unlabeled and labeled samples respectively. initially, l 0 = ∅. at the first iteration, the annotator labels n sample, p positive and n -p negative, to obtained l 1 . then, at each iteration i, the model is fine-tuned using l i , and the al strategy recommends a set of samples c i for annotation. these samples are labeled and u i and l i are updated as u i+1 = u i \c i , and l i+1 = l i ∪c i . the procedure is repeated until the annotation budget is exhausted, or the desired performance is achieved. we base our work on the low-resource text classification framework introduced by @xcite . following this work, we focus on binary text classification, given a small annotation budget and a potentially imbalanced dataset. this scenario matches common use cases in the legal domain, where the goal is to find phrases that correspond to a specific category, with the lowest possible number of actions, given a pool of unlabeled, imbalanced data. we perform 5 al iterations, and assume a more restricted annotation budget compared to @xcite , allowing only 10 annotations per iteration. for the first al iteration, we assume that 5 positive and 5 negative samples need to be annotated. ## methodology we propose an efficient active learning pipeline for fine-tuning pre-trained language models for legal text classification. our approach leverages available unlabeled data in three phases to adapt the pre-trained model to the downstream task (sec. 4.1), guide its embedding space to a semantically meaningful and comparable space (sec. 4.2), and reduce the number of actions required to collect the initial labeled set (sec. 4.3). finally, it leverages existing al strategies to efficiently fine-tune a classifier (sec. 4.4). we now explain each step in detail. an overview of this pipeline can be found in algorithm 1. ## experimental setup we evaluate our approach against four standard active learning strategies provided in the low-resource text classification framework (ein-dor et al., 2020): we consider pre-trained roberta and legal-bert @xcite as the baselines. however, we only evaluate our strategy using the pre-trained roberta as our goal is not to rely on domain-adapted models like legal-bert since they might not always be available. for example, if the data is in german, we can find a pre-trained roberta in german, but the legal-bert is pre-trained on english text only. ## results and discussion in this section, we provide the results of our experiments, and explain them in detail. we start by comparing our approach with and without the initial medoid sampling against standard al strategies (sec. 6.1). then, we show the effectiveness of knowledge distillation on top of task-adaptation (sec. 6.2). in addition, we demonstrate the efficiency of the initial sampling with cluster medoids (sec. 6.3). finally, we evaluate how well our approach performs for different al strategies (sec. 6.4). ## conclusion we propose an efficient active learning pipeline for legal text classification. our approach leverages the available unlabeled data to adapt the pre-trained language model to the downstream task, and guide its embeddings to a semantically meaningful space before fine-tuning. we use model distillation to produce semantically comparable embeddings. a future work can study the effect of other approaches like bert-flow @xcite and whitening @xcite on al with this pipeline. moreover, we design a simple strategy to efficiently acquire a labeled set of positive and negative samples for the initial iteration of active learning. our experiments over contract-nli and ledgar benchmarks demonstrate the effectiveness of our approach compared to standard active learning strategies. our results also show that our pipeline obtains very close performance to the fully-supervised approach with considerably less annotation cost. we test our methodology in the legal domain, and for four al strategies, but we expect it to generalize to other strategies like alps and badge, and other specialized domains, like medicine. we leave this evaluation as a future work.