Pre-trained Language Model Based Active Learning for Sentence Matching (2020)

Abstract: Active learning is able to significantly reduce the annotation cost for data-driven techniques. However, previous active learning approaches for natural language processing mainly depend on the entropy-based uncertainty criterion, and ignore the characteristics of natural language. In this paper, we propose a pre-trained language model based active learning approach for sentence matching. Differing from previous active learning, it can provide linguistic criteria from the pre-trained language model to measure instances and help select more effective instances for annotation. Experiments demonstrate our approach can achieve greater accuracy with fewer labeled training instances.

## introduction sentence matching is a fundamental technology in natural language processing. over the past few years, deep learning as a data-driven technique has yielded state-of-the-art results on sentence matching @xcite @xcite @xcite @xcite . however, this data-driven technique typically requires large amounts of manual annotation and brings much cost. if large labeled data can't be obtained, the advantages of deep learning will significantly diminish. to alleviate this problem, active learning is proposed to achieve better performance with fewer labeled training instances @xcite . instead of randomly selecting instances, active learning can measure the whole candidate instances according to some criteria, and then select more efficient instances for annotation @xcite @xcite @xcite . however, previous active learning approaches in natural language processing mainly depend on the entropy-based uncertainty criterion @xcite , and ignore the characteristics of natural language. to be more specific, if we ignore the linguistic similarity, we may select redundant instances and waste many annotation resources. thus, how to devise linguistic criteria to measure candidate instances is an important challenge. recently, pre-trained language models @xcite @xcite have been shown to be powerful for learning language representation. accordingly, pre-trained language models may provide a reliable way to help capture language characteristics. in this paper, we devise linguistic criteria from a pre-trained language model to capture language characteristics, and then utilize these extra linguistic criteria (noise, coverage and diversity) to enhance active learning. it is shown in figure 1 . experiments on both english and chinese sentence matching datasets demonstrate the pre-trained language model can enhance active learning. ## methodology in a general active learning scenario, there is a small set of labeled training data p and a large pool of available unlabeled data q. active learning is to select instances in q according to some criteria, and then label them and add them into p , so as to maximize classifier m performance and minimize annotation cost. more details of preliminaries about sentence matching and active learning are in the appendix. ## conclusion in this paper, we combine active learning with a pre-trained language model. we devise extra linguistic criteria from a pre-trained language model, which can capture language characteristics and enhance active learning. experiments show that our proposed active learning approach obtains better performance. appendix a: more details and discussions sentence matching task: given a pair of sentences as input, the goal of the task is to judge the relation between them, such as whether they express the same meaning. in formal, we have two sentences @xmath0,a 2 ,. . . ,a l a ] and @xmath1,b 2 ,. . . ,b l b ], where a i and b j denote the i-th and j-th word respectively in corresponding sentences, and l a and l b denote the length of corresponding sentences. through a shared word embedding matrix w @xmath2, we can obtain word embeddings of input sentences @xmath3,e(a 2 ),. . . ,e(a l a )] and @xmath4,e(b 2 ),. . . ,e(b l b )], where n e denotes the vocabulary size, d denotes the embedding size and e(a i ) and e(b j ) denote the word embedding of the i-th and j-th word respectively in corresponding sentences. and there is a sentence matching model m to predict a label ŷ based on a and b. when testing, we choose the label with the highest probability in prediction distribution p (y i |a, b; θ m ) as output, where θ m denotes parameters of the model m and y i denotes a possible label. when training, the model m is optimized by minimizing cross entropy: