Knowledge-Guided Paraphrase Identification (2021)

Abstract: Paraphrase identification (PI), a fundamental task in natural language processing, is to identify whether two sentences express the same or similar meaning, which is a binary classification problem. Recently, BERT-like pre-trained language models have been a popular choice for the frameworks of various PI models, but almost all existing methods consider general domain text. When these approaches are applied to a specific domain, existing models cannot make accurate predictions due to the lack of professional knowledge. In light of this challenge, we propose a novel framework, namely , which can leverage the external unstructured Wikipedia knowledge to accurately identify paraphrases. We propose to mine outline knowledge of concepts related to given sentences from Wikipedia via BM25 model. After retrieving related outline knowledge, makes predictions based on both the semantic information of two sentences and the outline knowledge. Besides, we propose a gating mechanism to aggregate the semantic information-based prediction and the knowledge-based prediction. Extensive experiments are conducted on two public datasets: PARADE (a computer science domain dataset) and clinicalSTS2019 (a biomedical domain dataset). The results show that the proposed outperforms state-of-the-art methods.

## introduction paraphrase identification (pi) is a classical yet fundamental natural language processing (nlp) task, which aims to determine whether a pair of sentences express the same or similar meaning @xcite . such a task can be used to examine whether a machine learning model really understands the semantic meanings of input sentences and is helpful for many other nlp tasks such as machine translation @xcite and question answering @xcite ; rinaldi id sentences knowledge paraphrase s1 a list of recommended data elements with uniform definitions that are relevant for a particular use and encourage uniform data collection and reporting. dataset no s2 a recommended list of data elements that have defined and uniform definitions that are specific to a type of healthcare industry. healthcare data s3 the lowest level of code made up of 0s and 1s. binary instruction yes s4 binary instructions used by the cpu. binary instruction et @xcite . to identify paraphrases automatically, machine learning models have been proposed. traditional models @xcite @xcite @xcite focus on leveraging lexical and syntactic features to measure the similarity between two sentences. recently, deep learning models are introduced and achieve the state-of-the-art performance. these models adopt convolutional neural networks (cnns) @xcite , long short-term memory (lstm) @xcite @xcite or pretrained language models like bert @xcite . they directly learn the implicit relation between a pair of input sentences. however, existing approaches all ignore the importance of knowledge associated with input sentences. in fact, each meaningful sentence usually belongs to a certain domain and contains domainspecific knowledge @xcite . when domain experts identify whether these two sentences are paraphrases or not, they first read sentences to comprehend the semantic meanings, and then analyze them based on the domain knowledge associated with the sentences, and finally make a decision. as shown in fig. 1 , although s1 and s2 contain several matching words, experts know that they are not paraphrases because the first sentence describes the concept "dataset" but the second sentence does not. s3 and s4 even do not share a lot of lexical and syntactic features, but they have the same meaning since binary instructions are made up of 0s and 1s, which corresponds to the computer science domain knowledge. therefore, for the pi task, only relying on lexical and syntactic features is insufficient, and it is indispensable to introduce domain knowledge into the models. we will meet several technical challenges when introducing domain knowledge into the pi task: (1) knowledge selection. even in a specific domain, there is a huge volume of domain knowledge. the format of domain knowledge is either structured knowledge base or unstructured text. thus, using which kind of domain knowledge and how to retrieve related knowledge for each sentence efficiently and accurately are new challenges. to solve the aforementioned challenges, in this paper, we propose a knowledge-infused gated model (named knowing), which is shown in fig. 2 . knowing consists of four main components: a base prediction module, a knowledge selection module, a knowledge fusion module, and an aggregation module. the base module is to encode sentence pairs and make predictions based on their lexical and syntactic information. then we incorporate domain knowledge, and the first step is to collect related knowledge. the knowledge selection module retrieves top-m related knowledge outlines from wikipedia via bm25 @xcite for each sentence pair. then the knowledge fusion module is to encode knowledge via an attention mechanism and get the knowledge-based prediction. in the end, the aggregation module aggregates the lexical and syntactic feature-based prediction and knowledge-based prediction via a novel gate mechanism. the main contributions can be summarized as follows: ## preliminaries before formally introducing the proposed model knowing, we first mathematically define our task and the knowledge that we use in this paper. ## conclusions in this paper, we investigated the important and challenging task of domain-specific paraphrase identification. since domain-specific text is difficult to be understood without domain knowledge, we proposed to incorporate wikipedia knowledge into our model. however, there are two major challenges: (1) how to select knowledge, and (2) how to incorporate the selected knowledge into state-ofthe-art paraphrase identification models automatically. to solve these challenges, we introduced wikipedia as external knowledge base and proposed a knowledge-infused gated model named knowing to fuse wikipedia knowledge with bert. the knowing contains four modules: a base prediction module, a knowledge selection module, a knowledge fusion module, and an aggregation module. the base prediction module is to learn sentence pair representations and make predictions only based on sentence pair themselves. the knowledge selection module is to retrieve relevant knowledge from wikipedia, and then knowledge fusion module applies an attention mechanism to synthesize the knowledge representations. finally, the aggregation module is to aggregate the based module's predictions and the knowledge-based predictions via a gate function. the experiments on two public domain-specific datasets show that the proposed knowing outperforms state-of-the-art baselines.