PARADE : A N ew D ataset for P araphrase I dentification R equiring C omputer S cience D omain K nowledge (2020)

Abstract: We present a new benchmark dataset called PARADE for paraphrase identification that requires specialized domain knowledge. PARADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as non-paraphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Experiments show that both state-of-the-art neural models and non-expert human annotators have poor performance on PARADE. For example, BERT after fine-tuning achieves an F1 score of 0.709, which is much lower than its performance on other paraphrase identification datasets. PARADE can serve as a resource for researchers interested in testing models that incorporate domain knowledge. We make our data and code freely available.

## introduction paraphrases are sentences that express the same (or similar) meaning by using different wording @xcite . automatically identifying paraphrases and non-paraphrases has proven useful for a wide range of natural language processing (nlp) applications, including question answering, semantic parsing, information extraction, machine translation, textual entailment, and semantic textual similarity. paraphrase identification (pi) is typically formalized as a binary classification problem: given two sentences, determine if they roughly express the same meaning. traditional paraphrase identification approaches @xcite @xcite @xcite ) mainly rely on lexical and syntactic overlap features to measure the semantic similarity between the two sentences. examples include string-based features (e.g., whether two sentences share the same words), part-ofspeech features (e.g., whether shared words have the same pos tags), and dependency-based features (e.g., whether two sentences have similar dependency trees). however, these shallow lexical and syntactic overlap features may not effectively capture the domain-specific semantics of the two sentences. a typical situation where models based on these overlap features may fail is a pair of sentences that overlap very little at the lexical and syntactic level but are semantically equivalent based on domain knowledge. consider the two paraphrases s1 and s2 in recent neural models @xcite @xcite that go beyond traditional approaches based on lexical and syntactic features have demonstrated state-of-theart performance on paraphrase identification. for example, bert and its variants @xcite @xcite @xcite have achieved the best results on the general language understanding evaluation (glue) benchmark @xcite on two paraphrase identification datasets: the microsoft research paraphrase corpus (mrpc) and quora question pairs (qqp). using massive pre-training data and a flexible bidirectional self-attention mechanism, bert and its variants are able to better model the semantic relationship between sentences. moreover, two recent studies @xcite observe that bert without fine-tuning can even capture world knowledge and can answer factual questions like "place of birth" and "who developed the theory of relativity." naturally, we are curious to know if these neural models can correctly identify paraphrases that require specialized domain knowledge like the examples shown in hence, our overarching research goal is to create new datasets and enable new models for highquality paraphrase identification based on domain knowledge. because previous paraphrase datasets @xcite @xcite @xcite were not originally designed and constructed from the perspective of domain knowledge, to date there is no such dataset that requires specialized domain knowledge to discern the quality of two candidate sentences as para-phrases. as a first step, we focus in this paper on the computer science domain. specifically, we require a dataset of paraphrases that overlap very little but are semantically equivalent, and of nonparaphrases that have overlap greatly but are not semantically equivalent based on computer science domain knowledge. correspondingly, there is a research gap in understanding if modern neural models can achieve exemplary performance on such a dataset, especially in comparison with existing paraphrase identification datasets (that lack such specialized domain knowledge). in sum, this paper makes four contributions: the key idea is to leverage large-scale online collections of user-generated flashcards. we treat definitions on each flashcard's back side that correspond to a common entity on the front side (e.g., "machine code") as candidate paraphrases. in this paper, we propose a novel framework to collect sentential paraphrases from online usergenerated flashcards, where different definitions (on the back of flashcards) of the same entity (on the front of flashcards) are probably paraphrases. the main advantage of this framework is that it can easily collect domain-specific paraphrases. since flashcard websites like quizlet are mainly used by students to prepare for quizzes and exams, these flashcards are often organized by subject, providing a rich source of domain-specific paraphrases. datasets for paraphrase identification: to our best knowledge, there are five publicly available sentential paraphrase identification datasets: microsoft research paraphrase corpus (mrpc) @xcite ) contains 5,801 pairs of sentences from news articles, pit-2015 @xcite ) contains 18,762 pairs of tweets on 500 distinct topics, twitter-url @xcite contains 51,524 pairs of tweets containing 5,187 distinct urls, quora question pairs (qqp) @xcite ) contains 400k foot_1 pairs of question pairs on quora and paws @xcite ) contains 53,402 pairs of sentences by using word scrambling methods based on qqp. these datasets were not originally designed and constructed from the perspective of domain knowledge. hence, we present parade, the first sentential dataset for paraphrase identification based on domain knowledge as shown in ## collecting domain-specific paraphrases from online flashcards in this section, we propose a novel framework that constructs a domain-specific paraphrase corpus from online user-generated flashcards. we choose computer science as the target domain in this paper as a first step. the framework can be easily applied to construct datasets of other domains. many web platforms provide flashcards like quizlet, studyblue, ankiweb, and cram. each flashcard generated by a user is made up of an entity on the front and a definition describing or explaining the entity on the back. the purpose of flashcards is to help users to understand and remember concepts like "machine code." our core idea is that two different definitions probably express the same meaning if they have the same entity on the front. hence, they can be paired as a candidate paraphrase. our framework can collect arbitrarily many definitions generated by users independently, leading to broad coverage of how native speakers are likely to describe an entity in a specialized domain. by pairing the va-riety of definitions about concepts (like "machine code"), paraphrases and non-paraphrases that requires specialized domain (e.g., computer science) knowledge to discern are generated and collected. ## parade dataset in section 3, we introduced our framework for generating domain-specific candidate paraphrases. although each one of the candidate paraphrases is focused on the same topic (entity), we still need to confirm that the two definitions express the same meaning. in this section, we introduce our annotation strategy for candidate paraphrases and formally present the parade dataset for paraphrase identification based on domain knowledge. ## experiments in this section, we present experiments that aim to answer the following research questions (rqs): ## conclusion and future work we have presented parade, a new dataset for sentential paraphrase identification requiring domain knowledge. we conducted extensive experiments and analysis showing that both state-of-the-art neural models and non-expert human annotators perform poorly on parade. in the future, we will continue to investigate effective ways to obtain domain knowledge and incorporate it into enhanced models for paraphrase identification. in addition, since parade provides entities like "machine code" for definitions, this new dataset could also be useful for other tasks like entity linking @xcite , entity retrieval @xcite and entity or word sense disambiguation @xcite .