P ara SCI : A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation (2021)

Abstract: We propose ParaSCI, the first large-scale paraphrase dataset in the scientific field, including 33,981 paraphrase pairs from ACL (ParaSCI-ACL) and 316,063 pairs from arXiv (ParaSCI-arXiv). Digging into characteristics and common patterns of scientific papers, we construct this dataset though intra-paper and inter-paper methods, such as collecting citations to the same paper or aggregating definitions by scientific terms. To take advantage of sentences paraphrased partially, we put up PDBERT as a general paraphrase discovering method. The major advantages of paraphrases in ParaSCI lie in the prominent length and textual diversity, which is complementary to existing paraphrase datasets. ParaSCI obtains satisfactory results on human evaluation and downstream tasks, especially long paraphrase generation.

## introduction a paraphrase is a restatement of meaning with different expressions @xcite . being very common in our daily language expressions, it can also be applied to multiple downstream tasks of natural language processing (nlp), such as generating diverse text or adding richness to a chatbot. at present, paraphrase recognition or paraphrase generation are largely limited to the deficiency of paraphrase corpus. especially, due to the permanent vacancy of paraphrase corpus in the scientific field, scientific paraphrase generation advances slowly. scientific paraphrases can not only be helpful for data augmentation of challenging scientific machine translation, but is also effective for polishing scientific papers. however, existing paraphrase datasets are mainly from news, novels, or social media platforms. most of them remain short sentences and interrogative or oral style. as a result, none of such training data can train out a scientific paraphrase generator. taking the sentence "we used pos tags predicted by the stanford pos tagger" as an example, the generated sentences from transformer @xcite models trained on existing paraphrase datasets foot_0 are "level basic topics : what is the basic purpose of stanford traditional hmo" and "a picture of a street sign with a sign on it", far from ground-truth paraphrases. we have noticed that the structure of scientific papers is nearly fixed. paraphrase sentence pairs appear not only within a paper (intra-paper) but also across different papers (inter-paper), which makes it possible to construct a paraphrase dataset in the scientific field. for example, repetitions of the same crucial contribution in a paper or explanations of the same term in different papers are potential paraphrases. based on such characteristics, we design different methods to extract paraphrase pairs (shown in section 4). in terms of the construction methods, existing methods merely focus on the paraphrase relationship between entire sentences, while hardly handle sentences with partial paraphrase parts, leaving much original corpus idle. we find that if part of a sentence paraphrases another short sentence, such sequences will be filtered out because the overall semantic similarity is not high enough. for paraphrase discovering in this case, we fine-tune bert to extract semantically equivalent parts of two sentences and name it pdbert. in order to train pdbert, we construct pseudo training data by stitching existing paraphrase sentences, and train a paraphrase extraction model using the pseudo training data. in the end, this model performs well on real scientific texts. after filtering, we obtain 350,044 paraphrase pairs and name this dataset parasci. it consists of two parts: parasci-acl (33,981 pairs) and parasci-arxiv (316,063 pairs). compared with other paraphrase datasets, sentences in parasci are longer and more sententially divergent. parasci can be used for training paraphrase generation models. furthermore, we hope that it can be applied to enlarge training data for other nlp tasks in the scientific domain. our main contributions include: ## related work paraphrases capture the essence of language diversity @xcite and play significant roles in many challenging nlp tasks, such as question answering @xcite , semantic parsing @xcite and machine translation @xcite . development in paraphrases relies heavily on the construction of paraphrase datasets. ## manual evaluation we conduct a manual analysis of our dataset in order to quantify its semantic consistency and literal variation lexically, phrasally and sententially. we employ 12 volunteers who are proficient in english to rate the instances. three human judgements are obtained for every sample and the final scores are averaged across different judges. ## paraphrase phenomenon occurrence in order to show the differences across paraphrase datasets, we sample 100 sentential paraphrases from each dataset and count occurrences of each phenomenon. @xcite grouped common paraphrase phenomenon into 6 categories : synonym (substitute a word with its synonym), voice (change the voice of sentence from active to passive or vice versa), word-form (change a word into a different form), break (break a long sentence down into small sentences), definition (substitute a word with its definition or meaning), structure (use different sentence structures to express the same thing). we report the average number of occurrences of each paraphrase type per sentence pair for each corpus in as shown in ## paraphrase generation to demonstrate further application of parasci, we train paraphrase generation models on quora, mscoco and parasci. we then test their generation ability on the same scientific corpus, including sentences from acl and arxiv respectively. for paraphrase generation from parasci-acl, we use 20,388 pairs for training, 6,796 pairs for validation and 6,797 pairs for test. for paraphrase generation from parasci-arxiv, we use 189,639 pairs for training, 63,212 pairs for validation and 63,121 pairs for test. the bleu scores and average lengths of generated sentences are shown in although there are many technology-related or scientific questions on quora, the paraphrase generation model trained on quora still fails to perform well in the scientific field, with low bleu scores and short average length. on the contrary, the paraphrase generation model trained on parasci keeps generating longer sentences. the bleu scores also demonstrate that the quality of the generated sentences is higher. this reflects the significant value of parasci on paraphrase generation. we show the generated paraphrases on different datasets in apart from that, generation models trained on parasci manifest another characteristic. as parasci consists of quantities of scientific terms and expressions, generation models trained on parasci bring valuable scientific knowledge to the output sentence. one thing to mention is that some abbreviations can be understood and utilized in the generation process. for instance, we generated "we ran mt experiments using the moses phrase-based translation system." for the sentence "we used moses as the phrase-based machine translation system.". as we use domain-related terms and corresponding abbreviations in scientific texts frequently, this advantage can add conciseness, technicality and naturality to the generated sentences. another thing to mention is that some common sense or scientific knowledge is also taken into paraphrase generation. for example, we input "the penn discourse treebank is the largest corpus richly annotated with explicit and implicit discourse relations and their senses." as the original sentence. it introduces the penn discourse treebank @xcite without any information related to its size and source, but the information is added to the generated sentences: "the penn discourse treebank is the largest available annotated corpora of discourse relations over 2,312 wall street journal articles." these cases reveal different aspects of parasci's advantages in paraphrase generation. in further work, we hope that parasci will contribute more to scientific paraphrase generation and subsequently, be applied to more downstream tasks in the scientific field. ## conclusion and future work in this paper, we describe the characteristics and construction process of parasci, a large-scale para-phrase dataset in the scientific field. it shows favorable results in the either automatic or manual evaluation. for future work, although we filter out more than 200 thousand paraphrase candidates to promise the quality of parasci, most candidates include valuable paraphrase patterns lexically or phrasally. therefore, more paraphrase patterns are remaining to be discovered. similarly, compared to the bulk data on arxiv or other scientific websites, we only use the tip of an iceberg to construct this dataset, and we are expecting to implement the methods in other scientific domains. for instance, we can obtain a biomedical paraphrase dataset from pubmed. we hope that parasci can be used to augment training data for various nlp tasks, such as machine translation in scientific field, and make more contributions to the development of nlp.