Automatically Ranked R ussian Paraphrase Corpus for Text Generation (2020)

Abstract: The article is focused on automatic development and ranking of a large corpus for Russian paraphrase generation which proves to be the first corpus of such type in Russian computational linguistics. Existing manually annotated paraphrase datasets for Russian are limited to small-sized ParaPhraser corpus and ParaPlag which are suitable for a set of NLP tasks, such as paraphrase and plagiarism detection, sentence similarity and relatedness estimation, etc. Due to size restrictions, these datasets can hardly be applied in end-to-end text generation solutions. Meanwhile, paraphrase generation requires a large amount of training data. In our study we propose a solution to the problem: we collect, rank and evaluate a new publicly available headline paraphrase corpus (ParaPhraser Plus), and then perform text generation experiments with manual evaluation on automatically ranked corpora using the Universal Transformer architecture.

## introduction a large amount of work is dedicated for a clear understanding of the nature of a paraphrase. on the one hand, traditional theories of language allow to trace the notion of paraphrase back to the ancient rhetorical tradition (cf. greek παράφρασις 'retelling') and treat it quite broadly in case of different types of prose, verse, musical pieces, etc. on the other hand, the generative trend in linguistic research encouraged description of transformations involved in the transition from deep to surface structures and at the same time responsible for the emergence of a wide range of paraphrases, cf. chomskian generative grammar giving account of various lexical transformations, melchuk's sense-text theory postulating the process of paraphrasing as synonymic conversion, etc. in recent works paraphrases are treated as "alternative expressions of the same (or similar) meaning" @xcite . ranking paraphrases as regards their similarity in form and meaning is reflected in a set of paraphrase classifications, where precise paraphrases are distinguished from quasi-paraphrases and non-paraphrases @xcite . at the same time, paraphrase corpora development required deep analysis of paraphrase transformations types (e.g. morphosyntactic, lexical and semantic shifts). paraphrasing plays an important role in a broad range of nlp tasks, including but not limited to question answering, summarization, information retrieval, sentence simplification, machine translation and dialogue systems. however, in order to be able to train a good paraphrasing system, large parallel corpora are required, which can be a problem in underdeveloped languages from a data resources standpoint. in order to bridge this gap, we propose a methodology to collect enough data for proper deep learning. ## motivation and related work paraphrase identification inspired a set of nlp competitions within semeval conferences in 2012, 2013, 2015 and 2016, so that baseline decisions and their improvements were worked out for english. there also exist several well-known manually annotated paraphrase datasets for english: microsoft paraphrase (dolan and brockett, 2005), quora question pairs and ms coco @xcite . however, russian is less represented in paraphrase research both in case of resource development and algorithm evaluation, a few exceptions being ainl paraphrase detection competition in 2016 based on paraphraser corpus and dialogue paraphrased plagiarism detection competition in 2017 based on paraplag corpus. alongside with paraphraser and paraplag, there are some para-phrase resources which include russian language, for instance by opusparcus @xcite and ppdb @xcite . in our study we mainly focus on the collection, evaluation and generation of the, so called, sentential paraphrases. this approach is different from the collection of ppdb, where sub-sentential paraphrases, such as individual word-pairs, were also included and paraplag with main focus on textlevel rephrasing. recent work @xcite @xcite provides solid evidence in favour of paraphrase generation by means of seq-2-seq architectures. the main problem, however, is that such systems require significant expansion of existing datasets for proper machine learning @xcite . the lack of data still remains the greatest obstacle to the development of a stable generation system which could be lexically rich and insensitive to rare words. e.g., the largest datasets supplied with proper annotation seldom exceed 100k samples in size. the authors of the aforementioned articles claim that any user generated content is valuable even though noisy to a certain extent. we propose a solution which overcomes the given problem, and it is based on the denoising procedure which has recently attracted growing attention. we argue that automatically matched and ranked datasets can be used for paraphrase generation task, especially in low-resource languages, by providing experimental results obtained on the russian opusparcus subcorpus and on the novel paraphraser plus corpus. ## source data the paraphraser plus corpus foot_0 is distilled from a database of news headlines, that was kindly provided by the russian internet monitoring service, "webground". although, the contents of the resources are pretty similar, the data itself in the original paraphraser corpus and the paraphraser plus corpus as well as the methodology used to collect the headlines are not the same by any means. it is important to note, however, that ranking model which will be described in the corresponding section was based on the original corpus. the headlines in "webground" were initially clustered by events over a ten year span, beginning from the year 2009. following the hypothesis that within such theme-based user-generated clusters the chance of seeing a paraphrase is particularly high, we formed sets of pairs of all possible combinations within each of them. after weeding out pairs, consisting of the same tokens, we were left with just over 56 million pairs of potential paraphrases. we have also discarded over 200 thousand headlines where it was not possible to verify the authorship. ## ranking methodology there are several known approaches to paraphrase ranking, including heuristic scoring @xcite and supervised modelling @xcite . heuristic scoring can be effectively conducted in resources with cross-linguistic support, such as ppdb and opuspracus. however, paraphraser, as well as our addition, is monolingual, therefore this approach was not possible. on the other hand, supervised modelling techniques can be adopted: there is a significant amount of labeled data in the original paraphraser corpus and several approaches to paraphrase identification in russian headlines have been thoroughly researched and summarized in @xcite . the methods included shallow neural networks, linguistic features based classifier and a combination of machine translation with semantic similarity. however, recent research conducted in @xcite shows that deep bidirectional pretrained monolingual transformers improve paraphrase detection in russian by a large margin. it was shown that finetuning a monolingual bert based model (rubert) on the paraphraser corpus yields results far better than all of the aforementioned approaches (see the training set in paraphraser includes 7,227 pairs of sentences, which are classified by humans into three classes: 2,582 non-paraphrases, 2,957 near-paraphrases,and 1,688 precise-paraphrases. the aforementioned rubert model was finetuned to a binary classification task: both nearparaphrases and paraphrases were considered to be a single class. such approach helps in automatic ranking: it is possible to sort the items in accordance to the probability of the paraphrase class in descending order. the fine-tuned rubert model is available as part of the deeppavlov library @xcite , which enabled us to adopt this approach in our corpus construction study. ## ranking evaluation in order to evaluate our supervised automatic ranking approach we randomly select a subsample of 500 pairs for manual annotation. to provide a more thorough comparison analysis we step aside from the original 3-way annotation scheme utilized in paraphraser and adopt the approach provided in @xcite with more similarity degrees. the annotation scheme from the original paper is provided in to measure the inter-annotator agreement we use fleiss kappa, which is a cohen's kappa generalization to more than two annotators (in our case -5); expected agreement is calculated on the basis of the assumption that random assignment of categories to items, by any annotator, is governed by the distribution of items among categories in the actual world. the annotators reach a fair agreement (kappa 0.267, p-@xmath0.05). the cosine similarity baseline solution of word2vec embeddings achieves a manual annotation pearson's correlation coefficient of 0.535. our supervised model rankings for paraphraser plus dramatically improve correlation with human judgments (@xmath1.734). ## paraphrase generation to test our initial hypothesis we conduct a paraphrase generation experiment on two datasets: opusparcus and our paraphraser plus. there exist several methods to generate paraphrases. the following techniques are known: rulebased @xcite , seq-2-seq @xcite @xcite , reinforcement learning @xcite , deep generative models @xcite and a varied combination @xcite of the later three. we show the results that can be achieved on large automatically ranked corpora using a sequence-to-sequence model based on the universal transformer architecture as it has demonstrated superior performance over the past year in multiple generative tasks, such as abstractive summarization, machine translation and, of course, paraphrase generation. @xcite @xcite @xcite . as pointed out in @xcite , the attention heads in the transformer model can be found very useful in learning grammatical, syntactical, morphological and semantical behavior in the language, which is essential in paraphrase generation. such results are being achieved thanks to the fact that input vectors are connected to every other via the attention mechanism, thus allowing the network to learn complex rephrasing dependencies. moreover, contrary to recurrent neural networks, a transformer can be trained in parallel. for both datasets, opusparcus and paraphraser plus, we used the same set of model hyperparameters: 4 layers in the encoder and decoder with 8 heads of attention. in addition, we added a dropout of @xmath2.3. the models were trained until convergence with the adam optimizer using a scaled learning rate, as proposed by the authors of the original transformer we also adopt byte-pair encoding (bpe), a data compression technique where often occuring pairs of bytes are replaced by additional extra-alphabet symbols. thanks to this approach, the most frequent parts of words are kept in the vocabulary, while rarely occuring words are replaced by a sequence of several tokens. languages with rich morphology benefit the most as the word endings could be separated since each word form is definitely less frequent than its stem. bpe encoding allows us to represent all words, including the ones unseen during training (e.g. first and last names, which are common in headlines), with a fixed vocabular. ## results we perform experiments on the above mentioned datasets, and report, both qualitative and quantitative results of our approach. as can be seen in reference: "teachers and doctors in russia may be prohibited from accepting gifts" generated: "the state duma proposed to ban doctors and teachers from accepting gifts" despite the fact that both of the training sets are noisy to a certain extent, the model is able to generalize and generate paraphrases of decent quality (from semantic and grammatical standpoint) for types of content it has never seen during the training phase. ## conclusion this study confirms our initial hypothesis that data size restrictions can be effectively resolved with automatically ranked corpora, especially in lowresource languages where large manually annotated datasets are not available. we also present a newly gathered paraphraser plus corpus and results achieved by a transformer model applied to it.