R ussian Paraphrasers: Paraphrase with Transformers (2021)

Abstract: This paper studies the generation methods for paraphrasing in the Russian language. There are several transformer-based models (Russian and multilingual) trained on a collected corpus of paraphrases. We compare different models, contrast the quality of paraphrases using different ranking methods and apply paraphrasing methods in the context of augmentation procedure for different tasks. The contributions of the work are the combined paraphrasing dataset, fine-tuned generated models for Russian paraphrasing task and additionally the open source tool for simple usage of the paraphrasers.

## introduction one of the prominent features of any natural language is its diversity. variability and ambiguity of natural languages lead to infinity of sequence combinations and one can always form a new sentence that has never been said before. however, there are approaches to automatic construction of texts with roughly the same meaning: paraphrases. paraphrasing is expressing the meaning of an input sequence in alternative ways while maintaining grammatical and syntactical correctness. paraphrases are of a great use in diverse applications on downstream nlp tasks and are presented in two main task forms: 1) paraphrase identification -detecting if a pair of text inputs has the same meaning; classification task. 2) paraphrase generation -producing paraphrases allows for the creation of more varied and fluent text; generation task. the identification of paraphrases is very useful in many tasks, such as multi-document summarization (identifying paraphrases allows to condense information repeated across documents), question answering (checking the sequences of the tests, keyword matching to find answers), semantic parsing and search (to find the same queries or documents) and many others @xcite . in this work we will discuss paraphrase generation applicability. paraphrase generation is used in different nlp applications (for example, in chatbots to diversify responses @xcite ) and sub-tasks. paraphrasers can be used to augment datasets with new data. for question answering systems, paraphrasing questions can not just increase the number of data examples for training ml-models @xcite , but are also used to match them with key words in the knowledge base. paraphrasers can help generate adversarial examples to evaluate model robustness -increasing the stability of ml-models: training models on a wide variety of examples in different styles, with different sentiment, but the same meaning or intent of the user. the demand for targeting paraphrasers for generating specific writing styles is also trending now @xcite . this type of paraphrasing performs different types of style transfer, such as changing style from rude to polite, or from professional to simple language. there are some general approaches for paraphrase generation. rule-based approaches @xcite and data-driven methods (madnani and dorr, 2010) are the oldest ones. currently, the most common approach is to consider the task as supervised learning using sequence-to-sequence models @xcite . the unsupervised approaches @xcite are also very common. other methods proposed include use of deep reinforcement learning @xcite . fine-tuning with large language models such as gpt2 is also a valuable approach that can be considered supervised @xcite or unsupervised @xcite . the majority of the resources and methods for paraphrasing are proposed for the english language. for the russian language there were several attempts of paraphrase corpora creation @xcite . in 2016 the collection of the russian paraphrase corpus and the paraphrase detection shared task @xcite were organized, which attracted attention to the topic and led to a number of further works on the identification of paraphrases @xcite and sentence similarity experiments @xcite . in this paper, we compare different language models for paraphrase generation in russian, namely rugpt2-large, rugpt3-large, and multilingual models -mt5. we prove that all these models can generate good russian paraphrases and test different ranking methods on generated examples. we provide the combined paraphrasing dataset, finetuned generated models for russian paraphrasing task, augmentation experiments on data for common nlp tasks, and additionally present the open source tool for user-friendly usage of the russian paraphrasers. this paper is structured as follows: in section 2 we present the methodology -the dataset we use 2.1, models we fine-tune 2.2, and range strategies for paraphrasers output 2.3; section 3 is devoted to evaluation and analysis of the paraphraser performance and results -the models scores 3.1, the augmentation procedure with paraphrasers 3.2, and 3.3 the discussion about the results in the context of paraphrase application; and section 4 concludes the paper. ## methodology language models achieve impressive results when trained in a semi-supervised manner to predict the next word or words in a sequence. they can be fine-tuned and used for a variety of downstream nlp tasks (text classification, sentiment analysis, ner etc.). good examples of such large language models that can be used for text generation are gpt-2, gpt-3, and mt5. in this section, we present our experiments with these models for russian trained on the prepared dataset. ## evaluation we propose a two-step evaluation procedure: 1) universal metrics between gold testset examples and generated models outputs and 2) application of paraphrasers on downstream tasks where we augment data. additionally, we will discuss the quality of the paraphrases evaluated by humans on subset of examples. ## conclusion paraphrase generation with large language models achieves impressive results. our experiments show that both multilingual and russian-oriented models are able to quickly learn the task of paraphrasing through fine-tuning training on a prepared russian set of paraphrase examples. this paper contributions are the corpus of paraphrases, 5 fine-tuned models for the russian language, comparison of them, range strategies for finding the best candidates, and the open source library in python for convenient use of the pre-trained paraphrasers. in future work, we would like to further explore the effectiveness of generated paraphrasers for different augmentation experiments and evaluate the models robustness in terms of reconstruction and generated paraphrases.