Creating an Aligned R ussian Text Simplification Dataset from Language Learner Data (2021)

Abstract: Parallel language corpora where regular texts are aligned with their simplified versions can be used in both natural language processing and theoretical linguistic studies. They are essential for the task of automatic text simplification, but can also provide valuable insights into the characteristics that make texts more accessible and reveal strategies that human experts use to simplify texts. Today, there exist a few parallel datasets for English and Simple English, but many other languages lack such data. In this paper we describe our work on creating an aligned Russian-Simple Russian dataset composed of Russian literature texts adapted for learners of Russian as a foreign language. This will be the first parallel dataset in this domain, and one of the first Simple Russian datasets in general.

## introduction automatic text simplification is a task of natural language processing aimed at making texts more readable and accessible to a broader audience. nowadays, this task is most often viewed as a neural machine translation problem @xcite . much like cross-lingual machine translation, intralingual neural text simplification requires a substantial amount of data in order to be able to train and test appropriate models. currently there are multiple datasets to choose from for english text simplification, such as simple ppdb @xcite , simple wiki @xcite and newsela @xcite . however, not a lot of simplification-specific datasets are available for other languages. a few exceptions are, for example, the spanish version of the newsela dataset and the alector parallel simplified french corpus @xcite . for russian, there are currently very limited options for publicly available simplification corpora. one such example is the dataset for one of the dialogue evaluation tasks (rusimplesenteval) foot_0 . however, this dataset consists mainly of texts from the wiki-large corpus @xcite that were translated into russian automatically and parallel data from the russian paraphrase corpus @xcite . only the texts from the development and test set have been simplified by human experts. we will, instead, produce a corpus of russian texts adapted by human experts for language learners, a unique resource for further work on russian text simplification. creating a new dataset for text simplification poses quite a few challenges. first of all, there needs to be a source of such data. possible options include simplified wikipedia articles like simple english wikipedia or vikidia foot_1 , a french website intended for young people with more accessible articles than in wikipedia, both in terms of language and content @xcite . other possible sources could be texts simplified for children or second language learners, since not many languages actually have a simplified wikipedia equivalent. there is also an option of simplifying some texts manually with the help of human experts, like it was done in newsela or alector, but this is expensive and slow. another issue that gathering simplified data poses is the fact that there is no one simple language. simplified texts can vary in complexity levels and intended target audience: for example, texts adapted for second language learners or children typically become more complex as the reader's age or l2 proficiency increases. the comprehensibility of the reading materials for users of easy language is typically higher than for plain language users @xcite . therefore, it is important to remember that it is impossible for one dataset to satisfy the needs of all simplified language users. in this paper, we will describe the process of creating a russian-simple russian parallel dataset made out of adapted russian literature texts intended for learners of russian as a foreign language (raafl). the corpus we produced is aligned on the paragraph level. ## data sources the contents of the dataset described here are simplified russian books and their original versions. the simplified books were adapted by professional raafl writers. they have been provided by the zlatoust publishing house 3 for the purposes of this research. the books and short stories in our dataset are all works by russian writers, most of them being classical literature, such as novels by a. s. pushkin, f. m. dostoevsky, and a. p. chekhov. there are also more modern works by writers like b. akunin and v. pelevin. among the books provided for this dataset are 6 collections of novels and 16 separate books. most of the books and collections of novels are aimed at readers with an intermediate level of russian, with some of them being aimed at lower and higher levels. all together, there are 2 books in this dataset dedicated for a2 level, 2 for levels a2-b1, 11 for b1, 1 for b1-b2, 3 for b2, and 3 collections of novels that fall between b1 and c1 levels. for evaluation and as an additional reference, we have also created a smaller dataset that is made of 300 adapted texts from textbooks for children who are learners of russian as a native language (1-4 school grades). those are short texts from exercises that we manually aligned with their original versions. at the moment, we are negotiating the possibility of giving open access for research purposes to parts of the collection. we aim at publishing a large portion of the parallel data with permissive licenses depending on the agreement we can establish. we will make that part available on github or similar websites. ## data collection and preprocessing the original format of the books and novel collections that were used for this dataset is pdf. extracting the text from pdf files creates the first 3 http://zlat.spb.ru/ challenge in our process. the conversion was done with the help of the apache tika software foot_2 that allowed us to turn pdf documents into accessible xml files. after that, the texts were further cleaned and normalized to remove unnecessary line breaks, diacritics and other noisy symbols with the help of a dedicated python script. the original texts that were downloaded from open sources were also processed in the same way and all texts were then stored in plain text files for further analysis and subsequent alignment. while studying the characteristics of simplified texts, we performed word and sentence tokenization and, in certain cases, lemmatization on our dataset. for lemmatization, we used pymystem3 foot_3 , a wrapper for yandex mystem 3.1, which is a morphological analyzer for russian @xcite . it is a dictionary-based algorithm, which has been proven to achieve up to 96,43% accuracy on pos tagging @xcite , and up to 82,08% accuracy on lemmatization @xcite . we also relied on mystem for word tokenization, considering only words that it recognizes as tokens. therefore, for, example, digits and punctuation were not considered words (tokens). for sentence tokenization, we used the nltk sentence tokenizer for russian which is based on ru punkt foot_4 . it is important to note that these instruments were used only for obtaining dataset statistics and not during text alignment or neural text simplification. ## characteristics of the simplified texts first of all, we wanted to see how different the adapted texts are compared to the corresponding original versions. in this study, we focus on automatic measures to provide essential properties of the texts and we leave a deeper analysis of the simplification level to future work. we looked at a number of metrics that are commonly used for determining the complexity of texts for raafl learners @xcite . the results are listed in it can be seen that original and adapted texts vary in length with the originals having more words on average. however, the number of sentences per paragraph stays almost the same. average word length and sentence complexity (approximated by the average number of punctuation characters per sentence) are slightly bigger in the original versions. adapted texts also have higher readability according to the flesch-kincaid grade level. nevertheless, the degree of lexical variation does not seem to change drastically according to the typetoken ratio (ttr). we also calculated the number of words in the texts that are supposed to be known to the reader on a certain level of language acquisition. there are established lexical minimums with appropriate vocabulary to every level of raafl (see @xcite , @xcite kozlova (2011), andryushina et al. (2018), andryushina et al. (2019a), and andryushina et al. (2019b) ). these lists consist of lemmatized words, so the texts were also lemmatized before counting the percentages of known words. we also considered people's names (that is, first names, surnames and patronymics) and location names to be known to the reader, since in literature the audience quickly becomes acquainted with characters and the places where the plot is set up. we identified those words with the help of pymystem3. for example, in russian's vocabulary. because most of our texts have more experienced readers as their target audience, the amount of known vocabulary for a1 and a2 levels is lower on average. we can see that even at the c1 level, an average speaker will most likely not recognize all words even in the adapted versions of texts. however, the percentage of known words is higher in the adapted books at all levels. ## text alignment further analysis and the use of the data for training automatic text simplification tools require an explicit alignment between original and simplified text segments. many alignment algorithms and tools have been proposed for the creation of parallel corpora mainly in the context of machine translation research. we used multiple alignment approaches ranging from semi-automatic to fully automatic tools. first of all, we tried intertext @xcite ) -an editor for managing parallel texts. it has multiple options for automatic pre-alignment of the text that can then be adjusted using a graphical interface. in our work, we tried hunalign @xcite . this aligner turned out to have some shortcomings, especially in aligning big texts where one version is significantly longer than the other. this is not a surprise as hunalign (as most of the other algorithms) is developed for cross-lingual sentence alignment and not for linking texts in the same language but different levels of complexity. nevertheless, the intertext interface provides a convenient tool that enabled a semi-automatic alignment of a smaller dataset that we can use for the evaluation of fully automatic alignment algorithms later on. using this method we created a gold standard of 302 aligned segments taken from textbook exercises. as mentioned above, we chose paragraph alignment for our texts. we believe that text simplification should not consider isolated sentences only but also look at additional context. we are interested in cases where larger portions of the text are removed, sentences are merged or split or even some information is expanded. also, creating sentence alignments out of an existing paragraph alignment is more effective and reliable than just aligning sentences from texts @xcite . hence, we can easily extend the dataset with additional linking between individual sentences later on. using bleualign on our data resulted in 7452 aligned paragraphs, and 9352 with cats-align. other statistics, such as total numbers of word tokens, sentences, and the number of unique words (vocabulary) can be found in ## conclusions we have described the process of creating a parallel russian-simple russian dataset made of literature texts adapted by human experts. this dataset is still a work in progress. for now, we have described some linguistic properties of our texts and established differences between various alignment strategies. we also provide initial tests on the use of the data for neural text simplification. in the future we plan to add sentence alignments and further improvements to the automatic text simplification derived from the data. data and models will be publicly released if copyrights permit.