Automatic text simplification of R ussian texts using control tokens (2023)

Abstract: This paper describes the research on the possibilities to control automatic text simplification with special tokens that allow modifying the length, paraphrasing degree, syntactic complexity, and the CEFR (Common European Framework of Reference) grade level of the output texts, i.e. the level of language proficiency a non-native speaker would need to understand them. The project is focused on Russian texts and aims to continue and broaden the existing research on controlled Russian text simplification. It is done by exploring available datasets for monolingual Russian machine translation (paraphrasing and simplification), experimenting with various model architectures, and adding control tokens that have not been used on Russian texts previously.

## introduction and related work easy and plain language are tailored languages @xcite often aimed at a specific audience, such as people with learning disabilities, children, or second language learners. easy language is even considered to be a rule-based variety that reverts to purposeful language planning and shows similarities with controlled languages @xcite . despite the growing number of tools for automatic text simplification, most simplified texts are still produced by experts who understand and cater to the needs of a particular group of readers. because of that, it seems reasonable to concentrate on the task of controllable text simplification so that in the future, simplification tools can be tailored to specific target audiences. at present, text simplification is often viewed as a monolingual text-to-text generation task borrowing ideas from statistical machine translation @xcite , and simplification models are trained in a similar fashion to translation models. the training requires large parallel datasets where the target sentences are simplified versions of the source sentences. there are multiple ways to control the output of text simplification tools. for example, editing operations can be directly controlled. @xcite presented a simplification model that could learn explicit editing operations such as additions, deletions, and keeping. @xcite proposed a sequence labeling model to predict which simplification operations should be performed as a first step for a complete simplification pipeline. the model is built on a corpus with automatically labeled simplification operations, and the approach is proven to produce more straightforward texts than end-to-end models. other research shows that, apart from controlling editing operations, it is also possible to control specific dimensions of the output texts. @xcite identify four attributes related to the text simplification process: the amount of compression, paraphrasing, lexical and syntactic complexity -and use control tokens that are put in front of the source sentences to modify these attributes in output texts. this approach was later used in @xcite and in @xcite . the latter was the winning solution for the rusimple-senteval @xcite shared task on russian text simplification. this methodology is used in the present study as well. other studies have shown that control tokens can be used for all kinds of linguistic attributes, including politeness and monotonicity (the closeness of the word order in the target sentence to the word order in the source sentence) @xcite . some studies also demonstrate the successful usage of control tokens to generate texts for a given school grade level @xcite . in this project, we use various datasets for monolingual russian machine translation tasks, namely paraphrasing and simplification, to build models for controllable text simplification. the data is described in section 2. section 3 talks about the control tokens used in this study, the process of choosing the optimal model architecture, and the results of the experiments. the final parts of the paper present the conclusions and discuss the limitations of this research. ## data for this project, four different data sources were used: the size of the dataset can be seen in the data only includes sentences with five tokens or longer. furthermore, to avoid hallucinations in the output (incoherent texts possibly including facts not justified by the training data), the larger parts of the dataset, paraphraser plus and opusparcus, were cleaned from sentence pairs where named entities do not match. the natasha toolkit foot_1 was used to ## conclusions this paper continues and expands previous research on controlled text simplification. we studied the influence of control tokens on russian texts using open-source datasets. also, another transformer architecture was tested not previously used for these kinds of experiments. in the end, the choice fell on mbart, but the experiments have shown that t5 can also learn the meaning of control tokens. two tokens were tested that have not been applied to russian data before. the findings show that the deptreedepth token does not perform as well on russian data as it did on english, according to previous research. the cefrgrade token can influence the model's output in a desirable way, but according to the experiments' results, it cannot be combined with other tokens. finally, it was confirmed that the other two tokens, nbchars and levsim, work well on russian data. some examples of the models' outputs can be found in appendix a. the best models' checkpoints and other supplementary materials can be found on github: https://github.com/annadmitrieva/ controlled_simplification_ru . the findings show that some tokens are "harder" to learn for the models than others. possible topics for future research include more in-depth studies of "difficult" tokens and finding methods for representing their attributes in more understandable ways to the models. another possible topic is studying how to combine tokens more effectively and why some combinations do not work well.