Controllable Text Simplification with Deep Reinforcement Learning (2022)

Abstract: We propose a method for controlling the difficulty of a sentence based on deep reinforcement learning. Although existing models are trained based on the word-level difficulty, the sentence-level difficulty has not been taken into account in the loss function. Our proposed method generates sentences of appropriate difficulty for the target audience through reinforcement learning using a reward calculated based on the difference between the difficulty of the output sentence and the target difficulty. Experimental results of English text simplification show that the proposed method achieves a higher performance than existing approaches. Compared to previous studies, the proposed method can generate sentences whose grade-levels are closer to those of human references estimated using a fine-tuned pre-trained model.

## introduction text simplification @xcite is a task of rewriting complex sentences into simpler versions while preserving the meaning. this technology assists people with language disabilities @xcite , language learners @xcite , and children @xcite in reading texts and learning a language. to maximize the effectiveness of text simplification, rewrites should be appropriate for the language ability of the target audience. therefore, controllable text simplification @xcite @xcite , which is controlled to match the target difficulty level, has been actively studied. controllable text simplification models are trained on a parallel corpus of complex and simple sentences with labels for the target difficulty level, such as newsela @xcite . although studies have focused on word-level difficulties @xcite , sentence-level difficulties were not taken into account. therefore, while these methods are effective for local editing, such as word substitution, there is room for improvement for global editing, for example, controlling the sentence length and structure. such global editing is crucial to improve the controllability of the sentence difficulty. to address this problem, we propose a controllable text simplification model based on deep reinforcement learning to take advantage of sentencelevel objectives. although deep reinforcement learning has also been used in traditional text simplification foot_0 @xcite , in this study, a novel reward function for difficulty control is designed. our reward is calculated based on the difference between the difficulty of the generated sentence and the target difficulty level. experimental results using newsela-auto @xcite show that the proposed method can generate sentences whose grade-levels are closer to those of human references estimated using a finetuned pre-trained model than the previous methods. in traditional text simplification @xcite , deep reinforcement learning has been used to improve the simplicity of the sentences generated using lstm-based simplification models @xcite . @xcite improved the simplification performance with sari @xcite , an evaluation metric for text simplification, as a reward. @xcite trained reward models for the grammaticality, synonymity, and simplicity through supervised learning using bert @xcite . we also use deep reinforcement learning based on a reward estimated by bert. however, our approach differs from that of @xcite in two ways. first, we target controlling the difficulty of the output sentences. second, we use a powerful transformer-based @xcite simplification model, which has been the mainstream in recent years @xcite @xcite . ## proposed method we improve the controllability of the sentence difficulty through reinforcement learning using a reward based on the sentence-level difficulty on a previous controllable text simplification model (scarton and specia, 2018). our model consists of a difficulty estimation model and a simplification model. the former model estimates the difficulty of the generated sentence, and the latter model is trained through reinforcement learning to minimize the difference between the estimated and target difficulties. foot_1 ## conclusion we proposed a deep reinforcement learning method for controllable text simplification that takes into account the sentence-level difficulty. we designed a reward based on the difference between the target difficulty and that of the generated sentence. experimental results show that our method is evaluated highly owing to its overall simplification in an automatic evaluation, and for its controllability of the sentence difficulty in a manual evaluation.