Text Simplification with Reinforcement Learning Using Supervised Rewards on Grammaticality, Meaning Preservation, and Simplicity (2020)

Abstract: We optimize rewards of reinforcement learning in text simplification using metrics that are highly correlated with human-perspectives. To address problems of exposure bias and loss-evaluation mismatch, text-to-text generation tasks employ reinforcement learning that rewards task-specific metrics. Previous studies in text simplification employ the weighted sum of sub-rewards from three perspectives: grammaticality, meaning preservation, and simplicity. However, the previous rewards do not align with human-perspectives for these perspectives. In this study, we propose to use BERT regressors fine-tuned for grammaticality, meaning preservation, and simplicity as reward estimators to achieve text simplification conforming to human-perspectives. Experimental results show that reinforcement learning with our rewards balances meaning preservation and simplicity. Additionally, human evaluation confirmed that simplified texts by our method are preferred by humans compared to previous studies.

## introduction text simplification is one of the text-to-text generation tasks that rewrites complex sentences into simpler ones. text simplification is useful for preprocessing of nlp tasks such as semantic role labeling @xcite and machine translation @xcite . it also has valuable applications such as assisting language learning @xcite and helping language-impaired readers @xcite . there are two problems in text-to-text generation with an encoder-decoder model: exposure bias and loss-evaluation mismatch @xcite . the former is that the model is not exposed to its own errors during training. the latter is that while the generated sentence is evaluated as a whole sentence during inference, it is evaluated at the token-level during training. to address these problems, reinforcement learning has been employed in text-to-text generation tasks, such as machine translation @xcite and abstractive summarization @xcite . these studies use metrics suitable for each task, such as bleu @xcite and rouge @xcite , as rewards. although reinforcement learning based text simplification models @xcite have used rewards metrics such as sari @xcite and fkgl @xcite , these metrics do not align with human-perspectives, i.e., human evaluation results @xcite @xcite . in this study, we train a text simplification model based on reinforcement learning with rewards that highly agree with human-perspectives. specifically, we apply a bert regressor @xcite on grammaticality, meaning preservation, and simplicity, respectively, as shown in figure 1 . experiments on the newsela dataset @xcite have shown that reinforcement learning with our rewards balances meaning preservation and simplicity. further, manual evaluation has shown that our outputs were preferred by humans compared to previous models. ## background: reinforcement learning for text simplification reinforcement learning in text-to-text generation tasks is performed as additional training for pretrained text-to-text generation models. it is a common technique to linearly interpolate a reward of reinforcement learning and the cross-entropy loss to avoid misleading training because of a large action space @xcite . we first explain an attention based encoderdecoder model (encdeca) @xcite in section 2.1 and then reinforcement learning for text simplification in section 2.2. ## bert-based supervised reward we propose a reward estimator r(x, ŷ ) consisting of sub-rewards for grammaticality r g , meaning preservation r m , and simplicity r s . these subrewards are combined by weighted sum with hyper parameters of δ and : to achieve a better correlation between each subreward and human perspectives, we employ bert regressors and fine-tune them using manually annotated datasets. ## end-to-end evaluation on text simplification in this section, we evaluate our rewards on an endto-end text simplification task using the newsela dataset 6 shown in ## conclusion we trained a text simplification model based on reinforcement learning with rewards that are highly correlated with human-perspectives. experimental results showed that existing rewards employing evaluation metrics tend to generate extremely simple sentence at the expense of meaning preservation. nevertheless, our bert-based rewards succeeded in balancing meaning preservation and simplicity. in addition, we confirmed that human evaluators prefer our simplified sentences to those generated by previous rewards. in this study, we set the equal weights to our subrewards. we plan to investigate the better weight balance of sub-rewards in the future.